Add encoder/decoder in structureDataset for snowflake.

[pohanhuang]

TL;DR

1. Add encoder/decoder in structureDataset for the snowflake.
2. Currently support the new protocol, snowflake, and sf.
   we can add them here, the default is a list of additional protocols, like the following.

Type

• Bug Fix
• Feature
• Plugin

Are all requirements met?

• Code completed
• Smoke tested
• Unit tests added
• Code documentation added
• Any pending items have an associated Issue

Complete description

Example code for user, we can parse the uro with the following format of uri,
"snowflake://{dummy_user}:{dummy_account}/{dummy_warehouse}/{dummy_database}/{dummy_schema}/{dummy_table}"

import mock
import pandas as pd
import pytest
from typing_extensions import Annotated

from flytekit import StructuredDataset, kwtypes, task, workflow, Secret

@task(secret_requests=[Secret(group="snowflake", key="private_key.pem")])
def t1() -> StructuredDataset:
    df = pd.DataFrame({"NAME": ["Tom", "Joseph"], "AGE": [220, 27]})
    return StructuredDataset(
        dataframe=df, uri="sf://phuang9527:rq06467.us-east4.gcp/COMPUTE_WH/AGENT/PEOPLE/PEOPLE"
    )

@task(secret_requests=[Secret(group="snowflake", key="private_key.pem")])
def t2() -> StructuredDataset:
    df = pd.DataFrame({"NAME": ["Tom", "Joseph"], "AGE": [235, 27]})
    return StructuredDataset(
        dataframe=df, uri="snowflake://phuang9527:rq06467.us-east4.gcp/COMPUTE_WH/AGENT/PEOPLE/PEOPLE"
    )

@task(secret_requests=[Secret(group="snowflake", key="private_key.pem")])
def t3(sd: StructuredDataset) -> pd.DataFrame:
    return sd.open(pd.DataFrame).all() # we will load the whole table for user


@workflow
def wf() -> pd.DataFrame:
    t1()
    return t3(sd=t2())

if __name__ == "__main__":
    wf()

Tracking Issue

Flyte Agent Ecosystem

[codecov]

Codecov Report

Attention: Patch coverage is 59.72222% with 29 lines in your changes missing coverage. Please review.

Project coverage is 54.84%. Comparing base (660f5aa) to head (0b47872).
Report is 671 commits behind head on master.

Files with missing lines	Patch %	Lines
flytekit/types/structured/snowflake.py	45.65%	25 Missing ⚠️
flytekit/core/type_engine.py	50.00%	2 Missing ⚠️
flytekit/types/structured/__init__.py	71.42%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1811      +/-   ##
==========================================
+ Coverage   54.33%   54.84%   +0.51%     
==========================================
  Files         288      295       +7     
  Lines       21934    22282     +348     
  Branches     3344     3357      +13     
==========================================
+ Hits        11917    12221     +304     
- Misses       9855     9902      +47     
+ Partials      162      159       -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pohanhuang]

Hi @pingsutw , just finished add ingencoder/decoder in structureDataset for snowflake and unit test, please approve the testing and take a look.

[pohanhuang]

Hi @pingsutw, please approve the CI test.

[pohanhuang]

Hi @pingsutw, please approve the CI test.

[pohanhuang]

Hi @pingsutw, just finished the changes, Please take a look.

[pohanhuang]

Hi @pingsutw, just finished the changes to get, please review.
Thanks a lot!

[pohanhuang]

Hi @pingsutw, just finished the changes to get, please review.
Thanks a lot!

[pohanhuang]

Hi @pingsutw,
I finished the code change to support multiple protocols and fixed the URI.

I will create the new PR to support the new URI.
new URI: uri="snowflake://dummy_user:dummy_account/dummy_warehouse/dummy_database/dummy_schema/dummy_table

Thanks a lot

[Future-Outlier]

This is done by other PR, so I'm going to close it, thank you @hhcs9527 !