[pr into #785] Turn structured dataset into dataclass

[pingsutw]

Signed-off-by: Kevin Su pingsutw@apache.org

TL;DR

Please replace this text with a description of what this PR accomplishes.

Type

• Bug Fix
• Feature
• Plugin

Are all requirements met?

• Code completed
• Smoke tested
• Unit tests added
• Code documentation added
• Any pending items have an associated Issue

Complete description

How did you fix the bug, make the feature etc. Link to any design docs etc

Tracking Issue

https://github.com/lyft/flyte/issues/

Follow-up issue

NA
OR
https://github.com/lyft/flyte/issues/

[codecov]

Codecov Report

Merging #802 (f70e32d) into structured-dataset-proposal (4f42207) will increase coverage by 0.10%.
The diff coverage is 93.05%.

@@                       Coverage Diff                       @@
##           structured-dataset-proposal     #802      +/-   ##
===============================================================
+ Coverage                        85.60%   85.71%   +0.10%     
===============================================================
  Files                              353      356       +3     
  Lines                            30465    30585     +120     
  Branches                          3674     3679       +5     
===============================================================
+ Hits                             26080    26216     +136     
+ Misses                            3716     3700      -16     
  Partials                           669      669              

Impacted Files	Coverage Δ
flytekit/models/types.py	98.83% <ø> (ø)
flytekit/types/structured/structured_dataset.py	80.35% <73.68%> (+6.25%)	⬆️
tests/flytekit/unit/core/hint_handling/a.py	81.81% <81.81%> (ø)
tests/flytekit/unit/core/hint_handling/b.py	83.33% <83.33%> (ø)
tests/flytekit/unit/core/test_interface.py	85.40% <91.66%> (+0.43%)	⬆️
flytekit/core/interface.py	82.51% <100.00%> (+0.69%)	⬆️
flytekit/core/type_engine.py	89.13% <100.00%> (+0.13%)	⬆️
flytekit/types/structured/basic_dfs.py	92.72% <100.00%> (+4.59%)	⬆️
...ekit/unit/core/hint_handling/test_hint_handling.py	100.00% <100.00%> (ø)
tests/flytekit/unit/core/test_type_engine.py	99.62% <100.00%> (+0.01%)	⬆️
... and 5 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4f42207...f70e32d. Read the comment docs.

[wild-endeavor]

couple questions

[wild-endeavor]

works for python 3.7-3.10 right?

[pingsutw]

yeah, I've tested it with python 3.7~3.10.

[wild-endeavor]

should we have done this at the top of this function? or is it okay here?

[pingsutw]

we extract the python_type in get_transformer instead of to_literal, so python_type could be Annotated here.

[wild-endeavor]

remind me again what this inspect.isclass is supposed to catch? can you add a comment? i keep forgetting.

[pingsutw]

That's for Annotated[pd.Dataframe, my_col]. I just moved expected_python_type = get_args(expected_python_type)[0] to the beginning of the to_python and to_literal. Therefore, we don't need inspect.isclass(t) any more, so I removed it.