JSON IDL#2600
Conversation
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
| lv = d.to_literal( | ||
| ctx, | ||
| {"x": datetime.datetime(2024, 5, 5)}, | ||
| Annotated[dict, kwtypes(allow_pickle=True)], | ||
| LiteralType(simple=SimpleType.STRUCT), | ||
| LiteralType(simple=SimpleType.JSON), |
There was a problem hiding this comment.
This is for pickle case.
* Add an exeception when filters' value isn't a list * Make the exception more specific Signed-off-by: Nelson Chen <asd3431090@gmail.com> * add an unit test for value_in Signed-off-by: Nelson Chen <asd3431090@gmail.com> * lint Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Nelson Chen <asd3431090@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: wayner0628 <a901639@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org>
* Show different of types in dataclass when transforming error Signed-off-by: Future-Outlier <eric901201@gmail.com> * add tests for dataclass Signed-off-by: Future-Outlier <eric901201@gmail.com> * fix tests Signed-off-by: Future-Outlier <eric901201@gmail.com> --------- Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
75fff7a to
2aff851
Compare
Signed-off-by: Future-Outlier <eric901201@gmail.com>
|
Note: if we use |
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
|
Hi, @JackUrb RFC: flyteorg/flyte#5607 |
Signed-off-by: Future-Outlier <eric901201@gmail.com>
|
Hi @Future-Outlier! Here are two cases that are currently not functional on def test_DataclassTransformer_subattribute_wrapping():
@dataclass
class Base:
a: int
b: int
@task
def add_list_elems(given_ints: list[int]) -> int:
return sum(given_ints)
@task
def get_base() -> Base:
return Base(a=10, b=20)
@workflow
def wf() -> int:
base = get_base()
result = add_list_elems(given_ints=[base.a, base.b, 3])
return result
result = wf()
assert result == 33
def test_DataclassTransformer_complex_subattribute_access():
@dataclass
class Base:
a: list[int]
b: list[str]
@task
def get_data() -> Base:
return Base(a=[1, 2, 3, 4, 5], b=["a", "b", "c", "d", "e"])
@task
def list_join(in_list: list[int | str]) -> str:
return "".join([str(i) for i in in_list])
@task
def list_sum(in_list: list[int]) -> int:
return sum(in_list)
@workflow
def wf() -> tuple[int, str, str]:
data = get_data()
summed_ints = list_sum(in_list=data.a)
joined_ints = list_join(in_list=data.a)
joined_strs = list_join(in_list=data.b)
return summed_ints, joined_ints, joined_strs
result = wf()
assert result[0] == 15
assert result[1] == "12345"
assert result[2] == "abcde" |
Let me take a look tomorrow, thank you <3 |
Signed-off-by: Future-Outlier <eric901201@gmail.com>
for the case 1, it works remotely and I'm fixing local execution.
Man, you are AMAZING, I am really lucky have you <3. |
Signed-off-by: Future-Outlier <eric901201@gmail.com>
eapolinario
left a comment
There was a problem hiding this comment.
This is looking pretty good. I only have a question about double-encoding json objects.
| json_str = json.dumps(python_val) | ||
| return Literal(scalar=Scalar(generic=_json_format.Parse(json_str, _struct.Struct()))) | ||
| json_bytes = msgpack.dumps(json_str) |
There was a problem hiding this comment.
do we really need to encode json objects to a string and then use msgpack to serialize the json-encoded string?
In other words, why can't we msgpack-encode python_val directly? As in msgpack.dumps(python_val)
There was a problem hiding this comment.
In this case I think it's because the Mashumaro JSON encoder (or available to_json method) is more feature rich than msgpack.
That being said, this could probably use a to_dict -> msgpack.dumps and msgpack.reads -> from_dict flow for the data classes that need it (extending the mashumaro mix in), instead of to_json/from_json. The rest of them can use mashumaro's msgpack encoder.
There was a problem hiding this comment.
yes, @JackUrb you are right, I am doing performance testing for alternatives.
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #2600 +/- ##
==========================================
- Coverage 49.76% 47.23% -2.53%
==========================================
Files 193 193
Lines 19659 19776 +117
Branches 4097 4121 +24
==========================================
- Hits 9783 9341 -442
- Misses 9345 9932 +587
+ Partials 531 503 -28 ☔ View full report in Codecov by Sentry. |
Signed-off-by: Future-Outlier <eric901201@gmail.com>
|
use metadata, foramt->msgpack, json-type->ejson, pickle don't need json type |
Signed-off-by: Future-Outlier <eric901201@gmail.com>
| from flytekit import StructuredDataset | ||
| from flytekit.types.directory import FlyteDirectory | ||
| from flytekit.types.file import FlyteFile | ||
| from flytekit.types.schema.types import FlyteSchema | ||
|
|
||
| FLYTE_TYPES = [FlyteFile, FlyteDirectory, StructuredDataset, FlyteSchema] | ||
|
|
||
| for k, v in original_dict.items(): | ||
| if k in expected_fields_dict: | ||
| if isinstance(v, dict): | ||
| self.assert_type(expected_fields_dict[k], v) | ||
| # todo: 1. check if expected_fields_dict[k] is a flyte type, if yes, then use v to construct a flyet types and assert them | ||
| expected_type = expected_fields_dict[k] | ||
| if expected_type in FLYTE_TYPES: | ||
| new_v = copy.deepcopy(v) | ||
| new_v = expected_type(**new_v) | ||
| self.assert_type(expected_fields_dict[k], new_v) | ||
| else: | ||
| self.assert_type(expected_fields_dict[k], v) |
There was a problem hiding this comment.
This is for accessing a data class with FLYTE_TYPES.
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>
|
TODO: test it with 2 level usecase |
|
Move to flyteorg/flyte#5742 |

Tracking issue
flyteorg/flyte#5318
Why are the changes needed?
What changes were proposed in this pull request?
JSONfirst, thenStructHow was this patch tested?
local execution, remote execution and unit tests.
Note: pydantic Transformer will be supported in the follow-up PR.
Tips for collapsed sections
dataclass transformer
dict transformer
backward compatibility (upstream
struct-> downstreamjson)reference task
structoutputcache
Setup process
Screenshots
Check all the applicable boxes
Related PRs
Docs link