[pr into #785] Remove fsspec#811
Conversation
Signed-off-by: Kevin Su <pingsutw@apache.org>
pingsutw
requested review from
eapolinario,
kumare3 and
wild-endeavor
as code owners
Codecov Report
@@ Coverage Diff @@
## structured-dataset-proposal #811 +/- ##
===============================================================
+ Coverage 85.71% 85.88% +0.16%
===============================================================
Files 356 356
Lines 30585 30727 +142
Branches 3679 3680 +1
===============================================================
+ Hits 26216 26389 +173
+ Misses 3700 3672 -28
+ Partials 669 666 -3
Continue to review full report at Codecov.
|
| try: | ||
| from typing import Annotated | ||
| except ImportError: | ||
| from typing_extensions import Annotated |
There was a problem hiding this comment.
Do we need to do this? We're already pulling typing_extensions as part of a regular flytekit installation, right?
There was a problem hiding this comment.
hmm? no we need this. this is for compatibility between 3.7/8 and 3.9
There was a problem hiding this comment.
Why? We already have typing_extensions, so we could rely on that for all versions, including 3.9-10.
There was a problem hiding this comment.
From the user side, they may use from typing import Annotated or from typing_extensions import Annotated. Therefore, we need to make sure both of them can work here.
| df = structured_dataset.dataframe | ||
| pq.write_table(df, path, filesystem=get_filesystem(path)) | ||
| local_dir = ctx.file_access.get_random_local_directory() | ||
| local_path = os.path.join(local_dir, f"{0:05}") |
There was a problem hiding this comment.
Can you leave a comment explaining why we need a file named 00000? This is already present in dataset_example.py (I'm not sure why), but that doesn't have a comment either.
There was a problem hiding this comment.
will add a comment... this is only to keep the semantics the same as existing behavior.
|
i think one of the integration tests timed out too, next run will probably be fine. |
Signed-off-by: Kevin Su <pingsutw@apache.org>
3 participants
TL;DR
util.pytest_workflows.pydata_persistence.py