Separate time partition by wild-endeavor · Pull Request #2114 · flyteorg/flytekit

wild-endeavor · 2024-01-18T02:16:51Z

This uses the separate time partition message added in flyteorg/flyte#4737.

Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>

…hat take dataclasses (#2013) Signed-off-by: jason.lai <jasonlai1218@gmail.com>

Signed-off-by: troychiu <y.troychiu@gmail.com>

* Error when multiple configuration plugins are installed Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com> * Adds semi-colon Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com> --------- Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>

Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future Outlier <eric901201@gmai.com>

Signed-off-by: pryce-turner <pryce.turner@gmail.com>

Signed-off-by: Kevin Su <pingsutw@apache.org>

Signed-off-by: Hongxin Liang <honnix@users.noreply.github.com>

Signed-off-by: nikki everett <nikki@union.ai>

* added _cmd_prefix handling Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed typing imports Signed-off-by: Daniel Rammer <daniel@union.ai> * add get_config Signed-off-by: Kevin Su <pingsutw@apache.org> * updating get_config to use underlying functions config Signed-off-by: Daniel Rammer <daniel@union.ai> --------- Signed-off-by: Daniel Rammer <daniel@union.ai> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org>

Signed-off-by: DouEnergy <douenergy@protonmail.com>

Signed-off-by: Austin Liu <austin362667@gmail.com>

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

* Accelerator docs Signed-off-by: Ketan Umare <kumare3@users.noreply.github.com> * updating docs Signed-off-by: Ketan Umare <kumare3@users.noreply.github.com> * Updated docs Signed-off-by: Ketan Umare <kumare3@users.noreply.github.com> * updated docs Signed-off-by: Ketan Umare <kumare3@users.noreply.github.com> * more docs update Signed-off-by: Ketan Umare <kumare3@users.noreply.github.com> * updated Signed-off-by: Ketan Umare <kumare3@users.noreply.github.com> * updated docs Signed-off-by: Ketan Umare <kumare3@users.noreply.github.com> --------- Signed-off-by: Ketan Umare <kumare3@users.noreply.github.com> Co-authored-by: Ketan Umare <kumare3@users.noreply.github.com>

* Add support for running remote-tasks and remote-workflows Signed-off-by: Ketan Umare <kumare3@users.noreply.github.com> * updated Signed-off-by: Ketan Umare <kumare3@users.noreply.github.com> --------- Signed-off-by: Ketan Umare <kumare3@users.noreply.github.com> Co-authored-by: Ketan Umare <kumare3@users.noreply.github.com>

Signed-off-by: Kevin Su <pingsutw@apache.org>

Bumps [fonttools](https://github.com/fonttools/fonttools) from 4.41.1 to 4.43.0. - [Release notes](https://github.com/fonttools/fonttools/releases) - [Changelog](https://github.com/fonttools/fonttools/blob/main/NEWS.rst) - [Commits](fonttools/fonttools@4.41.1...4.43.0) --- updated-dependencies: - dependency-name: fonttools dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* add monodocs index page in flytekit Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * updates Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> --------- Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

* add monodocs redirects to the flytekit readthedocs project Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * update requirements Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * add redirects enabling flag Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * update Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> --------- Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future Outlier <eric901201@gmai.com>

Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.2 to 3.1.3. - [Release notes](https://github.com/pallets/jinja/releases) - [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst) - [Commits](pallets/jinja@3.1.2...3.1.3) --- updated-dependencies: - dependency-name: jinja2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future Outlier <eric901201@gmai.com>

) Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.2 to 3.1.3. - [Release notes](https://github.com/pallets/jinja/releases) - [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst) - [Commits](pallets/jinja@3.1.2...3.1.3) --- updated-dependencies: - dependency-name: jinja2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.2 to 3.1.3. - [Release notes](https://github.com/pallets/jinja/releases) - [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst) - [Commits](pallets/jinja@3.1.2...3.1.3) --- updated-dependencies: - dependency-name: jinja2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future Outlier <eric901201@gmai.com>

Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>

Signed-off-by: Kevin Su <pingsutw@gmail.com>

* Use Latest FlyteIdl in all Dockerfile Signed-off-by: Future Outlier <eric901201@gmai.com> * lint Signed-off-by: Future Outlier <eric901201@gmai.com> * Revert all dockerfile except dev version Signed-off-by: Future-Outlier <eric901201@gmai.com> * remove sqlalchemy dockerfile Signed-off-by: Future-Outlier <eric901201@gmai.com> --------- Signed-off-by: Future Outlier <eric901201@gmai.com> Signed-off-by: Future-Outlier <eric901201@gmai.com> Co-authored-by: Future Outlier <eric901201@gmai.com>

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Fix typos and run spellcheck in CI Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove --interactive Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove CI check since it's run in pre-commit checks (lint make target) Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Fix typo in accelerators.py Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> --------- Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Signed-off-by: Kevin Su <pingsutw@gmail.com>

…ct is specified correctly, raise unbound errors, fix tests Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

codecov · 2024-01-19T22:50:05Z

Codecov Report

Attention: 660 lines in your changes are missing coverage. Please review.

Comparison is base (d3e1884) 78.73% compared to head (77004bd) 53.54%.
Report is 53 commits behind head on artifacts.

Files	Patch %	Lines
flytekit/core/artifact.py	21.36%	265 Missing ⚠️
flytekit/trigger.py	0.00%	79 Missing ⚠️
flytekit/remote/remote.py	15.85%	67 Missing and 2 partials ⚠️
flytekit/core/type_engine.py	5.12%	37 Missing ⚠️
flytekit/types/directory/types.py	8.57%	32 Missing ⚠️
flytekit/image_spec/image_spec.py	29.03%	22 Missing ⚠️
flytekit/extend/backend/base_agent.py	37.93%	16 Missing and 2 partials ⚠️
flytekit/loggers.py	67.27%	8 Missing and 10 partials ⚠️
flytekit/core/interface.py	51.51%	10 Missing and 6 partials ⚠️
flytekit/core/workflow.py	53.33%	13 Missing and 1 partial ⚠️
... and 26 more

Additional details and impacted files

@@              Coverage Diff               @@
##           artifacts    #2114       +/-   ##
==============================================
- Coverage      78.73%   53.54%   -25.20%     
==============================================
  Files            173      178        +5     
  Lines          16815    17582      +767     
  Branches        2310     3650     +1340     
==============================================
- Hits           13240     9414     -3826     
- Misses          3004     7737     +4733     
+ Partials         571      431      -140

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

* temp Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * stuff Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * temp Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * scaffolding areas mostly identified Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * add artifact to upload request Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * remove an unnecessary line in workflow Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * finish adding examples use cases maybe Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * add project/dom to get query Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * add from flyte idl Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * add project domain to as query Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * add condition in parameter to flyte idl Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * test stuff * Remove artifactID from literal oneof, add to metadata (#2) * Triggers (#6) * Minor changes to get time series example working #8 Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * switch channel (#10) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * fix tests ignore - pr into other pr (#1858) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Artf/update idl ux (#1920) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Artf/trigger (#1948) * Add triggers * Remove bind_partition_time and just assume users won't use that. It's just time_partition in the normal call function now. Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * remove the now deleted artifact spec (#1984) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Literal metadata model update (#2089) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Separate time partition (#2114) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Split service code (#2136) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * remove empty files Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * add noneness check to metadata and add test Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * remove sandbox test for now Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Artf/cleanup (#2158) * add a test Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * try updates Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> --------- Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Use python 3.9 to run make doc-requirements.txt Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * reasons not msg Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> --------- Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* temp Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * stuff Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * temp Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * scaffolding areas mostly identified Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * add artifact to upload request Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * remove an unnecessary line in workflow Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * finish adding examples use cases maybe Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * add project/dom to get query Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * add from flyte idl Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * add project domain to as query Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * add condition in parameter to flyte idl Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * test stuff * Remove artifactID from literal oneof, add to metadata (#2) * Triggers (#6) * Minor changes to get time series example working #8 Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * switch channel (#10) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * fix tests ignore - pr into other pr (#1858) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Artf/update idl ux (#1920) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Artf/trigger (#1948) * Add triggers * Remove bind_partition_time and just assume users won't use that. It's just time_partition in the normal call function now. Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * remove the now deleted artifact spec (#1984) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Literal metadata model update (#2089) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Separate time partition (#2114) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Split service code (#2136) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * remove empty files Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * add noneness check to metadata and add test Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * remove sandbox test for now Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Artf/cleanup (#2158) * add a test Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * try updates Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> --------- Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Use python 3.9 to run make doc-requirements.txt Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * reasons not msg Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> --------- Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

thomasjpfan and others added 30 commits December 22, 2023 13:17

Allows Secret groups to be optional and configurable (#2062)

68430db

Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>

[Core feature] Allow dictionaries to be passed to execute workflows t…

eab12e6

…hat take dataclasses (#2013) Signed-off-by: jason.lai <jasonlai1218@gmail.com>

fix context usage (#2070)

0e8abd9

Signed-off-by: troychiu <y.troychiu@gmail.com>

test (#2077)

452a2ec

Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future Outlier <eric901201@gmai.com>

Generalized subproc_execute and added tests (#2072)

d1ce05b

Signed-off-by: pryce-turner <pryce.turner@gmail.com>

Improve error handling for local execution (#2068)

76ebae2

Signed-off-by: Kevin Su <pingsutw@apache.org>

Download specific files in FlyteDirectory (#2059)

6f613e7

Signed-off-by: Kevin Su <pingsutw@apache.org>

Constraint the full fsspec family (#2087)

9bf7afa

Signed-off-by: Hongxin Liang <honnix@users.noreply.github.com>

Update default template used by pyflyte init (#2041)

5841a1e

Signed-off-by: nikki everett <nikki@union.ai>

remove unnecessary newline in plugin readme (#2090)

9c45cb2

Signed-off-by: DouEnergy <douenergy@protonmail.com>

add fs path exists checking (fix flyteorg/flyte#4470) (#2079)

f22cec2

Signed-off-by: Austin Liu <austin362667@gmail.com>

add monodocs redirect banner (#2091)

b5dbd85

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

Print log links in local execution (#2097)

dc35e7f

Signed-off-by: Kevin Su <pingsutw@apache.org>

add monodocs index page in flytekit (#2100)

d451ee2

* add monodocs index page in flytekit Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * updates Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> --------- Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>

Local Sync Agent (#2085)

ed91dd1

Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future Outlier <eric901201@gmai.com>

Enhance Airflow Plugin (#2099)

76f6db2

Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future Outlier <eric901201@gmai.com>

init (#2078)

a3a640a

Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future Outlier <eric901201@gmai.com>

Fix Flyin Plugin VSCode download bug (#1991)

ccc515e

Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future Outlier <eric901201@gmai.com>

Adds conda support to envd plugin (#2020)

892b474

Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>

Fix is_deferrable in airflow agent (#2109)

80ca660

Signed-off-by: Kevin Su <pingsutw@gmail.com>

Future-Outlier and others added 3 commits January 17, 2024 18:19

Changes default destination dir to be "working dir" (#2108)

7996c2e

use separate time partition in idl change flyteorg/flyte#4737

692b78c

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

wild-endeavor requested review from cosmicBboy, eapolinario, kumare3 and pingsutw as code owners January 18, 2024 02:16

wild-endeavor and others added 9 commits January 17, 2024 18:17

nit

0e66bc4

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

remove the constant as it's no longer needed

2f85ed0

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

Add @lru_cache to ImageBuildEngine.build (#2112)

6279b81

Signed-off-by: Kevin Su <pingsutw@gmail.com>

remove tag from detect_artifact, handle the case where just an artifa…

4b232f1

…ct is specified correctly, raise unbound errors, fix tests Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

try adding git path for pip

7677417

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

Merge remote-tracking branch 'origin/master' into artf/separate-time

ce58a8d

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

add clarifying comment that came up in testing

78d22e3

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

revert

77004bd

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

wild-endeavor merged commit 63b7b29 into artifacts Jan 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Separate time partition#2114

Separate time partition#2114
wild-endeavor merged 42 commits into
artifactsfrom
artf/separate-time

wild-endeavor commented Jan 18, 2024

Uh oh!

codecov Bot commented Jan 19, 2024 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

Uh oh!

Conversation

wild-endeavor commented Jan 18, 2024

Uh oh!

codecov Bot commented Jan 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

codecov Bot commented Jan 19, 2024 •

edited

Loading