pyflyte run remote file by ChungYujoyce · Pull Request #1670 · flyteorg/flytekit

ChungYujoyce · 2023-06-02T03:06:53Z

Add remote-file download and delete feature.

pyflyte run --remote https://raw.githubusercontent.com/flyteorg/flytesnacks/master/cookbook/core/flyte_basics/basic_workflow.py my_wf --a 3 --b test

Type

Bug Fix
Feature
Plugin

Are all requirements met?

Complete description

Users can provide a remote workflow file link and Flytekit would download it.
After workflow registration, the file would be removed automatically.

Tracking Issue

NA

Follow-up issue

NA

Signed-off-by: ChungYujoyce <joyce.bhps@gmail.com>

Signed-off-by: Kevin Su <pingsutw@apache.org>

codecov · 2023-06-07T20:39:23Z

Codecov Report

Merging #1670 (ccb6db0) into master (3370a96) will increase coverage by 0.01%.
The diff coverage is 75.00%.

❗ Current head ccb6db0 differs from pull request most recent head 49678b2. Consider uploading reports for the commit 49678b2 to get more accurate results

@@            Coverage Diff             @@
##           master    #1670      +/-   ##
==========================================
+ Coverage   71.03%   71.04%   +0.01%     
==========================================
  Files         336      337       +1     
  Lines       30798    30821      +23     
  Branches     5589     5595       +6     
==========================================
+ Hits        21876    21897      +21     
  Misses       8375     8375              
- Partials      547      549       +2

Impacted Files	Coverage Δ
flytekit/clis/sdk_in_container/build.py	`94.64% <0.00%> (ø)`
flytekit/clis/sdk_in_container/run.py	`85.01% <68.75%> (-0.55%)`	⬇️
flytekit/clis/sdk_in_container/constants.py	`100.00% <100.00%> (ø)`
tests/flytekit/unit/cli/pyflyte/test_run.py	`99.46% <100.00%> (+0.01%)`	⬆️

... and 12 files with indirect coverage changes

Signed-off-by: Kevin Su <pingsutw@apache.org>

kumare3 · 2023-06-10T06:02:33Z

Can you write an example

welcome · 2023-06-13T15:44:00Z

Congrats on merging your first pull request! 🎉

Signed-off-by: ChungYujoyce <joyce.bhps@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org>

* FLYTECTL_CONFIG env var higher precedence, config flag respected in pyflyte package (#1662) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Change flytekit Pytorch, TFJob and MPI plugins to use new kubeflow config (#1627) * upgrade tensorflow plugin to v1 Signed-off-by: Yubo Wang <yubwang@linkedin.com> * minor fix Signed-off-by: Yubo Wang <yubwang@linkedin.com> * fix tests and lints Signed-off-by: Yubo Wang <yubwang@linkedin.com> * move models file into task make backward compatible Signed-off-by: Yubo Wang <yubwang@linkedin.com> Signed-off-by: Yubo Wang <yubwang@linkedin.com> * add code example in README Signed-off-by: Yubo Wang <yubwang@linkedin.com> * bump flyteidl Signed-off-by: Yubo Wang <yubwang@linkedin.com> * add pytorch Signed-off-by: Yubo Wang <yubwang@linkedin.com> * add mpi and fix requirements.txt Signed-off-by: Yubo Wang <yubwang@linkedin.com> * lint and fmt Signed-off-by: Yubo Wang <yubwang@linkedin.com> * Regenerate requirements files using python 3.8 Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> --------- Signed-off-by: Yubo Wang <yubwang@linkedin.com> Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Yubo Wang <yubwang@linkedin.com> Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Root cert should be byte string when loading from caCertFilePath (#1669) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Explicitly set the content type for flyte deck (#1658) * Set content type for flyte deck Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * test Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> * nit Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> * Use protos of new kubeflow.pytorch plugin instead of legacy pytorch plugin (#1678) Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> Co-authored-by: Fabio Grätz <fabiogratz@googlemail.com> * More time info for time line deck (#1680) * more visualization Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * more visualization Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> --------- Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * Add http_proxy to client & Fix deviceflow (#1611) * Add http_proxy to client & Fix deviceflow RB=3890720 Signed-off-by: byhsu <byhsu@linkedin.com> * nit Signed-off-by: byhsu <byhsu@linkedin.com> * lint! Signed-off-by: byhsu <byhsu@linkedin.com> --------- Signed-off-by: byhsu <byhsu@linkedin.com> Co-authored-by: byhsu <byhsu@linkedin.com> * Pass verify flag to all authenticators (#1641) Signed-off-by: byhsu <byhsu@linkedin.com> * feat: Add Auth0/audience support for ClientCredentials flow (#1639) * feat: Add Auth0/audience support for ClientCredentials flow Signed-off-by: tnam <tommy.nam@aiq-index.com> * refactor: Remove unneeded variables & condense code Signed-off-by: tnam <tommy.nam@aiq-index.com> * refactor: Reduce verbosity of code Signed-off-by: tnam <tommy.nam@aiq-index.com> * refactor(chore): Remove unused commented code Signed-off-by: tnam <tommy.nam@aiq-index.com> * fix: Missing comma in input args - authenticator.py 213 Signed-off-by: tnam <tommy.nam@aiq-index.com> * style: Run pre-commit against all files Signed-off-by: tnam <tommy.nam@aiq-index.com> --------- Signed-off-by: tnam <tommy.nam@aiq-index.com> Co-authored-by: tnam <tommy.nam@aiq-index.com> * pyflyte run remote file (#1670) Signed-off-by: ChungYujoyce <joyce.bhps@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org> * upload deck.html only with deck enable (#1693) Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> * Add dask plugin #patch (#1366) * Add dummy task type to test backend plugin Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add docs page Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add dask models Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add function to convert resources Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add tests to `dask` task Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Remove namespace Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update setup.py Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add dask to `plugin/README.md` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add README.md for `dask` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Top level export of `JopPodSpec` and `DaskCluster` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update docs for images Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update README.md Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update models after `flyteidl` change Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update task after `flyteidl` change Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Raise error when less than 1 worker Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update flyteidl to >= 1.3.2 Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update doc requirements Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update doc-requirements.txt Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Re-lock dependencies on linux Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com> * Update dask API docs Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Fix documentation links Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Default optional model constructor arguments to `None` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Refactor `convert_resources_to_resource_model` to `core.resources` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Use `convert_resources_to_resource_model` in `core.node` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Incorporate review feedback Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Lint Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Do not bring dask changes Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Remove readthedocs Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Linting Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> * Force scipy<1.11.0 is on windows Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> --------- Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Yubo Wang <yubwang@linkedin.com> Signed-off-by: eduardo apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Fabio Grätz <fabiogratz@googlemail.com> Signed-off-by: Yicheng-Lu-llll <luyc58576@gmail.com> Signed-off-by: byhsu <byhsu@linkedin.com> Signed-off-by: tnam <tommy.nam@aiq-index.com> Signed-off-by: ChungYujoyce <joyce.bhps@gmail.com> Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Co-authored-by: Yubo Wang <yubowang2019@gmail.com> Co-authored-by: Yubo Wang <yubwang@linkedin.com> Co-authored-by: eduardo apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Fabio M. Graetz, Ph.D <fabiograetz@googlemail.com> Co-authored-by: Fabio Grätz <fabiogratz@googlemail.com> Co-authored-by: Yicheng-Lu-llll <51814063+Yicheng-Lu-llll@users.noreply.github.com> Co-authored-by: ByronHsu <byronhsu1230@gmail.com> Co-authored-by: byhsu <byhsu@linkedin.com> Co-authored-by: TomNam <92072956+PudgyPigeon@users.noreply.github.com> Co-authored-by: tnam <tommy.nam@aiq-index.com> Co-authored-by: ChungYujoyce <46030368+ChungYujoyce@users.noreply.github.com> Co-authored-by: bstadlbauer <11799671+bstadlbauer@users.noreply.github.com>

ChungYujoyce requested review from cosmicBboy, eapolinario, kumare3, pingsutw and wild-endeavor as code owners June 2, 2023 03:06

ChungYujoyce closed this Jun 2, 2023

ChungYujoyce deleted the downloadfile branch June 2, 2023 03:07

ChungYujoyce restored the downloadfile branch June 2, 2023 03:25

ChungYujoyce reopened this Jun 2, 2023

ChungYujoyce added 3 commits June 1, 2023 20:53

add download file

09676a9

Signed-off-by: ChungYujoyce <joyce.bhps@gmail.com>

add unit test for download file

5011b55

Signed-off-by: ChungYujoyce <joyce.bhps@gmail.com>

add remote file download

6717973

Signed-off-by: ChungYujoyce <joyce.bhps@gmail.com>

ChungYujoyce force-pushed the downloadfile branch from 0faca87 to 6717973 Compare June 2, 2023 03:53

pingsutw added 3 commits June 6, 2023 21:58

update

226d790

Signed-off-by: Kevin Su <pingsutw@apache.org>

remove http file

50c5209

Signed-off-by: Kevin Su <pingsutw@apache.org>

fix tests

ad072b8

Signed-off-by: Kevin Su <pingsutw@apache.org>

pingsutw changed the title ~~Downloadfile~~ pyflyte run remote file Jun 7, 2023

nit

8e9c9fb

Signed-off-by: Kevin Su <pingsutw@apache.org>

Merge branch 'master' of github.com:flyteorg/flytekit into downloadfile

ccb6db0

pingsutw previously approved these changes Jun 8, 2023

View reviewed changes

pingsutw marked this pull request as draft June 9, 2023 06:37

update

c6ac046

Signed-off-by: Kevin Su <pingsutw@apache.org>

pingsutw dismissed their stale review via c6ac046 June 10, 2023 03:08

Merge branch 'master' of github.com:flyteorg/flytekit into downloadfile

49678b2

pingsutw marked this pull request as ready for review June 10, 2023 03:09

pingsutw approved these changes Jun 13, 2023

View reviewed changes

pingsutw merged commit c3b0876 into flyteorg:master Jun 13, 2023

eapolinario pushed a commit that referenced this pull request Jun 29, 2023

pyflyte run remote file (#1670)

cbb968c

Signed-off-by: ChungYujoyce <joyce.bhps@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

pyflyte run remote file#1670

pyflyte run remote file#1670
pingsutw merged 10 commits into
flyteorg:masterfrom
ChungYujoyce:downloadfile

ChungYujoyce commented Jun 2, 2023 •

edited by pingsutw

Loading

Uh oh!

codecov Bot commented Jun 7, 2023 •

edited

Loading

Uh oh!

kumare3 commented Jun 10, 2023

Uh oh!

welcome Bot commented Jun 13, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

ChungYujoyce commented Jun 2, 2023 • edited by pingsutw Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Type

Are all requirements met?

Complete description

Tracking Issue

Follow-up issue

Uh oh!

codecov Bot commented Jun 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

kumare3 commented Jun 10, 2023

Uh oh!

welcome Bot commented Jun 13, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ChungYujoyce commented Jun 2, 2023 •

edited by pingsutw

Loading

codecov Bot commented Jun 7, 2023 •

edited

Loading