Improve handling of backwards compat for airflow.io#36199
Conversation
Older providers do not have a get_fs method that takes storage_options as arguments. If we encounter such provider and storage_options are passed we should error out instead if silently ignoring.
potiuk
left a comment
potiuk
left a comment
There was a problem hiding this comment.
Much more complete. @ephraimbuddy - if we have a chance to get it into RC2 still (or RC3 is on the horizon) this one is very good candidate.
| get_fs("file", storage_options={"foo": "bar"}) | ||
|
|
||
| # Reset the cache to avoid side effects | ||
| _register_filesystems.cache_clear() |
There was a problem hiding this comment.
Nice. Recalls the recent cache clearing issue :)
There was a problem hiding this comment.
But I think It would be better to use try/finally or - better - pytest fixture WITH try/finally. This way the cache will be cleared even if the test fails @bolkedebruin
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
I have already released RC2. Maybe RC3 but tests are failing |
|
Anyone working on this or should we leave it for 2.8.1? |
|
Im working on it but I'm also not feeling so well. So I'm a bit slow. |
|
Give me an hour or so |
Sorry about that. Take it easy |
…ompat' into improve_backwards_compat
Older providers do not have a get_fs method that takes storage_options as arguments. If we encounter such provider and storage_options are passed we should error out instead if silently ignoring. Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> (cherry picked from commit 6c94ddf)
ephraimbuddy
added
the
changelog:skip
5 participants
Older providers do not have a get_fs method that takes storage_options as arguments. If we encounter such provider and storage_options are passed we should error out instead if silently ignoring.
@potiuk
@ephraimbuddy if we do a rc3 for 2.8 this would be great to have in.
Closes: #36187
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.