Allow annotated FlyteFile as task input argument

[AdrianoKF]

TL;DR

This PR allows annotated FlyteFiles to be used as input arguments of a task, e.g., when calculating a custom hash using Annotated[FlyteFile, HashMethod(...)].

See the discussion on Slack for additional details.

Type

• Bug Fix
• Feature
• Plugin

Are all requirements met?

• Code completed
• Smoke tested
• Unit tests added
• Code documentation added
• Any pending items have an associated Issue

Complete description

Using an annotated FlyteFile type as an input to a task was previously impossible due to an exception being raised in FlyteFilePathTransformer.to_python_value.

This MR applies the fix previously used in FlyteFilePathTransformer.to_literal to permit using annotated FlyteFiles as either inputs or outputs of a task (see MR #1544), by unwrapping the origin type of the annotated type.

Tracking Issue

flyteorg/flyte#3424

Follow-up issue

NA

[pingsutw]

LGTM overall, thanks

[pingsutw]

we do this all the place in the type_engine.py.

flytekit/flytekit/core/type_engine.py

Lines 737 to 748 in a1636a8

	if get_origin(python_type) is Annotated:
	python_type = get_args(python_type)[0]
	if python_type in cls._REGISTRY:
	return cls._REGISTRY[python_type]
	# Step 2
	if hasattr(python_type, "__origin__"):
	# Handling of annotated generics, eg:
	# Annotated[typing.List[int], 'foo']
	if get_origin(python_type) is Annotated:
	return cls.get_transformer(get_args(python_type)[0])

could you move this function to type_engine.py, and import it instead.

[AdrianoKF]

Moved the function there and refactored type_engine.py quite a bit in 093536f. This should also make the handling of annotated types there more uniform (and added a few tests).

Is this what you had in mind?

[codecov]

Codecov Report

Merging #1632 (34600b0) into master (ba70f46) will increase coverage by 0.00%.
The diff coverage is 86.84%.

@@           Coverage Diff           @@
##           master    #1632   +/-   ##
=======================================
  Coverage   71.02%   71.02%           
=======================================
  Files         336      336           
  Lines       30724    30736   +12     
  Branches     5567     5566    -1     
=======================================
+ Hits        21821    21831   +10     
- Misses       8360     8361    +1     
- Partials      543      544    +1     

Impacted Files	Coverage Δ
flytekit/types/file/file.py	54.54% <66.66%> (+0.32%)	⬆️
flytekit/core/type_engine.py	61.15% <80.95%> (-0.14%)	⬇️
tests/flytekit/unit/core/test_flyte_file.py	88.81% <100.00%> (+0.17%)	⬆️
tests/flytekit/unit/core/test_type_engine.py	98.66% <100.00%> (+<0.01%)	⬆️

... and 1 file with indirect coverage changes

[pingsutw]

Thanks, LGTM. would you resolve branch conflict?

[AdrianoKF]

Thanks, LGTM. would you resolve branch conflict?

Sorry, was sick for the past days, just merged upstream master and resolved the conflicts.

[pingsutw]

@AdrianoKF hope you feel better.

[AdrianoKF]

Merged master again and fixed the test failures on Python 3.8 in 34600b0.

The remaining failing test on 3.11/Windows seems to be unrelated to my changes and I'm not familiar enough with that part of the code base to suggest a fix - anything I should do about that, @pingsutw?

[pingsutw]

@AdrianoKF never mind. That's flaky test.