python_wheel_task fails to deploy on DBR < 13.1 using Databricks CLI v0.208.1

[dgarridoa]

Hello,

I am having some troubles deploying a workflow with a python_wheel_task on a cluster with a Databricks Runtime less than 13.1. I followed the suggestions from databricks documentation and the issue #783 without success.

Steps to reproduce the error:

# create basic template
databricks bundle init
# go to project directory
cd <my-project>
# update cluster spark_version from 13.3 to 11.3
sed -i 's/13/11/' resources/*.yml
# add top-level mapping to make it works according to the documentation
echo '\nexperimental:\n  python_wheel_wrapper: true' >> databricks.yml

After running the validation command I got the following message:

databricks bundle validate                                                                                                                                                                       
>>> Error: python wheel tasks with local libraries require compute with DBR 13.1+. Please change your cluster configuration or set experimental 'python_wheel_wrapper' setting to 'true'

Environment

OS: Ubuntu 23.04 x86_64
Kernel: 6.2.0-34-generic
CPU: Intel i7-10700 (16) @ 4.800GHz
GPU: AMD ATI Radeon RX 5500/5500M / Pro 5500M
Memory: 7823MiB / 32005MiB

Databricks CLI v0.208.1
Python 3.9.16

Example

[andrewnester]

@dgarridoa Hi Diego! Could you please provide your full databricks.yml (if possible) and re-run databricks bundle validate with --log-level=DEBUG flag and provide the output?

Thank you!

[andrewnester]

Disregard the previous comment, it appears to be an issue on our side

[andrewnester]

The fix is merged and will be released in the next CLI release next week. In the meantime you can try to use snapshot version from here which already contains the fix
https://github.com/databricks/cli/releases/tag/snapshot

[dgarridoa]

Thanks, it worked.