[BUG] `databricks labs upgrade` fails when dependencies used in install script have breaking changes

[JCZuurmond]

Describe the issue

The databricks labs upgrade command fails when dependencies used in install script have breaking changes.

See the following issues for ucx reporting problems related to this:

• [BUG]: Unable to update UCX/reinstall UCX /run UCX commands in Databricks CLI databrickslabs/ucx#2223
• [BUG]: Not able to run databricks labs install ucx databrickslabs/ucx#2220
• [BUG]: cannot import name 'with_user_agent_extra' from 'databricks.sdk.core' databrickslabs/ucx#2102

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

1. Install older ucx version: databricks labs install ucx@v0.23.1
2. Upgrade to latest ucx version: databricks labs upgrade ucx

Expected Behavior

Upgrade should not fail

Actual Behavior

Upgrade fails due to missing dependencies.

OS and CLI version

Please include the version of the CLI (eg: v0.1.2) and the operating system (eg: windows). You can run databricks --version to get the version of your Databricks CLI

Is this a regression?

No

Debug Logs

Debug log for databricks --debug labs upgrade ucx

~/github/databrickslabs/ucx fix/install-deployment *2 ❯ databricks --debug labs upgrade ucx                                                                                                                      ▼ Py ucx 11:59:40
11:59:43  INFO start pid=46827 version=0.216.0 args="databricks, --debug, labs, upgrade, ucx"
11:59:43 DEBUG Latest ucx version is: v0.29.0 pid=46827
11:59:43 DEBUG Reading labs.yml@v0.29.0 from databrickslabs/ucx pid=46827
⣾ Cleaning up previous installation if necessary11:59:43 DEBUG Downloading zipball for v0.29.0 from databrickslabs/ucx pid=46827
⣷ Downloading and unpacking zipball for v0.29.011:59:45 DEBUG Unpacking zipball to: /Users/user/.databricks/labs/ucx/lib pid=46827
11:59:45 DEBUG Writing installed version info to: /Users/user/.databricks/labs/ucx/state/version.json pid=46827
11:59:45 DEBUG Launching installer script src/databricks/labs/ucx/install.py in /Users/user/.databricks/labs/ucx/lib pid=46827
11:59:45 DEBUG Loading login configuration from: /Users/user/.databricks/labs/ucx/config/login.json pid=46827
11:59:45 DEBUG Using workspace-level login profile: DEFAULT pid=46827
11:59:45 DEBUG Loading DEFAULT profile from /Users/user/.databrickscfg pid=46827 sdk=true
11:59:45 DEBUG Resolved login: Config: host=adb-REDACTED.azuredatabricks.net, cluster_id=REDACTED, account_id=REDACTED, profile=DEFAULT, config_file=/user/.databrickscfg. Env: DATABRICKS_HOST, DATABRICKS_CLUSTER_ID, DATABRICKS_ACCOUNT_ID pid=46827 sdk=true
11:59:45 DEBUG Passing down environment variables: DATABRICKS_ACCOUNT_ID, DATABRICKS_AUTH_TYPE, DATABRICKS_HOST, DATABRICKS_CLUSTER_ID pid=46827
11:59:45 DEBUG starting: /Users/user/.databricks/labs/ucx/state/venv/bin/python3 src/databricks/labs/ucx/install.py pid=46827
Traceback (most recent call last):
  File "/Users/user/.databricks/labs/ucx/lib/src/databricks/labs/ucx/install.py", line 27, in <module>
    from databricks.labs.lsql.dashboards import DashboardMetadata, Dashboards
ModuleNotFoundError: No module named 'databricks.labs.lsql.dashboards'
Error: installer: exit status 1
11:59:46 ERROR failed execution pid=46827 exit_code=1 error="installer: exit status 1"