Support generating SQL script for upgrades

[ephraimbuddy]

This PR attempts to add support for generating sql scripts for upgrade.
Example command:
airflow db upgrade --revision-range e8d98d8ss99:78daisdu38d
airflow db upgrade --range 2.0.0:2.2.3

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

I think this is quite needed. I actually find Alembic much worse and less intuitive than other solution (say Django migrations). So some tooling and possibly better instructions for those who attempt to add migrations is most welcome!

[uranusjr]

LGTM except the server_default thing I don’t really understand. (SQLAlchemy is so complicated…)

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[ephraimbuddy]

I will make a separate PR for SQLite & MSSQL. SQLite needs the copy_from arg for every batch operation. https://alembic.sqlalchemy.org/en/latest/batch.html#working-in-offline-mode
MSSQL custom codes for getting datetime column is an issue for offline mode

[ephraimbuddy]

I changed this to use version instead of revision.
Now we can do airflow db upgrade -v 2.0.0:2.2.3

cc: @uranusjr @ashb

[ashb]

Few minor questions, looks good though.

[ephraimbuddy]

Using github rebase button just messed my local branch up while fixing conflicts. It was a lot

[kaxil]

Can we also add user-facing docs about it somewhere plz

[kaxil]

We should have a pre-commit to verify that a tuple containing the current Airflow version is added to REVISION_HEADS_MAP to avoid forgetting it.

[ephraimbuddy]

It looks like a pre-commit won't be possible in this case as we only update the migration head when there's a new airflow release.

[kaxil]

How about something as simple as:

from airflow.utils.db import REVISION_HEADS_MAP
from airflow.version import version as airflow_version
from packaging.version import Version


if not Version(airflow_version).is_prerelease and airflow_version not in REVISION_HEADS_MAP:
    raise Exception(f"Airflow version {airflow_version} is not in the revision map {REVISION_HEADS_MAP}")

[dstandish]

i was also thinking about this issue. i saw that we had a table in the docs which contains this mapping. so what i was thinking to do was store that table in yaml form, and it could be used to generate the docs table and to support CLI commands which need to map version to revision, and that's what i do in #21601

[ephraimbuddy]

Looks good, I think the challenge would be to get the migration heads of the versions and discard every other migration in-between when going from version to version. It's still necessary to have a pre-commit or a ci job that makes sure we have the migration head updated before a release

[dstandish]

yes that's right.

[kaxil]

Approving it so it is not blocked for adding pre-commit - that can be a follow-up PR.