Create an `airflow upgrade-check` command in 1.10 to ease upgrade path to 2.0.

[ashb]

To make it easier for users to upgrade from 1.10 to 2.0 (when it eventually comes out) we should create a single upgrade-check command in 1.10 that checks the following things. We could also have a mode that makes some of these changes in place (with confirmation from user) to automate it.

Rules

Major breaking changes:

• ConnTypeIsNotNullableRule - Not-nullable conn_type column in connection table (done)
• UniqueConnIdRule - Unique conn_id in connection table Add UniqueConnIdRule rule and unittest #11222
• CutomOperatorUsesMetaclassRule - BaseOperator uses metaclass Create CustomOperatorUsesMetaclassRule to ease upgrade to Airflow 2.0 #11038
• UsingSQLFromBaseHookRule - Remove SQL support in base_hook Create UsingSQLFromBaseHookRule to ease upgrade to Airflow 2.0 #11039
• ChainBetwenDAGAndOperatorNotAllowedRule - Assigning task to a DAG using bitwise shift (bit-shift) operators are no longer supported
• AirflowMacroPluginRemovedRule - Removal of airflow.AirflowMacroPlugin class
• NoAdditionalArgsInOperatorsRule - Additional arguments passed to BaseOperator cause an exception Create NoAdditionalArgsInOperatorsRule to ease upgrade to Airflow 2.0 #11042
• MesosExecutorRemovedRule - Removal of Mesos Executor

Config related changes:

• HostnameCallableRule - Unify hostname_callable option in core section Create HostnameCallableRule to ease upgrade to Airflow 2.0 #11044
• StatNameHandlerNotSupportedRule - Drop plugin support for stat_name_handler
• LoggingConfigurationRule - Logging configuration has been moved to new section
• NoGCPServiceAccountKeyInConfigRule - Remove gcp_service_account_keys option in airflow.cfg file
• FernetEnabledRule - Fernet is enabled by default
• KubernetesWorkerAnnotationsRule - Changes to propagating Kubernetes worker annotations
• LegacyUIDeprecatedRule - Deprecate legacy UI in favor of FAB RBAC UI
• TaskHandlersMovedRule - GCSTaskHandler has been moved, WasbTaskHandler has been moved, StackdriverTaskHandler has been moved , S3TaskHandler has been moved, ElasticsearchTaskHandler has been moved, CloudwatchTaskHandler has been moved
• SendGridMovedRule - SendGrid emailer has been moved
• CustomExecutorsRequireFullPathRule - Custom executors is loaded using full import path

Import changes:

• ImportChangesRule - uses a map old_operator_name -> list of possible problems so we can create a single DagBag and scan all used operators and raise information about changes. It should also suggest what providers packages users should use.

How to guide

To implement a new rule we had to create a class that inherits from airflow.upgrade.rules.base_rule.BaseRule. It will be auto-registered and used by airflow upgrade-check command. The custom rule class has to have title, description properties and should implement check method which returns a list of error messages in case of incompatibility.

For example:

airflow/airflow/upgrade/rules/conn_type_is_not_nullable.py

Lines 25 to 42 in ea36166

	class ConnTypeIsNotNullableRule(BaseRule):
	title = "Connection.conn_type is not nullable"
	description = """\
	The `conn_type` column in the `connection` table must contain content. Previously, this rule was \
	enforced by application logic, but was not enforced by the database schema.
	If you made any modifications to the table directly, make sure you don't have null in the conn_type column.\
	"""
	@provide_session
	def check(self, session=None):
	invalid_connections = session.query(Connection).filter(Connection.conn_type.is_(None))
	return (
	'Connection<id={}", conn_id={}> have empty conn_type field.'.format(conn.id, conn.conn_id)
	for conn in invalid_connections
	)

Remeber to open the PR against v1-10-test branch.

[turbaszek]

I screened the UPDATING.md for "breaking" changes that can be check / communicated. I selected those explicitly saying "breaking" or sounding like that. In general, we can split them into three categories:

• python code changes (related to DAGs etc)
• config changes
• cli changes (related to scripts etc)

I ended up with a list of 59 of changes:

 'BaseOperator uses metaclass',
 'Not-nullable conn_type column in connection table',
 'Change signature of BigQueryGetDatasetTablesOperator',
 'Unify `hostname_callable` option in `core` section',
 'Changes in BigQueryHook',
 'Use project_id argument consistently across GCP hooks and operators',
 'GCSUploadSessionCompleteSensor signature change',
 'Remove SQL support in base_hook',
 'Changes to SalesforceHook',
 'Rename parameter name in PinotAdminHook.create_segment',
 'Rename parameter name in HiveMetastoreHook.get_partitions',
 'Remove unnecessary parameter in FTPHook.list_directory',
 'Remove unnecessary parameter in PostgresHook function copy_expert',
 'Change parameter name in OpsgenieAlertOperator',
 'Assigning task to a DAG using bitwise shift (bit-shift) operators are no longer supported',
 'Custom executors is loaded using full import path',
 'Removed sub-package imports from `airflow/__init__.py`',
 'Drop plugin support for stat_name_handler',
 'Standardize handling http exception in BigQuery',
 'Remove airflow.utils.file.TemporaryDirectory',
 'Chain and cross_downstream moved from helpers to BaseOperator',
 'Change python3 as Dataflow Hooks/Operators default interpreter',
 'Logging configuration has been moved to new section',
 'Simplification of CLI commands',
 'Remove gcp_service_account_keys option in airflow.cfg file',
 'Removal of airflow.AirflowMacroPlugin class',
 ' Change default aws_conn_id in EMR operators',
 'Removal of redirect_stdout, redirect_stderr',
 'Changes to SQLSensor',
 'Idempotency in BigQuery operators',
 'AWS Batch Operator',
 'Additional arguments passed to BaseOperator cause an exception',
 'Simplification of the TriggerDagRunOperator',
 'Changes in Google Cloud Platform related hooks',
 'Fernet is enabled by default',
 'Changes to Google PubSub Operators, Hook and Sensor',
 'Removed Hipchat integration',
 'The gcp_conn_id parameter in GKEPodOperator is required',
 'Changes to propagating Kubernetes worker annotations',
 'Remove provide_context',
 'Variables removed from the task instance context',
 'Moved provide_gcp_credential_file decorator to GoogleBaseHook',
 'Changes to S3Hook',
 'Changes to Google Transfer Operator',
 'Changes in  Google Cloud Transfer Hook',
 'CLI reorganization',
 'Removal of Mesos Executor',
 'Changes to SalesforceHook',
 'HTTPHook verify default value changed from False to True.',
 'Changes to GoogleCloudStorageHook',
 'Changes to CloudantHook',
 'Unify default conn_id for Google Cloud Platform',
 'Changes to sensor imports',
 'Deprecate legacy UI in favor of FAB RBAC UI',
 'CLI Changes',
 'Unification of `do_xcom_push` flag',
 'Changes to Dataproc related Operators',
 'Changes to skipping behaviour of LatestOnlyOperator',
 'Change default snowflake_conn_id for Snowflake hook and operators'

I like the idea proposed by @mik-laj in #9467 with defining Rules and that's something we should follow I think.

Questions that should be addressed before we move on:

• do we plan to only "warn" or should we consider also "auto-refactor"?
• if there's backward compatibility do we want to include such check?

If we are ok with the approach proposed by Kamil I can evaluate those changes and create issues so the work can be split.

[turbaszek]

@ash @mik-laj @potiuk @kaxil WDYT?

[mik-laj]

@turbaszek Can you create Google Docs about it? Without a broader context, the titles of changes are of little help.

I think we should now prepare a plan that will describe how we can detect breaking change and determine what to do when we detect them.

# Move operators to providers package (Rule 1)

We should load all DAG using DAGBag and check whether any deprecated operators are used. When it is detected, we can make changes automatically using Bowler.

This rule resolves the following entries from UPDATING.md
-....

A lot of changes are backward compatible, and if not, we also need to detect and notify them.

# Detectt provide_context in Python Operaator

We should check ....when this happens, we should inform the user that in Airflow this code may cause problems.

This rule resolves the following entries from UPDATING.md
- Remove provide_context

[potiuk]

I think some of the changes we can apply easily and safely. I think for example all the deprecation notices can be fixed rather easily - we already have information about it (we test all the depracations - I already used it in backport packages) - so we can super-easily and safely apply those in automated way. Similarly some of the changes above (rename parameter names etc.). I would really like to provide such tool to the users where it is no-brainer and can be automated.

For most other changes I think we should simply detect potential problems and flag them - providing helpful hint how the problem could be fixed.

And I agree GDOc about it might be better than issues/splitting them now. We can even split the work among ourselves and make recommendations individually for each of those - happy to share this with the rest. Then we can review them and turn into issues.

[kaxil]

We should definitely create the rules similar to what Kamil did in his PR and additionally have a flag that automatically fixes (where possible).

Google Doc, Confluence (https://cwiki.apache.org/confluence/display/AIRFLOW/) is fine if we want to review and add comments.

[ashb]

Gdoc fine for building up, but anything we point end users at should be in our official docs.

A mode to doc things automatically is desirable, but probably not on by default - as much as I wish everyone used git, I know some people out there don't, so we shouldn't change their files without confirmation (perhaps show a diff then ask y/n?)

[turbaszek]

so we shouldn't change their files without confirmation (perhaps show a diff then ask y/n?)

Bowler has this option

[turbaszek]

I've drafted this doc:
https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit?usp=sharing

The notes from UPDATING.md are split into few general groups (some are present in few places). I suppose that not all entries are important to users / will impact their Airflows.