[AIRFLOW-3731] Constrain mysqlclient to <1.4

[Fokko]

To maintain Python2 compatibility: https://pypi.org/project/mysqlclient/#history

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-3731] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-3731
  • In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-3731], code changes always need a Jira issue.

Description

• Here are some details about my PR, including screenshots of any UI changes:

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
  • All the public functions and the classes in the PR contain docstrings that explain what it does

Code Quality

• Passes flake8

[Fokko]

@kaxil PTAL

[Fokko]

@potiuk, In my opinion, this stuff needs to be fixed first before we move to a multi-layer image of Airflow. In this case, the image would still be working, but the master would be broken when using Python2 and MySQL 😢

[codecov-io]

Codecov Report

Merging #4558 into master will increase coverage by 0.29%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master   #4558      +/-   ##
=========================================
+ Coverage   73.81%   74.1%   +0.29%     
=========================================
  Files         421     421              
  Lines       27662   27662              
=========================================
+ Hits        20418   20499      +81     
+ Misses       7244    7163      -81

Impacted Files	Coverage Δ
airflow/hooks/dbapi_hook.py	79.03% <0%> (+0.8%)	⬆️
airflow/models/connection.py	63.29% <0%> (+1.26%)	⬆️
airflow/hooks/hive_hooks.py	75.26% <0%> (+1.84%)	⬆️
airflow/utils/sqlalchemy.py	81.81% <0%> (+4.54%)	⬆️
airflow/operators/mysql_operator.py	100% <0%> (+100%)	⬆️
airflow/operators/mysql_to_hive.py	100% <0%> (+100%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 233f6c7...9cc81e1. Read the comment docs.

[kaxil]

Aren't they just dropping python 3.4 support?

[Fokko]

They changed the public interface of the Connection, I've opened up an issue: PyMySQL/mysqlclient#325

[TylerBrock]

I think we need to move forward. Is there a way to install both and use <1.4 for python2 and >=1.4 for python3? I'm not sure if @kaxil is correct but if they are only removing support for python 3.4 then there shouldn't be an issue here.

I would be happy to take a stab at refactoring airflow's MySqlHook for 1.4 if nobody else is working on it.

[potiuk]

Yes. It's possible. In setup.py of v1-10-test we have:

    'sphinx>=2.1.2;python_version>="3.0"',
    'sphinx==1.8.5;python_version<"3.0"',

[OmerJog]

@TylerBrock an attempt has be done here #5430 but wan't completed.

[methane]

I removed __exit__ from mysqlclient because it was not many people expects, including Airflow.

airflow/tests/operators/test_operators.py

Lines 354 to 363 in 7bebe8d

	from airflow.hooks.mysql_hook import MySqlHook
	with MySqlHook().get_conn() as cur:
	cur.execute('''
	CREATE TABLE IF NOT EXISTS baby_names (
	org_year integer(4),
	baby_name VARCHAR(25),
	rate FLOAT(7,6),
	sex VARCHAR(4)
	)
	''')

airflow/tests/operators/test_operators.py

Lines 370 to 371 in 7bebe8d

	with MySqlHook().get_conn() as cur:
	cur.execute("DROP TABLE IF EXISTS baby_names CASCADE;")

The Airflow expect __exit__ close the connection. But it didn't. It did commit or rollback.
So there are connection leaks in your code.

Since this interface is different from many people expects, I am changing it:

1. deprecate it.
2. remove it. (now)
3. add __exit__ which closes the connection. (in near future).