PendingRollbackError: Can't reconnect until invalid transaction is rolled back. Please rollback() fully before proceeding (Background on this error)

[yonus]

Apache Airflow version

3.1.7

If "Other Airflow 3 version" selected, which one?

No response

What happened?

I deployed Airflow 3.1.7 using Helm chart 1.19.0.
After deployment, everything works correctly. However, after some time (not immediately), accessing the Airflow UI results in:
Internal Server Error (HTTP 500)
Restarting the api-server pods immediately resolves the issue and the UI becomes accessible again.
After some time, the issue reoccurs.

When I investigated logs of api server, I see following error logs:
PendingRollbackError: Can't reconnect until invalid transaction is rolled back. Please rollback() fully before proceeding (Background on this error)

This behavior is reproducible.

What you think should happen instead?

Airflow UI should remain accessible without requiring manual restart of API server pods.

How to reproduce

Deploy Airflow 3.1.7 with Helm chart 1.19.0
Access UI successfully
Wait for some time (hours)
Try to access UI again
Observe HTTP 500 Internal Server Error
Restart api-server pods
UI works again

Operating System

ubuntu

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==9.21.0
apache-airflow-providers-cncf-kubernetes
apache-airflow-providers-common-compat>=1.13.0
apache-airflow-providers-common-sql>=1.31.0
apache-airflow-providers-airbyte==5.3.2
apache-airflow-providers-slack==9.6.2
apache-airflow-providers-trino==6.4.2
pytz>=2023.3

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[shashihoushengqia]

I also encountered the same issue. The solution was to restart the apiServer every few hours, and this was for version 3.1.7.

[CincyBC]

I have encountered this issue on 3.1.6 (python 3.12, FAB provider v3.1.1) as well.

Full error trace:
026-03-01T14:59:39.382093Z [error ] Exception on /login/ [POST] [airflow.providers.fab.www.app] loc=app.py:1744 Traceback (most recent call last): File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2529, in wsgi_app response = self.full_dispatch_request() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1825, in full_dispatch_request rv = self.handle_user_exception(e) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1823, in full_dispatch_request rv = self.dispatch_request() ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1799, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/flask_appbuilder/security/decorators.py", line 36, in wrapped_view response = make_response(view(*args, **kwargs)) ^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/flask_appbuilder/security/views.py", line 590, in login user = self.appbuilder.sm.auth_user_db( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1967, in auth_user_db user = self.find_user(username=username) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1437, in find_user return self.session.scalars( ^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1778, in scalars return self.execute( ^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1717, in execute result = conn._execute_20(statement, params or {}, execution_options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20 return meth(self, args_10style, kwargs_10style, execution_options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection return connection._execute_clauseelement( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement ret = self._execute_context( ^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1808, in _execute_context conn = self._revalidate_connection() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 650, in _revalidate_connection self._invalid_transaction() File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 622, in _invalid_transaction raise exc.PendingRollbackError( sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. (Background on this error at: https://sqlalche.me/e/14/8s2b)

The link goes to this block in the sql alchemy docs:
Can’t reconnect until invalid transaction is rolled back¶
This error condition refers to the case where a Connection was invalidated, either due to a database disconnect detection or due to an explicit call to Connection.invalidate(), but there is still a transaction present that was initiated by the Connection.begin() method. When a connection is invalidated, any Transaction that was in progress is now in an invalid state, and must be explicitly rolled back in order to remove it from the Connection.

[CincyBC]

This issue is the same as this issue: #61518

@yonus , It looks like the immediate best solution is to pin the fab provider at 3.1.2. I have not tested this yet, but looks promising from the other thread.