Cherry-pick: Fix: system.databases always shows data lake catalog databases

[zvonand]

Cherry-picked from ClickHouse#103444.

Fix: system.databases always shows data lake catalog databases

---

system.databases now lists data lake catalog databases regardless of the show_data_lake_catalogs_in_system_tables setting.

The setting exists to prevent accidental expensive calls to remote catalog services (e.g. listing tables or columns). Knowing that a catalog database exists, however, requires no remote call — the DatabaseCatalog already keeps it in local memory. Hiding the database in system.databases based on that setting was inconsistent with SHOW DATABASES (which always showed catalogs since ClickHouse#89914) and confusing for users.

Changes:

• StorageSystemDatabases::fillData now passes with_datalake_catalogs = true unconditionally.
• InterpreterShowTablesQuery::execute no longer needs to force the setting for SHOW DATABASES (the rewrite to system.databases already returns the right result). The HACK is retained only for SHOW TABLES FROM <datalake_catalog> which still needs it.
• Updated test_database_glue integration test to assert that system.databases shows the catalog even with the setting off.

Followup to: ClickHouse#89914

Changelog category (leave one):

• Improvement

Changelog entry

SELECT * FROM system.databases now always lists data lake catalog databases regardless of the show_data_lake_catalogs_in_system_tables setting. Previously they were hidden by default, which was inconsistent with SHOW DATABASES that always showed them.

[github-actions]

Workflow [PR], commit [7d9e9cf]

[zvonand]

test fails do not look related at all. also. it was merged in upstream already.

[alsugiliazova]

Verification report: Altinity/ClickHouse PR #1690

---

Conclusion

PR is merged. CI on head was not green, but every failure is a pre-existing flake or job-level error — none traceable to the PR diff. No follow-up required beyond standard flake tracking.

---

Latest CI on head 7d9e9cf — failures

PR test workflow (9 failed checks, 91 success)

Check	Job status	Test FAILs	Class
Integration tests (amd_asan, db disk, old analyzer, 2/6)	error	test_replicated_database::test_sync_replica	Flake — 49 fails / 21 PRs / 90d
Integration tests (amd_asan, db disk, old analyzer, 4/6)	failure	test_storage_delta/test_cdf.py::test_cdf[]	Flake — 36 / 21
Integration tests (amd_asan, db disk, old analyzer, 3/6)	error	(none recorded)	Job error
Integration tests (amd_tsan, 5/6)	failure	7× test_scheduler_cpu_preemptive::* (fairness, downscaling, create_workload)	Flaky suite — 169 fails / 17 PRs / 90d
Integration tests (amd_tsan, 6/6)	error	(none recorded)	Job error
Integration tests (amd_binary, 4/5)	failure	2× test_backup_restore_s3::test_backup_to_s3_different_credentials[*-non_native_multipart]	Flake — 24+12 / 16+9
Integration tests (arm_binary, distributed plan, 2/4)	failure	test_storage_s3_queue::test_move_after_processing[another_bucket-AzureQueue]	Flake — 58 / 35
Stateless tests (amd_tsan, s3 storage, parallel, 2/2)	error	(none recorded)	Job error
BuzzHouse (amd_tsan)	failure	Unknown error	Fuzzer noise

Regression workflow (3 failed checks)

Check	DB top fail (PR-1690 builds, 30d)	Baseline fail rate (90d)
RegressionTestsRelease / Swarms / swarms	/swarms/feature/node failure/initiator out of disk space (×2)	373/408 ≈ 91% — broken at baseline
RegressionTestsAarch64 / Swarms / swarms	same	same
RegressionTestsAarch64 / AggregateFunctions (3) / aggregate_functions_3	/aggregate functions/part 3/state/argMaxState (×1)	high-volume suite, single-test flake

Regression DB on /PRs/1690/ builds (30d): 14 Fail / 41,756 OK ≈ 0.03%.

---

Related to PR diff?

PR is a cherry-pick fixing system.databases visibility for data lake catalog databases (Databases/*, Storages/DataLakes/* area).

Failing test	Diff overlap	Related?
test_replicated_database::test_sync_replica	none	No
test_storage_delta::test_cdf[]	none	No
test_backup_restore_s3::*	none	No
test_storage_s3_queue::*	none	No
test_scheduler_cpu_preemptive::*	none	No
BuzzHouse Unknown error	n/a	No
swarms / aggregate_functions_3 regression	none	No

No failure intersects the data-lake-catalog / system.databases code path.

---

Recommendations

1. No action on this PR — it is merged and clean.
2. The integration flakes here are all already known and recurring across many PRs; track via the existing flake list (no need to file new issues from this run).
3. The swarms initiator out of disk space scenario is broken at baseline (91% fail rate) — should already be tracked separately; if not, file a single tracking issue.

[alsugiliazova]

AI audit note: This review comment was generated by AI (Sonnet 4.6 / Cursor).

Audit for PR #1690 (system.databases always shows data lake catalog databases):

Confirmed defects:

No confirmed defects in reviewed scope.

Coverage summary:

Scope reviewed: StorageSystemDatabases::fillData, InterpreterShowTablesQuery::execute/getRewrittenQuery,
                DatabaseCatalog::getDatabases/isDatalakeCatalog, DatabaseCatalog.h comment,
                test_database_glue assertion update (4 files, 16+/20−).
Categories failed: none.
Categories passed: setting-gated visibility, SHOW DATABASES / system.databases consistency,
                   SHOW TABLES FROM <catalog> hack retained, thread safety (lock+copy semantics),
                   detach-during-read lifetime, access control on expanded set, C++ memory/RAII,
                   exception safety, protocol/API parity (LIKE/LIMIT), unused-declaration cleanup,
                   comment accuracy, test correctness (13/13 executed; 0 deferred).
Assumptions/limits: Static analysis only; pre-existing dead-code path (unreachable UNKNOWN_DATABASE throw
                    in fillData) noted but not introduced by this PR, and not a correctness risk.