feat: implement RESTCatalog with database and table CRUD by discivigour · Pull Request #160 · apache/paimon-rust

discivigour · 2026-03-30T09:03:46Z

Purpose

Linked issue: close #119

Implement a complete REST-based catalog (RESTCatalog) for Apache Paimon Rust,
supporting database and table CRUD operations, token-based FileIO for OSS data access.

Brief change log

New files:

catalog/rest/mod.rs: REST catalog module entry point.
catalog/rest/rest_catalog.rs: RESTCatalog implementing the Catalog trait with full
database and table CRUD (list, create, get, drop, alter, rename).
catalog/rest/rest_token.rs: RESTToken struct for table-level data access credentials.
catalog/rest/rest_token_file_io.rs: RESTTokenFileIO — a FileIO wrapper that lazily
fetches and caches table tokens from the REST server, enabling OSS data access with
short-lived credentials.
catalog/database.rs: Database struct representing a catalog database.
examples/rest_catalog_example.rs: End-to-end example for REST catalog operations.
examples/rest_catalog_read_append_example.rs: Example for reading append-only tables via REST catalog.
tests/mock_server.rs: Mock HTTP server (axum-based) simulating the Paimon REST API for testing.
tests/rest_catalog_test.rs: Integration tests covering all CRUD operations on the mock server.

Modified files:

api/rest_api.rs: Add options() accessor and load_table_token() method; fix variable
shadowing in config merge; remove #[allow(dead_code)] on options field.
catalog/filesystem.rs: Implement get_database for FileSystemCatalog.
io/storage_oss.rs: Use HashMap::remove instead of get + clone for security_token.
integration_tests/tests/read_tables.rs: Add REST catalog read tests using mock server.

Tests

tests/rest_catalog_test.rs: Full CRUD integration tests (list/create/get/drop/alter database,
list/create/get/drop/rename/alter table) against the mock server.
tests/rest_api_test.rs: REST API layer tests including DLF ECS token loader.
integration_tests/tests/read_tables.rs: End-to-end read tests for append-only and
primary-key (deletion-vector) tables via REST catalog backed by mock server.
cargo test -p paimon passes.

API and Format

Documentation

# Conflicts: # crates/paimon/tests/mock_server.rs

luoyuxia

@discivigour Thanks for the pr. Left some comments. PTAL

luoyuxia · 2026-03-31T12:13:46Z

crates/integration_tests/tests/read_tables.rs

+///
+/// The mock server returns table metadata pointing to Spark-provisioned data on disk.
+#[tokio::test]
+async fn test_rest_catalog_read_append_table() {


is there any difference between log table and dv table regarding to rest catalog?
If not, I think we just need to keep only one test.

Also, i think you can reuse scan_and_read_with_projection. Pass a catalog trait to it. It should works for both filesystem catalog and rest catalog.

luoyuxia · 2026-03-31T12:16:57Z

crates/integration_tests/Cargo.toml

 arrow-array = { workspace = true }
 tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
 futures = "0.3"
+serde_json = "1"


can be removed?

No, 'mod mock_server' needs it. But I will move it to dev-dependencies.

luoyuxia · 2026-03-31T12:18:42Z

crates/paimon/examples/rest_catalog_example.rs

+}
+
+#[tokio::main]
+async fn main() {


can these two example combine to single one example?

luoyuxia · 2026-03-31T12:33:25Z

crates/paimon/src/api/auth/dlf_provider.rs

 pub struct DLFAuthProvider {
    uri: String,
-    token: Option<DLFToken>,
+    token: tokio::sync::Mutex<Option<DLFToken>>,


nit:
use parking_lot::RwLock

RwLock<Option<DLFToken>>,

Thanks, but I think mutex is better here because get_or_refresh_token() have read-then-write operation.

luoyuxia · 2026-03-31T12:34:56Z

crates/paimon/src/catalog/rest/rest_catalog.rs

+        let warehouse = options
+            .get(CatalogOptions::WAREHOUSE)
+            .cloned()
+            .unwrap_or_default();


return error instead of default?

Good point.

luoyuxia · 2026-03-31T12:45:28Z

crates/paimon/src/catalog/rest/rest_token_file_io.rs

+        }
+
+        // Refresh the token
+        let new_token = self.refresh_token().await?;


please don't hold lock across .await points, it may well cause dead lock

luoyuxia · 2026-03-31T12:45:42Z

crates/paimon/src/catalog/rest/rest_token_file_io.rs

+        let api = match api_guard.as_ref() {
+            Some(existing) => existing,
+            None => {
+                let new_api = RESTApi::new(self.catalog_options.clone(), false).await?;


dito:
please don't hold lock across .await points, it may well cause dead lock

luoyuxia · 2026-03-31T12:46:53Z

crates/paimon/src/io/file_io.rs

-            .context(IoUnexpectedSnafu {
-                message: format!("Failed to list files in '{path}'"),
-            })?;
+        let entries = op.list_with(&list_path).await.context(IoUnexpectedSnafu {


why change this?

luoyuxia · 2026-03-31T12:47:23Z

crates/paimon/src/io/file_io.rs

-
+        let list_path_normalized = list_path.trim_start_matches('/');
        for entry in entries {
+            // opendal list_with includes the root directory itself as the first entry.


is it true?

Why does it work before?

luoyuxia · 2026-03-31T12:51:54Z

crates/paimon/src/catalog/filesystem.rs

        let mut dirs = Vec::new();
        for status in statuses {
            if status.is_dir {
+                // Skip the directory itself (opendal list_with includes the root entry)


is that true?

discivigour marked this pull request as draft March 30, 2026 09:05

discivigour force-pushed the restCatalog branch from c3b4acc to c4f6182 Compare March 30, 2026 09:40

umi added 7 commits March 31, 2026 14:40

proto

fe425a4

add

7a44f25

feat: implement RESTCatalog with database and table CRUD

a68df61

# Conflicts: # crates/paimon/tests/mock_server.rs

fix

235d1dd

clippy

c317b88

fix

80da0e0

fix

a8161d0

discivigour force-pushed the restCatalog branch from 04a01f6 to a8161d0 Compare March 31, 2026 06:45

fix

23820f5

discivigour marked this pull request as ready for review March 31, 2026 07:11

luoyuxia reviewed Mar 31, 2026

View reviewed changes

Conversation

discivigour commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Brief change log

Tests

API and Format

Documentation

Uh oh!

luoyuxia left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

discivigour Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

discivigour commented Mar 30, 2026 •

edited

Loading

discivigour Mar 31, 2026 •

edited

Loading