Support arrow struct by ColinLeeo · Pull Request #739 · apache/tsfile

ColinLeeo · 2026-03-09T01:39:56Z

No description provided.

support arrow.

codecov-commenter · 2026-03-09T02:02:53Z

Codecov Report

❌ Patch coverage is 40.48507% with 319 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.87%. Comparing base (dd27faf) to head (12bbcba).
⚠️ Report is 2 commits behind head on develop.

Files with missing lines	Patch %	Lines
cpp/src/cwrapper/arrow_c.cc	46.63%	180 Missing and 42 partials ⚠️
cpp/src/common/tablet.cc	0.00%	35 Missing ⚠️
cpp/src/utils/date_utils.h	0.00%	26 Missing ⚠️
cpp/src/cwrapper/tsfile_cwrapper.cc	0.00%	22 Missing ⚠️
cpp/src/reader/table_result_set.cc	50.00%	1 Missing and 5 partials ⚠️
cpp/src/common/container/bit_map.h	50.00%	4 Missing ⚠️
cpp/src/reader/result_set.h	0.00%	2 Missing ⚠️
cpp/src/reader/tsfile_reader.cc	50.00%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #739      +/-   ##
===========================================
- Coverage    62.09%   61.87%   -0.23%     
===========================================
  Files          700      704       +4     
  Lines        40142    41218    +1076     
  Branches      5650     5908     +258     
===========================================
+ Hits         24926    25503     +577     
- Misses       14522    14885     +363     
- Partials       694      830     +136

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

cpp/src/cwrapper/arrow_c.cc

cpp/src/reader/table_query_executor.h

cpp/test/reader/table_view/tsfile_reader_table_batch_test.cc

python/tsfile/tsfile_reader.pyx

python/requirements.txt

python/setup.py

jt2594838 · 2026-03-19T01:41:01Z

cpp/src/common/tablet.h

     */
    int add_timestamp(uint32_t row_index, int64_t timestamp);

+    int set_timestamps(const int64_t* timestamps, uint32_t count);


Add a comment to explain how cur_row_size is updated after calling this.

jt2594838 · 2026-03-19T01:48:53Z

cpp/src/common/tablet.cc

+int Tablet::set_column_values(uint32_t schema_index, const void* data,
+                              const uint8_t* null_bitmap, uint32_t count) {


add Tablet::set_column_values(uint32_t schema_index, const std::string* data, const uint32_t data_len,
const uint8_t* null_bitmap, uint32_t count)
to support STRING/TEXT/BLOB？

may also use char** to replace std::string* or std::vector to replace data_len.

We could add a bulk API like set_column_strings, but it wouldn't provide a meaningful performance gain over the current per-row add_value approach. Each string still needs to be individually allocated and copied into the PageArena via dup_from, so the internal implementation would just be a loop doing the same work. The bulk interface would only make the call site slightly cleaner, not faster.

Yes, that is my intention.

Using two sets of interfaces at the same time is a great burden to understanding.

jt2594838 · 2026-03-19T01:52:26Z

cpp/src/common/tablet.cc

+        // Convert Arrow bitmap (1=valid, 0=null) to TsFile bitmap (1=null,
+        // 0=valid) by inverting and writing directly.
+        char* tsfile_bm = bitmaps_[schema_index].get_bitmap();
+        uint32_t full_bytes = count / 8;
+        for (uint32_t i = 0; i < full_bytes; i++) {
+            tsfile_bm[i] = ~static_cast<char>(null_bitmap[i]);
+        }


null_bitmap -> nonnull_bitmap?

jt2594838 · 2026-03-19T02:07:00Z

cpp/src/cwrapper/arrow_c.cc

+                if (bm.test(i)) {
+                    // null row: write zero placeholder in Arrow buffer
+                    std::memset(static_cast<char*>(data_buffer) + i * type_size,
+                                0, type_size);


Is it possible to just skip to the next row?

jt2594838 · 2026-03-19T02:23:07Z

cpp/src/cwrapper/arrow_c.cc

+    for (uint32_t i = 0; i < column_count; ++i) {
+        children_arrays[i] = nullptr;
+    }
+
    for (uint32_t i = 0; i < column_count; ++i) {
        children_arrays[i] = static_cast<ArrowArray*>(
            common::mem_alloc(sizeof(ArrowArray), common::MOD_TSBLOCK));
        if (children_arrays[i] == nullptr) {


Is the first initialization necessary?

The cleanup routine after an error checks whether each pointer is null before freeing it. Without initialization, these pointers could contain garbage values, leading to undefined behavior.

jt2594838 · 2026-03-19T02:24:57Z

cpp/src/cwrapper/arrow_c.cc

+    out_schema->format = schema_data->format_strings->at(0).c_str();
+    out_schema->name = schema_data->name_strings->at(0).c_str();


Why is the format string a vector?

jt2594838 · 2026-03-19T02:29:26Z

cpp/src/cwrapper/arrow_c.cc

+    if (time_col_index < 0 || time_col_index >= n_cols)
+        return common::E_INVALID_ARG;


If the reg_schema already specifies the time column, may use it.

The time_column_index should be set by caller.

jt2594838 · 2026-03-19T02:39:03Z

cpp/src/cwrapper/tsfile_cwrapper.h

 // Write Arrow C Data Interface batch into a table (Arrow -> Tablet -> write).
+// time_col_index: index of the time column in the Arrow struct.
+//   >= 0: use the specified column as the time column.
+//   <  0: auto-detect by Arrow format "tsn:" (TIMESTAMP type).


jt2594838 · 2026-03-19T02:45:01Z

python/tsfile/tsfile_writer.pyx

        data: pyarrow.RecordBatch or pyarrow.Table
+        time_col_index: index of the time column in the Arrow schema.
+            >= 0: use the specified column as the time column.
+            <  0: auto-detect by Arrow timestamp type (default).


Where is the related logic?

ColinLeeo added 3 commits March 8, 2026 22:14

tmp code.

a878ad5

support arrow.

fix benchmark.

a4fc6be

support arrow write.

014580b

jt2594838 reviewed Mar 16, 2026

View reviewed changes

ColinLeeo added 4 commits March 18, 2026 21:57

Fix comment.

edca4bb

fix comment.

7f32918

fix some comment.

5fec95b

fix comment.

c8e4aa9

jt2594838 reviewed Mar 19, 2026

View reviewed changes

fix comment.

12bbcba

		int Tablet::set_column_values(uint32_t schema_index, const void* data,
		const uint8_t* null_bitmap, uint32_t count) {

		out_schema->format = schema_data->format_strings->at(0).c_str();
		out_schema->name = schema_data->name_strings->at(0).c_str();

		if (time_col_index < 0 \|\| time_col_index >= n_cols)
		return common::E_INVALID_ARG;

Conversation

ColinLeeo commented Mar 9, 2026

Uh oh!

codecov-commenter commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov-commenter commented Mar 9, 2026 •

edited

Loading