Skip to content

enable memory detect.#746

Open
ColinLeeo wants to merge 5 commits intodevelopfrom
support_memory_detect
Open

enable memory detect.#746
ColinLeeo wants to merge 5 commits intodevelopfrom
support_memory_detect

Conversation

@ColinLeeo
Copy link
Contributor

@ColinLeeo ColinLeeo commented Mar 19, 2026

Test Configuration

Write Configuration:
Table: "testTable0"
TAG columns: 2 (id0, id1) — STRING, PLAIN, UNCOMPRESSED
FIELD columns: 3 (s0, s1, s2) — INT64, PLAIN, UNCOMPRESSED
Devices: 10 per flush
Timestamps: 100 per device per flush
Rows per flush: 1000 (10 devices × 100 timestamps)
Flush cycles: 10
Total rows: 10000

Read Configuration:
Query: full table scan (time range [0, INT64_MAX])
Mode: row-by-row iteration via TableResultSet::next()
Sample points: every 2000 rows (5 samples total)

mem_stat

Memory Statistics & Tracking for TsFile C++

1. Write & Flush Behavior

  • PAGE_WRITER_OUTPUT_STREAM: allocated during encoding, fully released on flush — produces a sawtooth memory pattern.
  • TSFILE_WRITER_META: allocated during flush to serialize ChunkMeta/Statistics into the in-memory index.
  • Accumulates across flushes (~3 KB/flush) and is only released at writer close. This is by design — the full metadata index must be retained until the TsFileMetadata footer is written.
  • TSFILE_WRITE_STREAM: the file I/O output buffer, first appears after the initial flush.

2. Overall Memory Growth During Write

  • Total memory rises steadily due to TSFILE_WRITER_META accumulation (57.6 KB over 20 flushes).
  • All other allocations (encoder, compressor, TS2Diff) remain constant after initialization.

3. Residual Memory After Writer Close

  • 85.7 KB remains after close(), belonging to objects owned by the TsFileTableWriter shared pointer (PageArena, encoders, compressors, statistics).
  • These are fully released when the shared pointer goes out of scope — not a leak, just deferred destruction.

4. Read Process Memory

  • TSFILE_READER grows linearly at ~33.5 bytes/row during row-by-row iteration.
    Root cause: TableResultSet::next() allocates string values from a PageArena, which is an arena allocator — it only frees memory in bulk on destroy(), not per-allocation. Previous rows' string memory becomes unreachable but stays in the arena.
  • Potential fix: periodically reset the arena when row_record is reset.
    TSBLOCK (74.5 KB) is stable — decoded data buffers are reused across pages.
  • All reader-specific memory (TSFILE_READER, TSBLOCK, BLOOM_FILTER, DEVICE_*_ITER) is fully released after result set destroy + reader close.

5. Residual Memory After Reader Close

  • 85.7 KB remaining is entirely from the writer's deferred destruction (same objects as point 3). Reader itself releases all its allocations cleanly.

@codecov-commenter
Copy link

codecov-commenter commented Mar 19, 2026

Codecov Report

❌ Patch coverage is 66.66667% with 36 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.13%. Comparing base (b90f676) to head (1df055f).

Files with missing lines Patch % Lines
cpp/src/common/allocator/mem_alloc.cc 18.51% 22 Missing ⚠️
cpp/src/common/tablet.cc 80.43% 0 Missing and 9 partials ⚠️
cpp/src/common/allocator/byte_stream.h 82.35% 2 Missing and 1 partial ⚠️
cpp/src/encoding/int64_rle_decoder.h 66.66% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #746      +/-   ##
===========================================
- Coverage    62.15%   62.13%   -0.03%     
===========================================
  Files          702      702              
  Lines        40680    40738      +58     
  Branches      5780     5801      +21     
===========================================
+ Hits         25286    25313      +27     
- Misses       14613    14634      +21     
- Partials       781      791      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@hongzhi-gao
Copy link
Contributor

Good work!!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants