[Feature](multi-catalog) Add memory tracker for orc reader/writer and arrow parquet writer。#37234
Merged
morningman merged 1 commit intoapache:masterfrom Jul 30, 2024
Merged
Conversation
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
Contributor
Author
|
run buildall |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
|
||
| #include <map> | ||
|
|
||
| #include "orc/MemoryPool.hh" |
Contributor
There was a problem hiding this comment.
warning: 'orc/MemoryPool.hh' file not found [clang-diagnostic-error]
#include "orc/MemoryPool.hh"
^
kaka11chen
force-pushed
the
orc_parquet_memory_tracking
branch
from
6bb4b3f to
61ac8ab
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 40162 ms |
TPC-DS: Total hot run time: 175352 ms |
ClickBench: Total hot run time: 30.89 s |
kaka11chen
force-pushed
the
orc_parquet_memory_tracking
branch
from
61ac8ab to
671de96
Compare
| #endif // #if defined(USE_JEMALLOC) && defined(USE_MEM_TRACKER) | ||
| } | ||
|
|
||
| void free(char* p) override { |
Contributor
There was a problem hiding this comment.
warning: pointer parameter 'p' can be pointer to const [readability-non-const-parameter]
Suggested change
| void free(char* p) override { | |
| void free(const char* p) override { |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 41072 ms |
TPC-DS: Total hot run time: 172392 ms |
ClickBench: Total hot run time: 30.95 s |
kaka11chen
force-pushed
the
orc_parquet_memory_tracking
branch
from
671de96 to
1fa9e9f
Compare
Contributor
Author
|
run buildall |
| const iceberg::Schema* iceberg_schema = nullptr); | ||
|
|
||
| ~VOrcTransformer() = default; | ||
| ~VOrcTransformer(); |
Contributor
There was a problem hiding this comment.
warning: annotate this function with 'override' or (rarely) 'final' [modernize-use-override]
Suggested change
| ~VOrcTransformer(); | |
| ~VOrcTransformer() override; |
TPC-H: Total hot run time: 39593 ms |
TPC-DS: Total hot run time: 173515 ms |
ClickBench: Total hot run time: 30.69 s |
kaka11chen
force-pushed
the
orc_parquet_memory_tracking
branch
from
1fa9e9f to
6679a62
Compare
Contributor
Author
|
run buildall |
| std::mutex RecordSizeMemoryAllocator::_mutex; | ||
|
|
||
| template <bool clear_memory_, bool mmap_populate, bool use_mmap, typename MemoryAllocator> | ||
| void Allocator<clear_memory_, mmap_populate, use_mmap, MemoryAllocator>::sys_memory_check( |
Contributor
There was a problem hiding this comment.
warning: function 'sys_memory_check' exceeds recommended size/complexity thresholds [readability-function-size]
void Allocator<clear_memory_, mmap_populate, use_mmap, MemoryAllocator>::sys_memory_check(
^Additional context
be/src/vec/common/allocator.cpp:46: 112 lines including whitespace and comments (threshold 80)
void Allocator<clear_memory_, mmap_populate, use_mmap, MemoryAllocator>::sys_memory_check(
^| @@ -23,6 +23,10 @@ | |||
| // TODO: Readable | |||
|
|
|||
| #include <fmt/format.h> | |||
Contributor
There was a problem hiding this comment.
warning: 'fmt/format.h' file not found [clang-diagnostic-error]
#include <fmt/format.h>
^|
|
||
| void* alloc(size_t size) { | ||
| if (size <= N) { | ||
| if constexpr (Base::clear_memory) memset(stack_memory, 0, N); |
Contributor
There was a problem hiding this comment.
warning: statement should be inside braces [readability-braces-around-statements]
Suggested change
| if constexpr (Base::clear_memory) memset(stack_memory, 0, N); | |
| if constexpr (Base::clear_memory) { memset(stack_memory, 0, N); | |
| } |
| if (size > N) Base::free(buf, size); | ||
| } | ||
| void free(void* buf, size_t size) { | ||
| if (size > N) Base::free(buf, size); |
Contributor
There was a problem hiding this comment.
warning: statement should be inside braces [readability-braces-around-statements]
Suggested change
| if (size > N) Base::free(buf, size); | |
| if (size > N) { Base::free(buf, size); | |
| } |
| if (new_size <= N) return buf; | ||
| void* realloc(void* buf, size_t old_size, size_t new_size) { | ||
| /// Was in stack_memory, will remain there. | ||
| if (new_size <= N) return buf; |
Contributor
There was a problem hiding this comment.
warning: statement should be inside braces [readability-braces-around-statements]
Suggested change
| if (new_size <= N) return buf; | |
| if (new_size <= N) { return buf; | |
| } |
| /// Already was big enough to not fit in stack_memory. | ||
| if (old_size > N) return Base::realloc(buf, old_size, new_size, Alignment); | ||
| /// Already was big enough to not fit in stack_memory. | ||
| if (old_size > N) return Base::realloc(buf, old_size, new_size, Alignment); |
Contributor
There was a problem hiding this comment.
warning: statement should be inside braces [readability-braces-around-statements]
Suggested change
| if (old_size > N) return Base::realloc(buf, old_size, new_size, Alignment); | |
| if (old_size > N) { return Base::realloc(buf, old_size, new_size, Alignment); | |
| } |
| return arrow::Status::OK(); | ||
| } | ||
|
|
||
| void ArrowAllocator::deallocate_aligned(uint8_t* ptr, int64_t size, int64_t alignment) { |
Contributor
There was a problem hiding this comment.
warning: pointer parameter 'ptr' can be pointer to const [readability-non-const-parameter]
Suggested change
| void ArrowAllocator::deallocate_aligned(uint8_t* ptr, int64_t size, int64_t alignment) { | |
| void ArrowAllocator::deallocate_aligned(const uint8_t* ptr, int64_t size, int64_t alignment) { |
be/src/vec/exec/format/parquet/arrow_memory_pool.h:47:
- void deallocate_aligned(uint8_t* ptr, int64_t size, int64_t alignment);
+ void deallocate_aligned(const uint8_t* ptr, int64_t size, int64_t alignment);|
|
||
| #pragma once | ||
|
|
||
| #include "arrow/memory_pool.h" |
Contributor
There was a problem hiding this comment.
warning: 'arrow/memory_pool.h' file not found [clang-diagnostic-error]
#include "arrow/memory_pool.h"
^
kaka11chen
force-pushed
the
orc_parquet_memory_tracking
branch
from
6679a62 to
fe66e7f
Compare
kaka11chen
changed the title
Contributor
Author
|
run buildall |
kaka11chen
force-pushed
the
orc_parquet_memory_tracking
branch
from
fe66e7f to
ad0e5b4
Compare
Contributor
Author
|
run buildall |
Contributor
|
PR approved by anyone and no changes requested. |
kaka11chen
force-pushed
the
orc_parquet_memory_tracking
branch
from
766cffa to
1bce0fe
Compare
github-actions
bot
removed
the
approved
Contributor
Author
|
run buildall |
kaka11chen
force-pushed
the
orc_parquet_memory_tracking
branch
from
1bce0fe to
af452ad
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 39933 ms |
TPC-DS: Total hot run time: 173202 ms |
ClickBench: Total hot run time: 30.8 s |
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
dataroaring
pushed a commit
that referenced
this pull request
Jul 31, 2024
… arrow parquet writer。 (#37234) ## Proposed changes [Feature] (multi-catalog) Add memory tracker for orc reader/writer and arrow parquet writer。 ## Future work - Since the parquet reader is written by ourself and does not use the arrow third-party library, some memory usage needs to be added to the memory track. - Added read and write operator-level memory tracker to the profile.
3 tasks
zy-kkk
pushed a commit
that referenced
this pull request
Jul 31, 2024
Fix allocator.h compiling failed on mac which introduced by #37234.
kaka11chen
added a commit
to kaka11chen/doris
that referenced
this pull request
Aug 1, 2024
Fix allocator.h compiling failed on mac which introduced by apache#37234.
feiniaofeiafei
pushed a commit
to feiniaofeiafei/doris
that referenced
this pull request
Aug 9, 2024
Fix allocator.h compiling failed on mac which introduced by apache#37234.
dataroaring
pushed a commit
that referenced
this pull request
Aug 11, 2024
Fix allocator.h compiling failed on mac which introduced by #37234.
dataroaring
pushed a commit
that referenced
this pull request
Aug 16, 2024
Fix allocator.h compiling failed on mac which introduced by #37234.
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
[Feature] (multi-catalog) Add memory tracker for orc reader/writer and arrow parquet writer。
Future work