Skip to content

[improvement] improvement for light weight schema change#10860

Merged
yiguolei merged 10 commits intoapache:masterfrom
yiguolei:dynamic_schema
Jul 18, 2022
Merged

[improvement] improvement for light weight schema change#10860
yiguolei merged 10 commits intoapache:masterfrom
yiguolei:dynamic_schema

Conversation

@yiguolei
Copy link
Copy Markdown
Contributor

Proposed changes

  1. not use schema as lru cache key any more.
  2. load segment just use the rowset's original schema not the current read schema.
  3. generate column reader and column iterator using the original schema, using the read schema if it is a new column.
  4. using column unique id as key instead of column ordinals.

Problem Summary:

Describe the overview of changes.

Checklist(Required)

  1. Does it affect the original behavior: (Yes/No/I Don't know)
  2. Has unit tests been added: (Yes/No/No Need)
  3. Has document been added or modified: (Yes/No/No Need)
  4. Does it need to update dependencies: (Yes/No)
  5. Are there any changes that cannot be rolled back: (Yes/No)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@Lchangliang
Copy link
Copy Markdown
Contributor

LGTM

@yiguolei yiguolei force-pushed the dynamic_schema branch 3 times, most recently from 6a77cdf to 7d0f17e Compare July 18, 2022 00:56
Copy link
Copy Markdown
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman added kind/improvement area/schema-change Issues or PRs related to schema change labels Jul 18, 2022
@yiguolei yiguolei merged commit a2ed4b5 into apache:master Jul 18, 2022
miswujian pushed a commit to miswujian/doris that referenced this pull request Jul 28, 2022
* improvement for dynamic schema
not use schema as lru cache key any more.
load segment just use the rowset's original schema not the current read schema.
generate column reader and column iterator using the original schema, using the read schema if it is a new column.
using column unique id as key instead of column ordinals.
Co-authored-by: yiguolei <yiguolei@gmail.com>
whutpencil pushed a commit to whutpencil/incubator-doris that referenced this pull request Jul 29, 2022
* improvement for dynamic schema
not use schema as lru cache key any more.
load segment just use the rowset's original schema not the current read schema.
generate column reader and column iterator using the original schema, using the read schema if it is a new column.
using column unique id as key instead of column ordinals.
Co-authored-by: yiguolei <yiguolei@gmail.com>
eldenmoon pushed a commit to eldenmoon/incubator-doris that referenced this pull request Aug 1, 2022
* improvement for dynamic schema
not use schema as lru cache key any more.
load segment just use the rowset's original schema not the current read schema.
generate column reader and column iterator using the original schema, using the read schema if it is a new column.
using column unique id as key instead of column ordinals.
Co-authored-by: yiguolei <yiguolei@gmail.com>
@yiguolei
Copy link
Copy Markdown
Contributor Author

yiguolei commented Aug 3, 2022

part of #10136

@yiguolei yiguolei deleted the dynamic_schema branch March 30, 2023 10:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/schema-change Issues or PRs related to schema change kind/improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants