Skip to content

take schema from record batch when creating mem table if any #7637

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
matthewgapp wants to merge 1 commit into from
Closed

take schema from record batch when creating mem table if any #7637

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions datafusion/core/src/execution/context.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -534,6 +534,13 @@ impl SessionContext {
let physical = DataFrame::new(self.state(), input);

let batches: Vec<_> = physical.collect_partitioned().await?;
let schema = {
Copy link
Copy Markdown
Contributor Author

@matthewgapp matthewgapp Sep 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This solves the bug but feels like the input.schema() should already be correct. We shouldn't need to reach into the record batch to find out the true schema

Copy link
Copy Markdown
Contributor Author

@matthewgapp matthewgapp Sep 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't investigated enough to know what is the "correct" fix.

Copy link
Copy Markdown
Contributor

@alamb alamb Sep 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree input.schema() should be fixed (in the sense that in the RecordBatches produced don't match the schema declared) that is the root cause of the issue

Perhaps this is related to #7636 that you filed

Copy link
Copy Markdown
Contributor

@alamb alamb Sep 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See also #7636 (comment)

Copy link
Copy Markdown
Contributor Author

@matthewgapp matthewgapp Sep 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was my initial and quick attempt at fixing #7636 before I dove deeper toward a more correct solution. I'm going to close this out since it should be replaced by #7638

if let Some(batch) = batches.first().and_then(|b| b.first()) {
batch.schema().clone()
} else {
schema
}
};
let table = Arc::new(
// pass constraints to the mem table.
MemTable::try_new(schema, batches)?.with_constraints(constraints),
Expand Down