Skip to content

Support local path (working outside CCF dev server) for querying the index with DuckDb #14

Merged
lfoppiano merged 43 commits intomainfrom
luca/feature/support-local-path
Feb 12, 2026
Merged

Support local path (working outside CCF dev server) for querying the index with DuckDb #14
lfoppiano merged 43 commits intomainfrom
luca/feature/support-local-path

Conversation

@lfoppiano
Copy link
Collaborator

@lfoppiano lfoppiano commented Jan 19, 2026

Description

In this PR we adds:

  • download instructions through https://data.commoncrawl.org for the parquet files (maybe there is a better way to achieve the same result 🙏 😅 )
  • a new target local_parquet that allow the user to specify a local directory to query the index using DuckDb:

make local_parquet LOCAL_DIR=/my/local/parquet

This was suggested by Sebastian here

@lfoppiano lfoppiano mentioned this pull request Jan 19, 2026
@lfoppiano lfoppiano marked this pull request as ready for review January 26, 2026 15:19
Copy link

@sebastian-nagel sebastian-nagel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @lfoppiano, thanks!

To run it with local files some changes are required. At least, as far I could test it. See the inline comments.

Copy link

@sebastian-nagel sebastian-nagel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @lfoppiano. Looks good! Testing DuckDb with local files works seamlessly.

@lfoppiano
Copy link
Collaborator Author

Thank you for the review! ☺️

@lfoppiano lfoppiano merged commit 566831a into main Feb 12, 2026
1 check passed
@lfoppiano lfoppiano deleted the luca/feature/support-local-path branch February 12, 2026 16:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants