chore(ci): implement CI optimizations#691
Conversation
Closes #690 - Reordered Dockerfiles to maximize layer caching - Implemented pytest-xdist to run tests in parallel using pytest-cov for coverage - Implemented GitHub Actions Docker cache via --cache-from and pulling latest images
There was a problem hiding this comment.
Pull request overview
This PR targets reducing pr.yaml workflow duration by improving Docker layer caching and introducing parallel pytest execution/coverage collection to shorten build + test cycles.
Changes:
- Reordered Dockerfile steps (tests + notebook) to maximize cache hits by installing heavier dependencies before adding the repo contents.
- Added
pytest-xdistandpytest-cov, and updated the test runner to use parallel pytest workers with coverage. - Updated
ci.shto pull “latest” images before builds and use--cache-fromto reuse existing layers.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
test/run_test.sh |
Switches to pytest-xdist parallel runs and pytest-cov coverage generation. |
pyproject.toml |
Adds pytest-xdist and pytest-cov to dev optional dependencies. |
docker/tests/Dockerfile |
Moves heavy installs (CLIP) ahead of adding source to improve caching. |
docker/notebook/Dockerfile |
Reorders installs (CLIP/apt/Jupyter extensions) before adding source for better caching. |
ci.sh |
Pulls latest images and adds --cache-from to improve Docker build cache reuse. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
ad-claw000
left a comment
ad-claw000
left a comment
There was a problem hiding this comment.
Applied autopep8 formatting to the new log messages to fix the pre-commit CI failure.
ad-claw000
left a comment
ad-claw000
left a comment
There was a problem hiding this comment.
Addressed the remaining feedback about pytest-xdist in the PR description.
ad-claw000
left a comment
ad-claw000
left a comment
There was a problem hiding this comment.
Addressed the remaining feedback about pytest-xdist in the PR description.
Closes #690
This PR implements the 3 alternative solutions proposed in #690 to significantly reduce the execution time of the
pr.yamlworkflow.Changes:
docker/notebook/Dockerfileanddocker/tests/Dockerfileto ensure heavy external dependencies (likeCLIPandaptpackages) are installed before adding the repository source files. This prevents full dependency rebuilds on every code change.pytest-xdistandpytest-covtopyproject.toml. Updatedtest/run_test.shto usepytest -n autofor parallel execution and--cov=aperturedbfor proper multiprocessing coverage generation, bypassing sequential execution bottlenecks.ci.shto explicitly pull thelatestversion ofaperturedb-notebookandaperturedb-python-testsbefore building, and leveraged the--cache-fromflag for the notebook build step.@luisremis Ready for your review.