ENH: Drop *.o/*.a between ctest_build and ctest_test#6322
Conversation
dzenanz
left a comment
There was a problem hiding this comment.
LGTM. I have not tried running this. It would be good if someone else took a look.
|
| Filename | Overview |
|---|---|
| itk_common.cmake | Adds a post-build, pre-test cleanup step that removes all *.o and *.a files under CTEST_BINARY_DIRECTORY using file(GLOB_RECURSE) + file(REMOVE). Two opt-outs are provided: automatic skip under dashboard_do_coverage, and a new dashboard_keep_objects flag. Logic and flow are correct for Unix/Linux targets; Windows/MSVC intermediates are not addressed. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[ctest_build] --> B[ci_report_build_diagnostics]
B --> C{dashboard_do_coverage\nOR dashboard_keep_objects?}
C -- Yes --> F[ci_section_start: test]
C -- No --> D[GLOB_RECURSE *.o *.a]
D --> E2{count > 0?}
E2 -- Yes --> E3[file REMOVE all matched files]
E2 -- No --> E4[no-op]
E3 --> E5[unset variables]
E4 --> E5
E5 --> F
F --> G[ctest_test]
G --> H{dashboard_do_coverage?}
H -- Yes --> I[ctest_coverage]
H -- No --> J{dashboard_do_memcheck?}
I --> J
J -- Yes --> K[ctest_memcheck]
J -- No --> L[ctest_submit]
K --> L
Reviews (1): Last reviewed commit: "ENH: Drop *.o/*.a between ctest_build an..." | Re-trigger Greptile
blowekamp
left a comment
There was a problem hiding this comment.
Looks reasonable to me. Consider windows files extensions too.
Add an automatic post-build, pre-test cleanup step to itk_common.cmake that deletes intermediate object files (*.o) and static archives (*.a) from CTEST_BINARY_DIRECTORY before ctest_test runs. Linked executables and shared libraries are already complete by the time ctest_build returns; the *.o files and *.a archives remaining in the tree are not needed for ctest_test, ctest_memcheck, or ctest_submit and can total many gigabytes for a default-configured ITK build. Removing them frees disk for the test phase, which on size-constrained CI runners (GitHub Actions Ubuntu 22.04 has ~73 GiB root with ~2 GiB free post-build) can otherwise reach "No space left on device" while writing logs or large BigIO test outputs. The cleanup is skipped when dashboard_do_coverage is set (gcov needs the .gcno notes co-located with the corresponding .o files in the build tree) and can also be skipped per-client via the new dashboard_keep_objects opt-out variable. The step is wrapped in a CI fold (ci_section_start / ci_section_end "cleanup_objects_*") so it appears next to the build and test sections in the GitHub Actions log.
e6de4ac to
6ba12c3
Compare
|
Pushed 6ba12c3 addressing two greptile P2 findings:
@dzenanz thanks for the LGTM — these two refinements don't change the spirit of the change, just broaden it to the MSVC pipelines and tidy the log output. |
62fc83a
into
InsightSoftwareConsortium:dashboard
Widen the post-build `find build -delete` to cover `*.obj` and `*.lib` so the windows-2022 job benefits from the same disk recovery as Linux and macOS, and run `ccache --evict-older-than 1d` before `--cleanup` to match the eviction policy introduced for dashboard clients in InsightSoftwareConsortium#6322. Follow-up to InsightSoftwareConsortium#6322; that PR moved equivalent logic into itk_common.cmake for clients that drive CTest via dashboard scripts. The Pixi workflow runs cmake + ctest directly and so needs its own inline cleanup.
Widen the post-build `find build -delete` to cover `*.obj` and `*.lib` so the windows-2022 job benefits from the same disk recovery as Linux and macOS, and run `ccache --evict-older-than 1d` before `--cleanup` to match the eviction policy introduced for dashboard clients in InsightSoftwareConsortium#6322. Follow-up to InsightSoftwareConsortium#6322; that PR moved equivalent logic into itk_common.cmake for clients that drive CTest via dashboard scripts. The Pixi workflow runs cmake + ctest directly and so needs its own inline cleanup.
Delete Windows compiler intermediates (*.obj, *.lib) in the post-build "Free disk space" step so windows-2022 gets the same disk recovery as Linux and macOS, and run `ccache --evict-older-than 1d` before `--cleanup` to match the eviction policy introduced for dashboard clients in InsightSoftwareConsortium#6322. Scope the deletion by path: *.obj also names Wavefront mesh fixtures (ITKIOMeshOBJ, Cuberille) and *.lib names a DCMTK Makefile template, so match compiled objects only under CMakeFiles/ and archives only under lib/ to avoid removing test data before the test phase. Follow-up to InsightSoftwareConsortium#6322; that PR moved equivalent logic into itk_common.cmake for clients that drive CTest via dashboard scripts. The Pixi workflow runs cmake + ctest directly and so needs its own inline cleanup.
The post-build intermediate cleanup added in InsightSoftwareConsortium#6322 globs *.obj and *.lib recursively under CTEST_BINARY_DIRECTORY and deletes the matches before ctest_test. Both extensions are overloaded: *.obj also names Wavefront mesh fixtures (ITKIOMeshOBJ Baseline/bunny.obj, box.obj; Cuberille mesh.obj) and *.lib names a DCMTK Makefile template. GLOB_RECURSE descends into ExternalData/, so those fixtures were removed between build and test, failing itkOBJMeshIOTest1/2 and itkMeshFileReadWriteOBJ* on every dashboard client (ARMBUILD, Azure ITK.* pipelines). Scope the deletion by path: match compiled objects only under CMakeFiles/ and archives only under lib/, where every compiler-emitted object and archive lives on all platforms. Verified against a real build tree: 5579 compiled objects and 148 archives still removed; all seven *.obj mesh fixtures and the DCMTK Makefile.lib spared.
Delete Windows compiler intermediates (*.obj, *.lib) in the post-build "Free disk space" step so windows-2022 gets the same disk recovery as Linux and macOS, and run `ccache --evict-older-than 1d` before `--cleanup` to match the eviction policy introduced for dashboard clients in InsightSoftwareConsortium#6322. Scope the deletion by path: *.obj also names Wavefront mesh fixtures (ITKIOMeshOBJ, Cuberille) and *.lib names a DCMTK Makefile template, so match compiled objects only under CMakeFiles/ and archives only under lib/ to avoid removing test data before the test phase. Follow-up to InsightSoftwareConsortium#6322; that PR moved equivalent logic into itk_common.cmake for clients that drive CTest via dashboard scripts. The Pixi workflow runs cmake + ctest directly and so needs its own inline cleanup.
…test-fixtures BUG: Spare test fixtures from dashboard post-build cleanup (#6322 regression)
Delete Windows compiler intermediates (*.obj, *.lib) in the post-build "Free disk space" step so windows-2022 gets the same disk recovery as Linux and macOS, and run `ccache --evict-older-than 1d` before `--cleanup` to match the eviction policy introduced for dashboard clients in InsightSoftwareConsortium#6322. Scope the deletion by path: *.obj also names Wavefront mesh fixtures (ITKIOMeshOBJ, Cuberille) and *.lib names a DCMTK Makefile template, so match compiled objects only under CMakeFiles/ and archives only under lib/ to avoid removing test data before the test phase. Follow-up to InsightSoftwareConsortium#6322; that PR moved equivalent logic into itk_common.cmake for clients that drive CTest via dashboard scripts. The Pixi workflow runs cmake + ctest directly and so needs its own inline cleanup.
…cts-and-ccache-evict COMP: Extend Pixi-Cxx post-build cleanup to MSVC artifacts (followup #6322)
Adds an automatic post-build, pre-test cleanup step to
itk_common.cmakethat deletes intermediate*.oand*.afiles fromCTEST_BINARY_DIRECTORYbetweenctest_buildandctest_test. Free disk for the test phase on size-constrained CI runners (the GitHub Actions Ubuntu 22.04 runner has ~73 GiB root with ~2 GiB free immediately post-build; tests can hit "No space left on device" before they get a chance to run).What this does
Just after
ctest_buildreturns and beforectest_testis called, the script removes every*.oand*.aunderCTEST_BINARY_DIRECTORY. The linked executables and shared libraries are already complete at this point; the intermediates aren't needed byctest_test,ctest_memcheck, orctest_submit.Two opt-outs:
dashboard_do_coverage— automatic skip.gcovneeds the.gcnonote files colocated with the corresponding.ofiles in the build tree.dashboard_keep_objects— new per-client opt-out, e.g. for clients that re-link incrementally between dashboard iterations.The step is wrapped in a CI fold (
ci_section_start/ci_section_end"Free build-tree object files") so it appears next to the build and test sections in the GitHub Actions log.Motivating CI failure
From PR #6318's
Pixi-Cxx (ubuntu-22.04)job:The GitHub Actions workflow has been working around this by running
find build -type f -name "*.o" -delete+find build -type f -name "*.a" -deletein a dedicated "Free disk space after build" step (see.github/workflows/itk.pixi.yml). This PR moves that same logic into the canonical dashboard script so it applies to every consumer ofitk_common.cmake(Azure, CircleCI, Jenkins, custom nightly clients) — not just GitHub Actions.