New cleaning tools

[MakisH]

Closes #133.

Main files:

• tools/cleaning-tools.sh: Contains one function for each solver (e.g. clean_openfoam), one shared function to clean the preCICE log files, and one function clean_tutorial.
• <tutorial>/<case>/clean.sh: Only triggers the right clean_<solver>.
• <tutorial>/clean-tutorial.sh: The same for every tutorial, only triggers clean_tutorial.
• clean-all.sh: Triggers all clean-tutorial.sh, cleaning everything:

~/github/tutorials [cleaning]$ ./clean-all.sh 
- Cleaning up all tutorials...
-- Cleaning up all cases in /home/makish/github/tutorials/elastic-tube-3d...
--- Cleaning up OpenFOAM case in /home/makish/github/tutorials/elastic-tube-3d/fluid-openfoam
Cleaning case /home/makish/github/tutorials/elastic-tube-3d/fluid-openfoam
---- Cleaning up preCICE logs in /home/makish/github/tutorials/elastic-tube-3d/fluid-openfoam
--- Cleaning up CalculiX case in /home/makish/github/tutorials/elastic-tube-3d/solid-calculix
---- Cleaning up preCICE logs in /home/makish/github/tutorials/elastic-tube-3d/solid-calculix
-- Cleaning up all cases in /home/makish/github/tutorials/flow-over-heated-plate-nearest-projection...
--- Cleaning up OpenFOAM case in /home/makish/github/tutorials/flow-over-heated-plate-nearest-projection/fluid-openfoam
...

This design allows to clean the files at any of the three levels, with the assumption "from this level and deeper".

Maintainability is tremendously improved, as now most changes can happen in one place. The rest should be short and static enough (but also trivial to update, as the files are exactly the same). I could even symlink directly to one file in the tools/ (trying to think of counter-arguments).

Technical notes and design choices

• All cleaning scripts are portable POSIX-compatible shell scripts (i.e. #!/bin/sh), without any Bash-specific feature.
• I have checked them with shellcheck for potential issues.
• Every script starts with set -e -u. These will make the script exit on errors and on undefined variables.
• The scripts cannot be started from arbitrary directories (I may implement this, but I don't see much added value). The check we had before (cd ${0%/*} || exit 1) is not correct.
• clean_precice_logs() lists all types of files explicitly. This should be a sanity check whenever we add new output file types.
• Every clean_<solver> executes inside a subshell (( do_stuff )). This makes it a bit more state-independent ("do I need to cd back at the end?").
• The number of dashes in the beginning of each line (-, --, ...) signifies the level and they serve as separators. Normally, there should be additional output from each file that is removed. The almost duplicate OpenFOAM output comes from the cleanCase of OpenFOAM.
• In most (all?) OpenFOAM cases we have a first step where we copy the backup 0.orig/ to 0/. This is normally not needed, so I left this deletion step out of the function and added it explicitly to each case.
• A few more files are intentionally cleaned up now:
  • The .foam files are now deleted and not recreated. This should be done by the run.sh (there was some motivation for this in the past, but it is not really relevant anymore). (update: done)
  • The elastic-tube-3d previously skipped deleting the OpenFOAM mesh, probably to save time. This was always confusing. I now delete them as well and I will add them to the .gitignore later. (update: reverted the change, handling similar to 0.orig in run.sh)
• There is no participant information used anywhere, but this should be safe enough.

Still open

• Check if the CalculiX, deal.II, Nutils, SU2, and code_aster are cleaned up completely (I just distilled the previous lists for now).
• Add the missing clean.sh and clean-tutorial.sh to the not yet restructured cases and remove the exclusion from clean-all.sh. But that's probably a future todo.

This is part of a trilogy, with the next steps being a common .gitignore (#87) and a few fixes in the run scripts (issue pending).
Note that running clean-all.sh will currently fail, until the leftover FSI/ directory is removed (see #168). Just remove it locally to test it.

Review

@uekerman Please comment on the mechanism.
@fsimonis Please have a quick look at the scripts: do you see anything problematic?
@davidscn Please try to run 1-2 cases and check that it is working as expected.

[davidscn]

* `tools/cleaning-tools.sh`: Contains one function for each solver (e.g. `clean_openfoam`), one shared function to clean the preCICE log files, and one function `clean_tutorial`.

* `<tutorial>/<case>/clean.sh`: Only triggers the right `clean_<solver>`.

* `<tutorial>/clean-tutorial.sh`: The same for every tutorial, only triggers `clean_tutorial`.

* `clean-all.sh`: Triggers all `clean-tutorial.sh`, cleaning everything:

* The scripts cannot be started from arbitrary directories (I may implement this, but I don't see much added value). The check we had before (`cd ${0%/*} || exit 1`) is not correct.

Very nice. The only thing I don't like that much is the duplication on the lowest level. Since we have a naming convention, it might be possible to extract the current participant name from the working path. Using symlinks as you already proposed would be a great idea and then you could just have a single script for all cases on the lowest level.

* In most (all?) OpenFOAM cases we have a first step where we copy the backup `0.orig/` to `0/`. This is normally not needed, so I left this deletion step out of the function and added it explicitly to each case.

I already had cases, where OpenFOAM complains about missing boundary conditions. So, might depend on the version.

* A few more files are intentionally cleaned up now:
  
  * The `.foam` files are now deleted and not recreated. This should be done by the `run.sh` (there was some motivation for this in the past, but it is not really relevant anymore).
  * The `elastic-tube-3d` previously skipped deleting the OpenFOAM mesh, probably to save time. This was always confusing. I now delete them as well and I will add them to the `.gitignore` later.

No, the reason for the elastic-tube-3d was not to to save time, but to keep the mesh. We have no blockMesh script for the mesh (I guess it has been created with an external software). If you delete it here, it will be gone (or you need to restore it from the git tree).

Still open

* Check if the CalculiX, deal.II, Nutils, SU2, and code_aster are cleaned up completely (I just distilled the previous lists for now).

This should be part of the control step in #160.

[uekerman]

I like the mechanism, great stuff 🚀
(I did not test it right now)

[MakisH]

@davidscn Thank you very much for the detailed comments!

Very nice. The only thing I don't like that much is the duplication on the lowest level. Since we have a naming convention, it might be possible to extract the current participant name from the working path. Using symlinks as you already proposed would be a great idea and then you could just have a single script for all cases on the lowest level.

I would like to keep the deepest level case-specific: it allows to remove special files or do additional tricks (e.g. backup something, then restore it). But I used symlinks for the intermediate level, where all files are identical.

I already had cases, where OpenFOAM complains about missing boundary conditions. So, might depend on the version.

I am not sure what you mean here, but I interpret it as "let's keep it as it is".

No, the reason for the elastic-tube-3d was not to to save time, but to keep the mesh. We have no blockMesh script for the mesh (I guess it has been created with an external software). If you delete it here, it will be gone (or you need to restore it from the git tree).

Good catch, thank you. I renamed polyMesh to polyMesh.orig and treated it similarly to 0.orig in run.sh. This makes the clean.sh the same as any other OpenFOAM clean.sh.

I think we have the same issue in the heat-exchanger, but there we have the Download_meshes script as a backup. I don't expect people to run this multiple times and I would prefer to avoid keeping a large backup.

This should be part of the control step in #160.

I checked and they are working fine: OpenFOAM, CalculiX, deal.II, SU2

Pending After merging:

• Check FEniCS cleaning
• Check Nutils cleaning
• Check code_aster cleaning
• Adapt also the Quickstart case
• Update the system tests

[MakisH]

I think the only reliable way to make the scripts work when invoked from any directory would require switching to bash from sh. I am not sure if it is worth the pain sticking to sh, so I would like to postpone this feature to after also considering the run.sh scripts.

. "${BASH_SOURCE%/*}/../../tools/cleaning-tools.sh"

This will not not work on the standard sh or the (default in Ubuntu) dash. I understand that it will also not work on zsh.

OpenFOAM works around this issue by specifying the tools path in an environment variable, but we want to avoid going down that road, as it would make more things unnecessarily complicated for the user.

At the end of the day, I am not sure if the solvers themselves are so portable and nobody ever complained for the tutorials using bash (though this fact is surprising).

[davidscn]

Very, nice 💥
As I said, I will check the cleaning scripts when checking the tutorials again.