Silent stop at land initialization — coupler/grid issue with large custom domain

[tboas]

Description

eCLM stops silently during land surface initialization without any specific error message.
The model execution halts after reading namelists and printing lnd2atm_inparm settings,
with no clear error message or MPI traceback. The job continues running until the wall
clock time limit is reached without producing output.

Setup

• Domain: DE-009, ~1000x1000 grid cells
• PFTs: 78
• Soil data: Custom soil texture from SoilGrids250m
• Surface file: Generated with static file generator

Investigation

• No mismatch found between domain file and surface file
• Surface file passed all validation checks (no NaN/Inf, valid soil texture,
  land use fractions sum to 100)
• The setup ran successfully with a uniform test surface file
  (flags: -pft_idx 13 -pft_frc 100 -soil_cly 60 -soil_col 10 -soil_fmx 0.5 -soil_snd 40)
  but failed with the customized SoilGrids surface file

Workaround

Applied a Germany land mask to the domain file to reduce the number of active land cells.
The run now works with the customized SoilGrids surface file and the masked domain file.

[tboas]

What are the recommended steps for diagnosing potential resource limitations when running eCLM on the full domain (~1000x1000 grid cells)? Do we have any scaling tests for eCLM on a domain of approx. that size?

[AGonzalezNicolas]

I use this ideal case with 1120x1120 grid cells.
Have you tried to increase the resources?

[tboas]

Yes, I have tried with a different number of nodes. The testcase is in SP mode though, I am running BGC mode which makes a big difference in compute time.

[AGonzalezNicolas]

Please @tboas, add more information about your case:

• working directory. This way we can check them (if we have access).
• also the log-messages of the err-log and out-log files

[kvrigor]

This issue looks similar to what you had before @AGonzalezNicolas : eclm NetCDF error message for large grids

@tboas you can also try changing the NetCDF format into parallel NetCDF, i.e. pio_typename = 'netcdf4p' on all *_modelio.nml files:

&pio_inparm
 pio_netcdf4p_format = '64bit_offset'
 pio_numiotasks = <ntasks from jobscript>
 pio_rearranger = 1
 pio_root = 1
 pio_stride = 1
 pio_typename = 'netcdf4p'
/

[tboas]

My working directory is here: /p/scratch/cjibg36/jibg3676/testing_DE009/prod_run

[AGonzalezNicolas]

I have no access to cjibg36. Try to copy the log-error and log-out messages here. Also try what @kvrigor suggests (pio_typename = 'netcdf4p' in all *_modelio.nml files). In your working directory, apply:

sed -i "s/'netcdf'/'netcdf4p'/g" *modelio.nml

[kgoergen]

Just a side-remark (as we may be using eCLM for larger grids in the context of the ParFlow DE06 quasi-operational runs as well, and we need eCLM with the large TSMP2 runs over Europe with >3Mio grid cells), most likely not relevant:

• llview might already hint whether you are running out of memory writing serial netCDF and collecting large fields on the I/O processor
• it may be the netCDF file type, you may need large file support, or a HDF5 based newer netCDF format

[kvrigor]

• it may be the netCDF file type, you may need large file support, or a HDF5 based newer netCDF format

just FYI for all: in eCLM/CLM5 this can be set by setting pio_typename = 'netcdf4p'. All supported output formats are documented on a previous issue

[kvrigor]

I was able to run Theresa's test case; however it failed after 90 minutes at around model timestep 4500. The error is related to NetCDF I/O failure:

 NetCDF: HDF error
 pio_support::pio_die:: myrank=          -1 : ERROR: pionfread_mod.F90.in:
         204 : NetCDF: HDF error

I'm not yet sure if this is a random failure or a repeatable one. Nevertheless, going back to Theresa's original problem:

eCLM stops silently during land surface initialization without any specific error message.

This is solved by increasing --ntasks to around 512 MPI tasks. This speeds up the generation of initial condition file which involves a time-consuming interpolation step. Once finidat_interp_dest.nc is generated, it could be re-used on the next runs to speed up eCLM initialization.

!
! Modify these params in lnd_in to re-use the generated initial conditions file
! 
finidat = 'finidat_interp_dest.nc'  ! initial conditions file
use_init_interp = .false.           ! do not interpolate finidat file

@tboas Please try if this works for your case. Also please start with a short simulation range and then gradually increase it when previous jobs succeed. We don't know yet if you'll also get the HDF error as I did.

Addendum: Automatically adjusting namelist parameters on the jobscript

The eCLM sbatch script snippet below

• Adjusts all processor-related namelist parameters based on --nodes and --ntasks-per-node settings
• Sets the eCLM subcomponent logging
• Making sure that pio_typename = 'netcdf4p'

# Compute total tasks based on --nodes and --ntasks-per-node setting
NTASKS_TOTAL=$(echo "${SLURM_NNODES} * ${SLURM_NTASKS_PER_NODE}" | bc)
echo "Using ${NTASKS_TOTAL} processes for SLURM job ${SLURM_JOB_NAME}"

# Set up log folders
mkdir -p logs timing/checkpoints
if [[ -z ${SLURM_JOB_ID} || "${SLURM_JOB_ID}" == " " ]]; then
  LOGID=$(date +%Y-%m-%d_%H.%M.%S)
else
  LOGID=${SLURM_JOB_ID}
fi
LOGDIR=$(realpath logs)

# Automatically adjust namelist parameters
comps=(atm cpl esp glc ice lnd ocn rof wav)
for comp in "${comps[@]}"; do
  LOGFILE="$LOGID.comp_${comp}.log"
  sed -i "s#diro.*#diro = \"$LOGDIR\"#" ${comp}_modelio.nml
  sed -i "s#logfile.*#logfile = \"$LOGFILE\"#" ${comp}_modelio.nml
  sed -i "s#pio_numiotasks.*#pio_numiotasks = ${NTASKS_TOTAL}#" ${comp}_modelio.nml
  sed -i "s#pio_typename.*#pio_typename = \"netcdf4p\"#" ${comp}_modelio.nml
  sed -i "s#${comp}_ntasks.*#${comp}_ntasks = ${NTASKS_TOTAL}#" drv_in
done

# Run eclm
srun eclm.exe

[tboas]

Thanks @kvrigor, that solved the issue.

The model now runs successfully using the finidat_interp_dest.nc from your initialization on JURECA. It completed two simulation years with 512 tasks before hitting the walltime limit. I’m also using pio_typename = 'netcdf4p'.

@AGonzalezNicolas, I would like to run some scaling tests with this setup, lets talk about this.

Side note: Paul and I observed different errors with what should be the same configuration. In my case, the initialization, specifically the interpolation from the larger restart file, crashed silently before the finidat_interp_dest.nc file was written and continued until the walltime limit was reached. Any ideas about that?

Another side note: This setup currently works only on JURECA. On JUWELS, I still encounter memory issues (as seen in LLview) during initialization of the land component. I tested different task counts on the large-memory partition, latest was 576 tasks.