Low GPU Utilization When a3fe enter the stage of ensemble equilibration

[gkxiao]

Environment​​

• OS: Ubuntu 24.04.2 LTS
• Hardware: Dual NVIDIA RTX 4090 GPUs (24GB VRAM each), 64+ CPU cores
• Software:
a3fe: 0.33
GROMACS (compiled with CUDA)

GROMACS version:     2025.1
Precision:           mixed
Memory model:        64 bit
MPI library:         thread_mpi
OpenMP support:      enabled (GMX_OPENMP_MAX_THREADS = 128)
GPU support:         CUDA
NBNxM GPU setup:     super-cluster 2x2x2 / cluster 8 (cluster-pair splitting on)
SIMD instructions:   AVX2_256
CPU FFT library:     fftw-3.3.10-sse2-avx-avx2-avx2_128
GPU FFT library:     cuFFT
Multi-GPU FFT:       none
RDTSCP usage:        enabled
TNG support:         enabled
Hwloc support:       disabled
Tracing support:     disabled
C compiler:          /usr/bin/cc GNU 13.3.0
C compiler flags:    -fexcess-precision=fast -funroll-all-loops -mavx2 -mfma -Wno-missing-field-initializers -O3 -DNDEBUG
C++ compiler:        /usr/bin/c++ GNU 13.3.0
C++ compiler flags:  -fexcess-precision=fast -funroll-all-loops -mavx2 -mfma -Wno-missing-field-initializers -Wno-cast-function-type-strict SHELL:-fopenmp -O3 -DNDEBUG
BLAS library:        Internal
LAPACK library:      Internal
CUDA compiler:       /usr/local/cuda-12.6/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2024 NVIDIA Corporation;Built on Tue_Oct_29_23:50:19_PDT_2024;Cuda compilation tools, release 12.6, V12.6.85;Build cuda_12.6.r12.6/compiler.35059454_0
CUDA compiler flags: -O3 -DNDEBUG
CUDA driver:         12.60
CUDA runtime:        12.60

• cat run_somd.sh

#!/bin/bash
#SBATCH -o somd-array-gpu-%A.%a.out
#SBATCH -n 1
#SBATCH --time 24:00:00
#SBATCH --gres=gpu:1

lam=$1
echo "lambda is: " $lam

srun somd-freenrg -C somd.cfg -l $lam -p CUDA

• a3fe script: run_a3fe.py

import a3fe as a3
calc = a3.Calculation(ensemble_size = 5)
calc.setup()
# Get optimised lambda schedule with thermodynamic speed
# of 2 kcal mol-1
calc.get_optimal_lam_vals(delta_er = 2)
# Run adaptively with a runtime constant of 0.0005 kcal**2 mol-2 ns**-1
# Note that automatic equilibration detection with the paired t-test
# method will also be carried out.
calc.run(adaptive=True, runtime_constant = 0.0005)
calc.wait()
calc.analyse()
calc.save()

Observed Behavior

• When a3fe begins to enter the ensemble equilibration step, the GPU load drops sharply.
check the slurm job:

scontrol show jobs 46054
JobId=46054 JobName=ensemble_equil_bound.sh
   UserId=gkxiao(997) GroupId=gkxiao(984) MCS_label=N/A
   Priority=1 Nice=0 Account=(null) QOS=normal
   JobState=RUNNING Reason=None Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:36:28 TimeLimit=1-00:00:00 TimeMin=N/A
   SubmitTime=2025-06-12T15:14:11 EligibleTime=2025-06-12T15:14:11
   AccrueTime=2025-06-12T15:14:11
   StartTime=2025-06-12T15:14:12 EndTime=2025-06-13T15:14:12 Deadline=N/A
   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2025-06-12T15:14:12 Scheduler=Main
   Partition=batch AllocNode:Sid=master:250144
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=master
   BatchHost=master
   NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   ReqTRES=cpu=1,mem=1M,node=1,billing=1,gres/gpu=1
   AllocTRES=cpu=1,node=1,billing=1,gres/gpu=1
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
   Command=/public/gkxiao/software/a3fe/4zlz_gaff2/bound/ensemble_equilibration_2/ensemble_equil_bound.sh
   WorkDir=/public/gkxiao/software/a3fe/4zlz_gaff2/bound/ensemble_equilibration_2
   StdErr=/public/gkxiao/software/a3fe/4zlz_gaff2/bound/ensemble_equilibration_2/somd-array-gpu-46054.4294967294.out
   StdIn=/dev/null
   StdOut=/public/gkxiao/software/a3fe/4zlz_gaff2/bound/ensemble_equilibration_2/somd-array-gpu-46054.4294967294.out
   Power=
   TresPerNode=gres/gpu:1

check the slurm task:

cat bound/ensemble_equilibration_2/ensemble_equil_bound.sh
#!/bin/bash
#SBATCH -o somd-array-gpu-%A.%a.out
#SBATCH -n 1
#SBATCH --time 24:00:00
#SBATCH --gres=gpu:1

python -c 'from a3fe.run.system_prep import slurm_ensemble_equilibration_bound; slurm_ensemble_equilibration_bound()'

• Two gmx mdrun processes each consuming ​​~32 CPU cores​​ (3245% CPU usage via top).

Tasks: 1511 total,   4 running, 1504 sleeping,   0 stopped,   3 zombie
%Cpu(s): 50.9 us,  0.1 sy,  0.0 ni, 49.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  6.1/257752.1 [||||||                                                                                              ]
MiB Swap:  0.0/8192.0   [                                                                                                    ]

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2909717 gkxiao    20   0 9953.0m 372272 145244 R  3245   0.1      6,09 /usr/local/gromacs/bin/gmx mdrun -deffnm gromacs -c /public/gkxiao/software/a3fe/4zlz_gaff2/bound/ensemble_equilibration_2/gromacs_out.gro
2909711 gkxiao    20   0 9941.9m 292484 142192 R  3242   0.1      6,21 /usr/local/gromacs/bin/gmx mdrun -deffnm gromacs -c /public/gkxiao/software/a3fe/4zlz_gaff2/bound/ensemble_equilibration_1/gromacs_out.gro

• GPUs at ​​1% utilization​​ with minimal VRAM usage (392MB/24GB per GPU via nvidia-smi).

Thu Jun 12 15:20:33 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090 D      On  |   00000000:01:00.0 Off |                  Off |
| 30%   48C    P0             64W /  425W |     415MiB /  24564MiB |      1%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 4090 D      On  |   00000000:41:00.0 Off |                  Off |
| 30%   53C    P0             61W /  425W |     415MiB /  24564MiB |      1%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      4636      G   /usr/lib/xorg/Xorg                              4MiB |
|    0   N/A  N/A   2909711      C   /usr/local/gromacs/bin/gmx                    392MiB |
|    1   N/A  N/A      4636      G   /usr/lib/xorg/Xorg                              4MiB |
|    1   N/A  N/A   2909717      C   /usr/local/gromacs/bin/gmx                    392MiB |
+-----------------------------------------------------------------------------------------+

Expected Outcome

Implementing these changes should:
• Raise GPU utilization to ​​>90%​​ .
• Reduce CPU core usage per process to ​​<16 cores​​, balancing workload.
• Improve simulation throughput by ​​5–10×​​ based on GROMACS benchmarks.

[Roy-Haolin-Du]

Hi, thanks for your information.

To determine whether this is an a3fe issue or a GROMACS GPU configuration problem.
Let's first manually test GROMACS MD submission to see if GPU acceleration works properly by running a fast MD simulation.

Since GROMACS 2025.1 is quite new, some GPU acceleration settings might have changed. I recommend checking the GROMACS 2025.1 release note (https://manual.gromacs.org/documentation/2025.1/release-notes/2025/2025.1.html#) for any GPU-related changes.
You can see GPU checked in gromacs? GPU task assignment in gromacs.log file?

Also, could you share your input & bound folders? I'd like to examine the files more carefully to help diagnose the issue.
This would be super helpful!

Thank you!

[gkxiao]

Hi Roy,
Here is the input & bound folders: a3fe_issue_50.tar.gz.
Thank you very much,
Gaokeng Xiao

[Roy-Haolin-Du]

Thanks for the files you provided.

I’ve submitted your job on my workstation, and everything looks good and is running smoothly.

Thu Jun 12 14:14:39 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4060        Off |   00000000:19:00.0 Off |                  N/A |
| 44%   65C    P0             N/A /  115W |     165MiB /   8188MiB |     55%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce GTX 980 Ti      Off |   00000000:1A:00.0 Off |                  N/A |
| 22%   57C    P8             16W /  250W |       9MiB /   6144MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+                                                                               
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      2013      G   /usr/lib/xorg/Xorg                              4MiB |
|    0   N/A  N/A   3990982      C   /usr/local/gromacs-2023.1/bin/gmx             142MiB |
|    1   N/A  N/A      2013      G   /usr/lib/xorg/Xorg                              3MiB |
+-----------------------------------------------------------------------------------------+

GROMACS version:    2023.1
Precision:          mixed
Memory model:       64 bit
MPI library:        thread_mpi
OpenMP support:     enabled (GMX_OPENMP_MAX_THREADS = 128)
GPU support:        CUDA
NB cluster size:    8
SIMD instructions:  AVX_512
CPU FFT library:    fftw-3.3.8-sse2-avx-avx2-avx2_128-avx512
GPU FFT library:    cuFFT
Multi-GPU FFT:      none
RDTSCP usage:       disabled
TNG support:        enabled
Hwloc support:      disabled
Tracing support:    disabled
C compiler:         /usr/bin/cc GNU 11.4.0
C compiler flags:   -fexcess-precision=fast -funroll-all-loops -mavx512f -mfma -mavx512vl -mavx512dq -mavx512bw -Wno-missing-field-initializers -O3 -DNDEBUG
C++ compiler:       /usr/bin/c++ GNU 11.4.0
C++ compiler flags: -fexcess-precision=fast -funroll-all-loops -mavx512f -mfma -mavx512vl -mavx512dq -mavx512bw -Wno-missing-field-initializers -Wno-cast-function-type-strict -fopenmp -O3 -DNDEBUG
BLAS library:       External - detected on the system
LAPACK library:     External - detected on the system
CUDA compiler:      /usr/local/cuda/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2024 NVIDIA Corporation;Built on Wed_Aug_14_10:10:22_PDT_2024;Cuda compilation tools, release 12.6, V12.6.68;Build cuda_12.6.r12.6/compiler.34714021_0
CUDA compiler flags:-std=c++17;--generate-code=arch=compute_50,code=sm_50;--generate-code=arch=compute_52,code=sm_52;--generate-code=arch=compute_60,code=sm_60;--generate-code=arch=compute_61,code=sm_61;--generate-code=arch=compute_70,code=sm_70;--generate-code=arch=compute_75,code=sm_75;--generate-code=arch=compute_80,code=sm_80;--generate-code=arch=compute_86,code=sm_86;--generate-code=arch=compute_89,code=sm_89;--generate-code=arch=compute_90,code=sm_90;-Wno-deprecated-gpu-targets;--generate-code=arch=compute_53,code=sm_53;--generate-code=arch=compute_80,code=sm_80;-use_fast_math;-Xptxas;-warn-double-usage;-Xptxas;-Werror;-D_FORCE_INLINES;-fexcess-precision=fast -funroll-all-loops -mavx512f -mfma -mavx512vl -mavx512dq -mavx512bw -Wno-missing-field-initializers -Wno-cast-function-type-strict -fopenmp -O3 -DNDEBUG
CUDA driver:        12.60
CUDA runtime:       12.60

Coincidentally, we have the same CUDA version and driver version.
So the difference might come from GROMACS 2025.1(I use 2023.1), which could have introduced some changes.

I’d suggest checking the changes introduced in GROMACS 2025.1, or alternatively, try using an earlier version.

Best,

[gkxiao]

Thank you for your help, I will try using an earlier version.

Best Regards,

[gkxiao]

Hi Roy,

I’ve been investigating the low CPU utilization observed in the ensemble_equilibration logs. I identified that the gmx mdrun command being called by a3fe was simply:

 gmx mdrun -deffnm gromacs -c gromacs_out.gro

To significantly improve GPU utilization (now consistently above 73%), I modified the command to:

gmx mdrun -nb gpu -pme gpu -bonded gpu -update gpu -deffnm gromacs -c gromacs_out.gro

Here is output:

                       :-) GROMACS - gmx mdrun, 2024 (-:

Executable:   /public/gkxiao/software/gromacs/gromacs_2024build20250613/bin/gmx
Data prefix:  /public/gkxiao/software/gromacs/gromacs_2024build20250613
Working dir:  /public/gkxiao/software/a3fe/4zlz_gaff2/bound/ensemble_equilibration_1
Command line:
  gmx mdrun -nb gpu -pme gpu -bonded gpu -update gpu -deffnm gromacs -c gromacs_out.gro


Back Off! I just backed up gromacs.log to ./#gromacs.log.2#
Reading file gromacs.tpr, VERSION 2024 (single precision)
Changing nstlist from 20 to 50, rlist from 0.834 to 0.924

Update groups can not be used for this system because atoms that are (in)directly constrained together are interdispersed with other atoms

1 GPU selected for this run.
Mapping of GPU IDs to the 2 GPU tasks in the 1 rank on this node:
  PP:0,PME:0
PP tasks will do (non-perturbed) short-ranged and most bonded interactions on the GPU
PP task will update and constrain coordinates on the GPU
PME tasks will do all aspects on the GPU
Using 1 MPI thread
Using 64 OpenMP threads 


Back Off! I just backed up gromacs.xtc to ./#gromacs.xtc.2#

Back Off! I just backed up gromacs.edr to ./#gromacs.edr.2#
starting mdrun ''
2500000 steps,   5000.0 ps.

Writing final coordinates.

NOTE: 22 % of the run time was spent in pair search,
      you might want to increase nstlist (this has no effect on accuracy)

               Core t (s)   Wall t (s)        (%)
       Time:    34068.464      532.378     6399.3
                 (ns/day)    (hour/ns)
Performance:      811.453        0.030

GROMACS reminds you: "Can I have everything louder than everything else?" (Deep Purple)

I’ve reviewed the a3fe documentation but haven’t been able to locate the specific parameters to adjust for optimal GPU usage. Could you point me to the relevant section or provide guidance on what parameters to explore?

Thanks!

[Roy-Haolin-Du]

Hi! Thanks for your good thinking.

Adding explicit GPU acceleration flags dramatically improves performance but current BioSimSpace GROMACS implementation is default configuration.

If want optimize behavior, need modify the _generate_args() method in _gromacs.py.
https://github.com/OpenBioSim/biosimspace/blob/8ba863f612baee764c8b5935df9c40dbc2daef1e/python/BioSimSpace/Sandpit/Exscientia/Process/_gromacs.py#L513C5-L526C49.

For example:

def _generate_args(self):
    """Generate the dictionary of command-line arguments."""
    self.clearArgs()
    
    self.setArg("mdrun", True)
    self.setArg("-deffnm", self._name)
    self.setArg("-c", self._crd_file)
    
    # Add GPU optimization flags
    self.setArg("-nb", "gpu")      # Non-bonded on GPU
    self.setArg("-pme", "gpu")     # PME on GPU
    self.setArg("-bonded", "gpu")  # Bonded interactions on GPU
    self.setArg("-update", "gpu")  # Coordinate updates on GPU
    
    # Existing protocol-specific arguments
    if isinstance(self._protocol, (_Protocol.Metadynamics, _Protocol.Steering)):
        self.setArg("-plumed", "plumed.dat")

By the way, the detailed GROMACS Parameter Setting Process in a3fe is as follows:

1. Creating a gromacs object in a3fe system_prep.py:
   

   a3fe/a3fe/run/system_prep.py

   Line 722 in f8e1948

   process = _BSS.Process.Gromacs(system, protocol, work_dir=work_dir)

   
   2.Using _generate_args() method generate parameters.
   https://github.com/OpenBioSim/biosimspace/blob/8ba863f612baee764c8b5935df9c40dbc2daef1e/python/BioSimSpace/Sandpit/Exscientia/Process/_gromacs.py#L513C5-L526C49.
   3.setArg() method storage the parameter, which comes from parent class Process in:
   https://github.com/OpenBioSim/biosimspace/blob/8ba863f612baee764c8b5935df9c40dbc2daef1e/python/BioSimSpace/Sandpit/Exscientia/Process/_process.py#L1523-L1542.
   4.Calling process.start() and parameter conversion mechanism (getArgStringList()) from _gromacs.py to execute command.

which helps to understand the flow.

Have a try and let me now if it works or not.

By the way, @lohedges Lester, sorry to trouble you! Can you help me confirm if this is the right way to add GPU parameters in GROMACS? or more easier way?
Much appreciated and many thanks in advance!

Best,

Roy

[lohedges]

Hi @Roy-Haolin-Du. The process objects from the core BioSImSpace code have an extra_args keyword argument that does what you want. This is a dictionary that can be used to set additional command-line options, or overrride the default values. If necessary, I can try to port this over to the the sandpit. (I'm not sure why it wasn't added.) I wouldn't directly modify the _generate_args method since we don't want to break any code that is used by Recursion. (The sandpit should only get bug fixes, rather than new features, which is probably why the extra_args doesn't appear there.)

[lohedges]

You can also update the arguments after the process is created using process.setArg, so that might be the easiest way for now.

[Roy-Haolin-Du]

Thanks, Lester! Thanks so much for your suggestion!
@gkxiao , for now, you can temporarily modify your local code to suit your own GPU usage.
In system_prep.py:

a3fe/a3fe/run/system_prep.py

Line 722 in f8e1948

process = _BSS.Process.Gromacs(system, protocol, work_dir=work_dir)

.

...
process = _BSS.Process.Gromacs(system, protocol, work_dir=work_dir)
process.setArg("-nb", "gpu")
process.setArg("-pme", "gpu")
process.setArg("-bonded", "gpu") 
process.setArg("-update", "gpu")
process.start()
...

That will give you gmx mdrun -deffnm gromacs -c gromacs_out.gro -nb gpu -pme gpu -bonded gpu -update gpu.
This is probably the most convenient method for now.

And we'll take time to think carefully about how to integrate this in a more elegant way later on.

Cheers,

Roy

[gkxiao]

Hi Roy and the team,

Just wanted to give a huge thanks to Roy and everyone who contributed to this discussion! I really appreciate the collaborative effort to resolve this.

I’ve been testing the solution, and I’ve found that the “-nb gpu -pme gpu -bonded gpu -update gpu“ flag is only effective during the stage of ensemble equilibration. Prior to this stage, adding "-nb gpu" flag is allowed and adding “-pme gpu -bonded gpu” results in an error. This means we can’t directly add the GPU flags within system_prep.py; instead, we need to dynamically add them during the ensemble equilibration phase.

Here are some experiments:

• "-nb gpu -pme gpu -bonded gpu" flag is used:

INFO - 2025-06-13 19:52:22,957 - Leg (type = BOUND)_1 - Waiting for job Job (virtual_job_id = 1, slurm_job_id= 46125), status = JobStatus.QUEUED to complete
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /public/gkxiao/software/a3fe/4zlz_gaff2/run.py:3 in <module>                                     │
│                                                                                                  │
│    1 import a3fe as a3                                                                           │
│    2 calc = a3.Calculation(ensemble_size=3)                                                      │
│ ❱  3 calc.setup()                                                                                │
│    4 calc.get_optimal_lam_vals()                                                                 │
│    5 calc.run(adaptive=False, runtime = 5) # Run non-adaptively for 5 ns per replicate           │
│    6 calc.wait()                                                                                 │
│                                                                                                  │
│ /public/gkxiao/software/anaconda3/envs/a3fe/lib/python3.12/site-packages/a3fe/run/calculation.py │
│ :211 in setup                                                                                    │
│                                                                                                  │
│   208 │   │   │   │   stream_log_level=self.stream_log_level,                                    │
│   209 │   │   │   )                                                                              │
│   210 │   │   │   self.legs.append(leg)                                                          │
│ ❱ 211 │   │   │   leg.setup(configs[leg_type])                                                   │
│   212 │   │                                                                                      │
│   213 │   │   # Save the state                                                                   │
│   214 │   │   self.setup_complete = True                                                         │
│                                                                                                  │
│ /public/gkxiao/software/anaconda3/envs/a3fe/lib/python3.12/site-packages/a3fe/run/leg.py:232 in  │
│ setup                                                                                            │
│                                                                                                  │
│    229 │   │   if self.prep_stage == _PreparationStage.PARAMETERISED:                            │
│    230 │   │   │   self.solvate_input(slurm=cfg.slurm)  # This also adds ions                    │
│    231 │   │   if self.prep_stage == _PreparationStage.SOLVATED:                                 │
│ ❱  232 │   │   │   system = self.minimise_input(slurm=cfg.slurm)                                 │
│    233 │   │   if self.prep_stage == _PreparationStage.MINIMISED:                                │
│    234 │   │   │   system = self.heat_and_preequil_input(slurm=cfg.slurm)                        │
│    235 │   │   if self.prep_stage == _PreparationStage.PREEQUILIBRATED:                          │
│                                                                                                  │
│ /public/gkxiao/software/anaconda3/envs/a3fe/lib/python3.12/site-packages/a3fe/run/leg.py:510 in  │
│ minimise_input                                                                                   │
│                                                                                                  │
│    507 │   │   │   │   self.leg_type                                                             │
│    508 │   │   │   ):                                                                            │
│    509 │   │   │   │   if not _os.path.isfile(f"{self.input_dir}/{file}"):                       │
│ ❱  510 │   │   │   │   │   raise RuntimeError(                                                   │
│    511 │   │   │   │   │   │   f"SLURM job failed to produce {file}. Please check the output of  │
│    512 │   │   │   │   │   │   f"last slurm log in {self.input_dir} directory for error."        │
│    513 │   │   │   │   │   )                                                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯

And then check the output of SLURM log:

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ in <module>:1                                                                │
│                                                                              │
│ /public/gkxiao/software/anaconda3/envs/a3fe/lib/python3.12/site-packages/a3f │
│ e/run/system_prep.py:775 in slurm_minimise_bound                             │
│                                                                              │
│   772                                                                        │
│   773 def slurm_minimise_bound() -> None:                                    │
│   774 │   """Perform minimisation for the bound leg"""                       │
│ ❱ 775 │   minimise_input(leg_type=_LegType.BOUND, input_dir=".", output_dir= │
│   776                                                                        │
│   777                                                                        │
│   778 def slurm_minimise_free() -> None:                                     │
│                                                                              │
│ /public/gkxiao/software/anaconda3/envs/a3fe/lib/python3.12/site-packages/a3f │
│ e/run/system_prep.py:496 in minimise_input                                   │
│                                                                              │
│   493 │   # Minimise                                                         │
│   494 │   print(f"Minimising input structure with {cfg.steps} steps...")     │
│   495 │   protocol = _BSS.Protocol.Minimisation(steps=cfg.steps)             │
│ ❱ 496 │   minimised_system = run_process(solvated_system, protocol)          │
│   497 │                                                                      │
│   498 │   # Save, renaming the velocity property to foo so avoid saving velo │
│   499 │   # velocities sometimes causes issues with the size of the floats o │
│                                                                              │
│ /public/gkxiao/software/anaconda3/envs/a3fe/lib/python3.12/site-packages/a3f │
│ e/run/system_prep.py:737 in run_process                                      │
│                                                                              │
│   734 │   │   print(process.stderr())                                        │
│   735 │   │   process.getStdout()                                            │
│   736 │   │   process.getStderr()                                            │
│ ❱ 737 │   │   raise _BSS._Exceptions.ThirdPartyError("The process failed.")  │
│   738 │   system = process.getSystem(block=True)                             │
│   739 │   if system is None:                                                 │
│   740 │   │   print(process.stdout())                                        │
╰──────────────────────────────────────────────────────────────────────────────╯
ThirdPartyError: The process failed.
Loading solvated system...
Minimising input structure with 1000 steps...
MPI rank:    0 (out of 4)

Inconsistency in user input:
Cannot compute PME interactions on a GPU, because:
  PME GPU does not support:
    Non-dynamical integrator (use md, sd, etc).

For more information and tips for troubleshooting, please check the GROMACS
website at https://manual.gromacs.org/current/user-guide/run-time-errors.html
-------------------------------------------------------
None
All output has been redirected to the stdout stream!
None

• "-nb gpu -bonded gpu" flag is used:

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ in <module>:1                                                                │
│                                                                              │
│ /public/gkxiao/software/anaconda3/envs/a3fe/lib/python3.12/site-packages/a3f │
│ e/run/system_prep.py:775 in slurm_minimise_bound                             │
│                                                                              │
│   772                                                                        │
│   773 def slurm_minimise_bound() -> None:                                    │
│   774 │   """Perform minimisation for the bound leg"""                       │
│ ❱ 775 │   minimise_input(leg_type=_LegType.BOUND, input_dir=".", output_dir= │
│   776                                                                        │
│   777                                                                        │
│   778 def slurm_minimise_free() -> None:                                     │
│                                                                              │
│ /public/gkxiao/software/anaconda3/envs/a3fe/lib/python3.12/site-packages/a3f │
│ e/run/system_prep.py:496 in minimise_input                                   │
│                                                                              │
│   493 │   # Minimise                                                         │
│   494 │   print(f"Minimising input structure with {cfg.steps} steps...")     │
│   495 │   protocol = _BSS.Protocol.Minimisation(steps=cfg.steps)             │
│ ❱ 496 │   minimised_system = run_process(solvated_system, protocol)          │
│   497 │                                                                      │
│   498 │   # Save, renaming the velocity property to foo so avoid saving velo │
│   499 │   # velocities sometimes causes issues with the size of the floats o │
│                                                                              │
│ /public/gkxiao/software/anaconda3/envs/a3fe/lib/python3.12/site-packages/a3f │
│ e/run/system_prep.py:737 in run_process                                      │
│                                                                              │
│   734 │   │   print(process.stderr())                                        │
│   735 │   │   process.getStdout()                                            │
│   736 │   │   process.getStderr()                                            │
│ ❱ 737 │   │   raise _BSS._Exceptions.ThirdPartyError("The process failed.")  │
│   738 │   system = process.getSystem(block=True)                             │
│   739 │   if system is None:                                                 │
│   740 │   │   print(process.stdout())                                        │
╰──────────────────────────────────────────────────────────────────────────────╯
ThirdPartyError: The process failed.
Loading solvated system...
Minimising input structure with 1000 steps...
MPI rank:    0 (out of 4)

Inconsistency in user input:
Bonded interactions can not be computed on a GPU:
  Cannot compute bonded interactions on a GPU, because GPU implementation
requires a dynamical integrator (md, sd, etc).

For more information and tips for troubleshooting, please check the GROMACS
website at https://manual.gromacs.org/current/user-guide/run-time-errors.html
-------------------------------------------------------
None
All output has been redirected to the stdout stream!
None

• "-nb gpu" flag is used:
  Yes, it works and enter the stage of ensemble equilibration. However, without GPU acceleration for the pme and bonded interactions, ensemble equilibration is significantly prolonged.

Therefore, the simulation flags still requires a staged approach:

• Energy Minimization: CPU handles PME (-pme cpu), while GPU handles non-bonded interactions (-nb gpu).
• Equilibration/Production Simulation: Enables full GPU acceleration (-nb gpu -pme gpu -bonded gpu -update gpu).

Thanks again for your help!

Gaokeng

[Roy-Haolin-Du]

Hi Gaokeng,

Thank you very much for your helpful testing!
Your results reveal that GPU acceleration in GROMACS has protocol-specific limitations:

Energy Minimization Stage
✅ -nb gpu (Non-bonded on GPU): Works
❌ -pme gpu, -bonded gpu: Cause errors

Equilibration / Production Stage
✅ -nb gpu, -pme gpu, -bonded gpu, -update gpu: All work perfectly

Thanks again and this is really valuable test.

Perhaps the code could be motified like this by adding a simple condition:

process = _BSS.Process.Gromacs(system, protocol, work_dir=work_dir)

#For non-bonded interactions is always safe to use
process.setArg("-nb", "gpu")  

#Additional options (e.g., PME, bonded, update) should only be applied outside the minimization stage
if not isinstance(protocol, _BSS.Protocol.Minimisation): 
    process.setArg("-pme", "gpu")
    process.setArg("-bonded", "gpu") 
    process.setArg("-update", "gpu")

process.start()

Thanks and have a good day!

Cheers,

Roy

[gkxiao]

Hi Roy,

Now, It works !

Thank you very much,

Gaokeng