MultiAct

Improving compositional text-to-motion generation with inference-time cross-attention optimization.

MultiAct helps a pretrained HumanML3D Motion Diffusion Model (MDM) express multiple actions from a single prompt. When a baseline sample captures the dominant action but neglects another—for example, walking backward without raising its hands—MultiAct emphasizes the missing action token during reverse diffusion. The method operates at inference time and does not retrain the base model.

Results

Each comparison shows the MDM baseline on the left and MultiAct on the right. Select the links below each preview to watch the original MP4s.

“A person walks backwards while raising his hands up”

Baseline MP4 · MultiAct MP4

“A person dribbles a ball while moving backward”

Baseline MP4 · MultiAct MP4

How it works

MultiAct adapts Attend-and-Excite to motion diffusion. For selected early reverse-diffusion timesteps, it:

1. records the decoder's frame-to-text cross-attention;
2. measures attention to a user-selected action token;
3. optimizes the current noisy motion latent to increase that token's attention; and
4. resumes the standard denoising step with the updated latent.

The base model weights remain frozen. The key controls are the emphasized token, decoder layer, number of latent-optimization steps, optimizer step size, and diffusion-step cutoff.

Setup

Generation requires a CUDA-capable GPU because the current inference path calls .cuda() directly. The recommended environment uses Python 3.12 and uv.

uv venv --python 3.12
source .venv/bin/activate

# Install a CUDA-enabled PyTorch build suitable for your system first.
uv pip install torch torchvision torchaudio

uv pip install -r requirements.txt
uv pip install click einops matplotlib scipy scikit-learn spacy tqdm transformers wandb pytest ruff gdown
python -m spacy download en_core_web_sm

Install ffmpeg if you want to render MP4 outputs. The original Conda environment is also available in environment.yml.

Required assets

Before running inference, make sure these files are available:

body_models/smpl/SMPL_NEUTRAL.pkl
dataset/HumanML3D/Mean.npy
dataset/HumanML3D/Std.npy
models/ofir_decoder/model000430000.pt
models/ofir_decoder/args.json

The SMPL, GloVe, and text-to-motion evaluator dependencies can be downloaded with:

bash prepare/download_dependencies.sh

Model checkpoints are not all downloaded by that script. Place each checkpoint next to its matching args.json.

Verify CUDA and the expected assets:

uv run python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'no CUDA')"
test -f body_models/smpl/SMPL_NEUTRAL.pkl
test -f dataset/HumanML3D/Mean.npy
test -f dataset/HumanML3D/Std.npy
test -f models/ofir_decoder/model000430000.pt
test -f models/ofir_decoder/args.json

Generate motion

Run commands from the repository root. tests/simple_generation.py is the preferred CLI for local experiments.

Baseline

uv run python tests/simple_generation.py generate \
  --model models/ofir_decoder/model000430000.pt \
  --prompt "a person walks backwards while raising his hands up" \
  --seed 1304 \
  --save_dir out/baseline_raising

Add --save_npy to save the raw generated motion tensor alongside the MP4.

MultiAct

Choose an action word that appears in the prompt and pass it with --ae:

uv run python tests/simple_generation.py generate3 \
  --model models/ofir_decoder/model000430000.pt \
  --prompt "a person walks backwards while raising his hands up" \
  --ae raising \
  --ae_layer 5 \
  --ae_steps 12 \
  --step_size 0.01 \
  --apply_ae_steps 47 \
  --seed 1304 \
  --batch 1 \
  --save_dir out/multiact_raising

Use the same prompt and seed for a controlled baseline comparison.

MultiAct controls

Option	Meaning	Typical value
--ae	Exact prompt token to emphasize	raising
--ae_layer	Decoder cross-attention layer used by the loss	3–5
--ae_steps	Adam steps at each selected diffusion timestep	6–12
--step_size	Latent optimizer learning rate	0.01–0.02
--apply_ae_steps	Diffusion timestep cutoff for applying MultiAct	46–47
--batch	Number of stochastic samples	1–3

Settings are prompt-dependent. Useful built-in configurations are listed in COMMANDS near the end of tests/simple_generation.py; run one by index with:

uv run python tests/simple_generation.py command \
  --model models/ofir_decoder/model000430000.pt \
  --index 19 \
  --batch 1 \
  --save_dir out/command_19

Evaluation

The eval command supports repeated samples, raw tensor output, offline runs, alternate attention losses, and optional second-token optimization.

uv run python tests/simple_generation.py eval \
  --model models/ofir_decoder/model000430000.pt \
  --prompt "a person walks backwards while raising his hands up" \
  --ae raising \
  --ae_layer 5 \
  --ae_steps 12 \
  --step_size 0.01 \
  --apply_ae_steps 47 \
  --batch 3 \
  --offline \
  --save_dir out/eval_raising

Tests and quality checks

Run the fast test suite without GPU-heavy integration cases:

uv run pytest tests/ -v -m "not integration"

Run the generation integration tests on a CUDA machine with the checkpoint installed:

uv run pytest tests/test_simple_generation_cli.py -v -m integration

Lint and format with Ruff:

uv run ruff check .
uv run ruff format .

Repository map

Path	Purpose
tests/simple_generation.py	Main generation, comparison, and evaluation CLI
sample/generate.py	Lower-level sampling and motion conversion helpers
model/attend_excite_utils.py	MultiAct latent optimization and attention losses
model/mdm_any.py	Integration of MultiAct into the MDM denoising pass
model/mdm_base.py	Core MDM and attention-storing decoder components
models/ofir_decoder/	Expected local checkpoint directory

Generated samples should be kept under out/ or another explicit output directory.

Acknowledgements

This repository builds on Human Motion Diffusion Model (MDM). If you use the underlying model, please cite:

@article{tevet2022human,
  title={Human Motion Diffusion Model},
  author={Tevet, Guy and Raab, Sigal and Gordon, Brian and Shafir, Yonatan and Bermano, Amit H and Cohen-Or, Daniel},
  journal={arXiv preprint arXiv:2209.14916},
  year={2022}
}

It also incorporates components and ideas from guided-diffusion, MotionCLIP, text-to-motion, ACTOR, joints2smpl, and MoDi.

License

The repository is distributed under the MIT License. Its dependencies, models, and datasets may have separate license terms.