ZeyuLing/Motius-DiffuseStyleGesture-BEAT

DiffuseStyleGesture+ Model Card

DiffuseStyleGesture is the IJCAI 2023 work Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models. Motius integrates the released DiffuseStyleGesture+ BEAT checkpoint, which adds text semantics, speaker ID, 30 seed frames, and rolling 150-frame diffusion windows.

• Paper: https://arxiv.org/abs/2305.04919
• DSG+ paper: https://arxiv.org/abs/2308.13879
• Official source: https://github.com/YoungSeng/DiffuseStyleGesture
• Motius release: https://huggingface.co/ZeyuLing/Motius-DiffuseStyleGesture-BEAT
• Pipeline: motius.pipelines.diffusestylegesture.DiffuseStyleGesturePipeline

Release Snapshot

Item	Value
Task	Speech-to-Gesture
Training data	BEAT speakers 2 and 10
Benchmark	BEAT2 English speaker 2
Input	WavLM/acoustic 1133D + aligned FastText 301D + speaker ID
Native motion	BEAT 75-joint BVH
Motius motion	SMPL-X 55-joint axis-angle + zero expression + translation
Motion rate	30 FPS
Window	150 frames, 30 seed frames, 120-frame stride
Diffusion	Official 1,000-step sampler, seed 123456
Checkpoint	model001080000.pt
Checkpoint SHA-256	2e25d8b257b44e5110a61713476a146e99c34926c763b5dd1c18a6884a857c77
Runtime revision	9649995cdd0b8ce0d42bd02567b2bd03829a0540
Inference-pinned artifact revision	8707378e3835ac2d7e17dc977272640190a1daf5
License	MIT model/runtime; CC-BY-SA-3.0 FastText subset

Setup

bash tools/setup_diffusestylegesture_env.sh
git clone https://github.com/YoungSeng/DiffuseStyleGesture \
  ref_repo/DiffuseStyleGesture
git -C ref_repo/DiffuseStyleGesture checkout \
  9649995cdd0b8ce0d42bd02567b2bd03829a0540

The exact original-format WavLM checkpoint is required. The Transformers conversion is not used: on the released DSG+ parity clip its WavLM block has only 0.96376 cosine similarity, while the original checkpoint reproduces the released features at 0.9999999999 cosine similarity.

gdown 12-cB34qCTvByWT-QtOcZaqwwO21FLSqU \
  -O outputs/checkpoints/diffusestylegesture/WavLM-Large.pt

The FastText source is the official crawl-300d-2M.vec.zip. The release contains the 790 vectors used by the fixed benchmark population as beat2_speaker2_fasttext_vocab.npz (one source token, berties, is OOV). The DSG+ checkpoint and source use MIT terms; this compact vector subset retains FastText's CC-BY-SA-3.0 terms.

Usage

Build the official conditioning features, then run inference:

python tools/build_diffusestylegesture_beat2_features.py \
  --runtime-root ref_repo/DiffuseStyleGesture/BEAT-TWH-main \
  --manifest outputs/evaluation/speech_to_gesture/beat2/_resources/hliu1997_beat2_speaker2_test/manifest.json \
  --data-root outputs/evaluation/speech_to_gesture/beat2/_resources/hliu1997_beat2_speaker2_test/beat_english_v2.0.0 \
  --wavlm-checkpoint outputs/checkpoints/diffusestylegesture/WavLM-Large.pt \
  --fasttext-compact outputs/checkpoints/diffusestylegesture/beat2_speaker2_fasttext_vocab.npz \
  --out-dir outputs/evaluation/speech_to_gesture/beat2/diffusestylegesture/features

python tools/eval_diffusestylegesture_beat2.py \
  --runtime-root ref_repo/DiffuseStyleGesture/BEAT-TWH-main \
  --artifact ZeyuLing/Motius-DiffuseStyleGesture-BEAT \
  --manifest outputs/evaluation/speech_to_gesture/beat2/_resources/hliu1997_beat2_speaker2_test/manifest.json \
  --data-root outputs/evaluation/speech_to_gesture/beat2/_resources/hliu1997_beat2_speaker2_test/beat_english_v2.0.0 \
  --feature-dir outputs/evaluation/speech_to_gesture/beat2/diffusestylegesture/features \
  --out-dir outputs/evaluation/speech_to_gesture/beat2/diffusestylegesture/predictions

DSG+ writes a native 30 FPS, Y-up BEAT BVH. Motius applies the public FreeTalker BEAT-to-SMPL-X joint map with identity axes and without FreeTalker's historical six-frame downsample, which is only appropriate for original 120 FPS BEAT files.

Conditioning stays on the official audio-feature clock. The 15 benchmark WAVs produce 6–16 more feature frames than their paired SMPL-X references; Motius does not interpolate or time-warp those features. It runs diffusion for the native feature length and crops only the final motion to the reference length.

Evaluation Results

Evaluator	Clips	FGD ↓	BC ↑	Diversity	uTMR FID ↓	uTMR Paired Dist. ↓
BEAT2 Official	15	65.9364	6.3858	12.2055	0.0930	16.0251

The uTMR columns consume only canonical 30 FPS SMPL-22 joint positions. They ignore text and expression coefficients; the official metric FK explicitly sets the SMPL-X translation vector to zero. Each long clip is split into balanced windows no longer than 20 seconds; embeddings are L2-normalized only for FID. Paired distance and embedding diversity use native uTMR features.

These values do not reproduce the DSG+ paper's FGD 14.461: that number uses the GENEA/Talking-With-Hands challenge autoencoder and test population, whereas this row uses the fixed Motius BEAT2 speaker-2 SMPL-X protocol. The upstream authors also document that the manual BVH-to-SMPL-X conversion can depress motion realism on BEAT2, even for reconstruction. Accordingly, this released row is a verified cross-protocol Motius result rather than a paper-number claim. The exact GENEA autoencoder checkpoint and Table 1 split were not released, so the public artifacts are insufficient for an exact 14.461 reproduction.

Preview

Open the audio-synchronized 15-case Three.js comparison. It provides shared camera and timeline controls for BEAT2 GT, Language of Motion, EMAGE, and DiffuseStyleGesture+, plus native SMPL-X NPZ downloads.

Citation

@inproceedings{yang2023diffusestylegesture,
  title={DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models},
  author={Yang, Sicheng and Wu, Zhiyong and Li, Minglei and Zhang, Zhensong and Hao, Lei and Bao, Weihong and Cheng, Ming and Xiao, Long},
  booktitle={IJCAI},
  year={2023}
}