Domyn is training a 1T-parameter MoE on Europe's Jupiter supercomputer, and published the recipe

The EUROPA consortium, led by Domyn with Fraunhofer IAIS and Fraunhofer IIS, has started pretraining a roughly one-trillion-parameter Mixture-of-Experts model on Jupiter, the EuroHPC exascale machine at Forschungszentrum Jülich. Its engineering write-up explains how the team raised throughput from 90 to about 1,300 tokens per second per GPU on 384 GH200 superchips.

The training run is in progress. The consortium’s stated goal, under the Frontier AI Grand Challenge it won in June, is an open frontier model released with weights and training recipes. Nothing is released yet.

The model and the machine

Setting	Value
Total parameters	951.9B (32.8B active per token)
Experts	384 routed, top-8
Sequence length	65,536 tokens
Hardware per replica	96 nodes, 384 GH200
Optimizer	Muon for about 99.5% of parameters, AdamW for the rest

The hard part is that Jupiter is shared and built for simulation. Domyn has user-level access only and cannot change the kernel, interconnect or node memory setup. Compute nodes have 120 GB of Grace memory each, and the post says InfiniBand between nodes is six times slower than links between GPUs in the same node.

What moved the number

• Layout. 384-way fully sharded data parallelism, with expert parallelism kept to four GPUs inside a node so the MoE all-to-all stays on NVLink.
• Longer sequences. Going from 8,192 to 65,536 tokens raised compute per byte of weights gathered. Muon, which keeps one optimizer state instead of Adam’s two, freed the memory to do it.
• Offloading. Activations and optimizer state move to Grace memory over NVLink-C2C, a cousin of activation checkpointing that moves saved activations instead of recomputing all of them.
• Less traffic. FP8 weight gathers, BF16 gradient all-to-alls and NCCL_MAX_CTAS=16 to leave more SMs for compute. Data balancing alone gave 13%.

At the final setting one replica processes about 43B tokens a day. With four replicas on 1,536 GPUs at the 89% scaling efficiency it measured, Domyn says the job would process about 150B tokens a day. Fixes found along the way went into PyTorch and TorchTitan, the library PyTorch also tuned for Muse Spark Open.

Domyn lists what it has not settled: the long-run quality cost of its group-limited routing, FP8 for expert matrix math, and whether pipeline parallelism would have been faster.

Why this matters

Most frontier MoE recipes assume dedicated Blackwell clusters; this one shows what changes on older, shared Hopper hardware where you cannot touch the system. If you train large models on university or national HPC allocations, the host-memory and bandwidth tricks here are reusable today, and the upstream fixes mean you get some of them just by updating PyTorch and TorchTitan.