Benedikt Droste, David Fitzek, Ruben Härle, Lukas Helff, Maximilian Idahl, Alex Jude, Abbas Goher Khan, Maurice Kraus, Timm Ruland, Richard Rutmann, Sebastian Sztwiertnia, Markus Frey, Daniil Gurgurov, Jan Pfister, Tom Röhr, Sebastian von Rohrscheidt, Jörg Bienert, Nicolas Flores-Herr, Simon Gottschalk, Andreas Hotho, Kristian Kersting, Joachim Köhler, Alexander Löser, Wolfgang Nejdl, Simon Ostermann, Jan Plogsties, Patrick Putzky, Mehdi Ali, Michael Fromm, Max Lübbering
4 min
Abstract
We present Soofi S 30B-A3B, a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model for German and English. Its hybrid design activates only 3B of 30B parameters per token and keeps the inference cache near-constant as context grows, giving it a decisive throughput advantage over dense models for long-context, high-concurrency deployment. Pretrained on roughly 27 trillion tokens with deliberately up-weighted German, Soofi S matches dense 14 to 27B models on aggregate English and German benchmarks while achieving the best code aggregates in both languages among 17 open base models, and outperforms every European sovereign baseline in our comparison, including ones far larger in active parameters. Among fully open models, Soofi S obtains the highest English and German evaluation scores, ahead of Olmo 3 32B and Apertus 70B. Soofi S was built end-to-end on the German Industrial AI Cloud, a sovereign HPC scale AI infrastructure operated by Deutsche Telekom in Munich. Soofi S will be released under highly permissive, open-access terms: weights, selected intermediate checkpoints, full per-source data accounting, hyperparameters, and training and evaluation code. Where source licenses permit, data-construction artifacts are released under permissive licenses; commercially licensed sources are documented with aggregate statistics and exact mixture accounting.
Alex: That kind of efficiency seems particularly useful for organizations that want to run AI on their own servers rather than sending data to a cloud provider. Is that what the paper means by "sovereign" AI?
Sam: Yes. The idea of sovereign AI is that an organization hosts and controls its own system, so sensitive data never leaves its infrastructure. Making the model efficient enough to run on standard hardware—rather than requiring a data center full of specialized chips—is what makes that practical. The paper positions this as a meaningful step toward that goal, particularly for German-speaking institutions with strict data privacy requirements.
Alex: And they're open about how they built it?
Sam: They are. The researchers have released their full data accounting and training recipes—meaning they've documented exactly what data was used and how the training process was structured. That level of transparency is relatively uncommon in this field.
Alex: How do they actually manage the training process for something this complex?
Sam: They use what they call a Warmup–Stable–Decay schedule. Think of it like training for a long-distance race: you start at a gentle pace to let your body adjust, settle into a steady rhythm for the bulk of the work, and then in the final stretch, you sharpen your focus. In the model's case, that final phase means switching exclusively to the highest-quality data available, to refine the skills the model has already developed.
Alex: So it's a very deliberate, staged process—not just throwing data at the model and hoping for the best.
Sam: Exactly. The overall picture is of a carefully engineered system designed with practical deployment in mind from the start. The architecture choices, the training schedule, the transparency about data—they all point toward the same goal: a model that organizations can actually run themselves, on their own terms.
Alex: That's a useful frame. It's less about chasing benchmark scores and more about making capable AI genuinely usable outside of large tech companies. Thanks for walking through it, Sam.
Sam: My pleasure. And thanks to everyone listening—this is ResearchPod.