Emile de Bruyn, Moritz Schäffler, Jannik Schneider, Sebastian R. Schmidt, Chanwoong Hwang, Aleena Siji, Utkarsh Upadhyay, Stefan Bauer, Alexander Schug, Stefan Kesselheim, Artur Yakimovich, Alina Bazarova
8 min
Abstract
Protein motif scaffolding benchmarks assess whether generative models can build scaffolds around functionally important residues. Their conclusions depend on which motifs are tested and which criteria define success. Existing benchmarks typically use small hand-curated motif sets and static refolding criteria, but protein function often depends on conformational ensembles and transitions. Here, we address both benchmark scope and success criteria. We introduce BILBY-PRO, a systematically derived benchmark of structurally conserved functional motifs from PROSITE-linked experimental structures, comprising 220 cases from 174 PROSITE motif-pattern entries. Evaluating six recent scaffold-generation models reveals distinct patterns in target coverage and unique-solution yield: RFDiffusion and Proteina solve more targets, whereas ESM3 produces more unique successful scaffolds on the targets it solves. We further introduce BILBY-MD, a reference-based all-atom molecular dynamics evaluation framework. Applied to statically solved designs from the original MotifBench benchmark, it compares motif drift, scaffold drift, motif-contact preservation, and motif-dihedral free-energy landscapes against reference simulations. Of 55 statically solved model-problem pairs, 31 satisfy our MD-supported criterion and only 8 pass all checks. Compact, self-contained motifs are often MD-supported, whereas interface-like or context-dependent motifs fail more frequently. Static refolding success therefore provides useful evidence of local motif recovery but does not establish dynamical fidelity of the full scaffold. BILBY highlights three complementary axes of motif-scaffolding evaluation: target coverage, unique-solution yield, and MD-supported dynamical fidelity.
Sam: That gives broader coverage, but it still inherits the database’s preferences.
Alex: It does. These patterns mostly produce compact motifs within a single protein chain. Discontinuous functional sites are less well represented. And overlap with the generators’ training data cannot be excluded, so this isn’t a clean test of completely unfamiliar structures.
Sam: What changes when the authors compare generators across this broader collection?
Alex: They separate breadth from yield. Coverage asks whether a model solves a target at all under a fixed generation budget. The other score rewards structurally distinct successful solutions, with diminishing returns when a target is already well solved.
Sam: So a model that solves many targets isn’t necessarily the model that offers the most alternatives for a particular target.
Alex: That distinction appears in the results. RFDiffusion and Proteina solve the broadest range of targets. ESM3 solves fewer, but often produces a high yield of distinct successful scaffolds on those it solves. There isn’t one ranking that captures both strengths.
Sam: Now for the harder test. Did the authors simulate designs from that whole expanded collection?
Alex: No, and that boundary matters. The dynamics analysis uses statically successful designs from the original MotifBench tasks. BILBY-PRO broadens the static benchmark; BILBY-MD, the separate dynamics framework, tests a selected set from the older panel.
Sam: What makes the dynamics comparison informative? A natural protein can move too.
Alex: They compare each design with a simulation of its experimental reference structure. Molecular dynamics follows atomic motion under a physical model, here with water explicitly represented. The question isn’t whether the design moves at all, but how its behavior compares with that reference.
Sam: And which successful design do they choose when a model produces several?
Alex: They select the one most similar to an experimentally observed fold, using a structural similarity score. That favors candidates the authors expect to be relatively stable. So they aren’t deliberately testing the most unusual or obviously weak successful designs.
Sam: What do they measure beyond whether the whole protein changes shape?
Alex: They check local motif drift separately from whole-scaffold drift. They also compare contacts between motif residues: whether the same local interaction patterns persist. Finally, they compare the range of conformations sampled by the motif.
Sam: That last check seems essential. A motif might keep many contacts but still spend its time in the wrong shapes.
Alex: That’s what the free-energy landscape comparison addresses: a map of sampled shapes and their relative likelihoods. They examine both dominant structural variation and slower conformational changes. Those are complementary views of the same trajectories, not independent experiments.
Sam: How often does static success survive this broader assessment?
Alex: There are fifty-five successful model–problem pairs in the primary analysis. A pair is called dynamics-supported if it passes at least three of four checks. Thirty-one meet that rule.
Sam: That allows one kind of disagreement. How many pass everything?
Alex: Only eight pass every check. Local motif geometry is preserved more often than whole-scaffold stability. And the ensemble comparison catches cases that simple structural drift or contact measurements can miss.
Sam: Could that mean the original static label tells us very little?
Alex: The unsuccessful designs argue against that. They are much less likely to pass the dynamics screen, and none pass every check. Static success is informative; the overreach is treating it as evidence for complete dynamical fidelity.
Sam: Give me a concrete example of where the static test earns that confidence.
Alex: The ferredoxin motif, associated with an iron–sulfur cofactor, is supported across all the tested models. Its compact local geometry stays well preserved. By contrast, a nucleotide-binding motif from V-ATPase is solved by several models but shows severe scaffold drift and weak agreement in slower motif motions.
Sam: So the distinction may be whether the motif carries its own structural support, rather than borrowing it from a larger setting.
Alex: That is the authors’ interpretation. Compact, self-contained motifs fare better in this sample. Interface and epitope fragments fare worse, consistent with dependence on missing partners or broader structural context. It’s a pattern, not a universal rule.
Sam: What most limits how far we should take those results?
Alex: The simulations are finite and unequal in length. Designs run for two hundred nanoseconds; references run for one microsecond. Slower instabilities may be missed, and the support thresholds are heuristic, although sensitivity checks preserve the main interpretation.
Sam: So passing this screen still doesn’t demonstrate that a designed protein actually performs its intended function.
Alex: Correct. This is simulation-based validation, not biochemical proof. And because the analysis selects promising candidates from a limited panel, I wouldn’t treat it as a definitive dynamics leaderboard for the generators.
Sam: Who should read the full document, and where should they start?
Alex: Protein designers and benchmark developers should read the dynamics-support criteria and problem-wise performance sections first. Then check the appendix’s threshold sensitivity analysis. Those show what “supported” means and where local preservation separates from whole-protein stability.
Sam: And for researchers who only need the main lesson?
Alex: Keep static screening, but don’t ask it to establish more than it measures. Recovering the right shape is not the same as keeping it.
Sam: Thanks for listening.