ResearchPod Summary
Modern enterprise AI systems require large corpora of coherent data for training and evaluation. However, production data is often protected by strict privacy regulations and proprietary database schemas, creating a significant barrier to agent development. This paper investigates whether one can synthesize realistic, structurally valid enterprise data by leveraging the existing programmatic constraints of enterprise APIs rather than traditional statistical models.
Instead of treating data synthesis as a statistical row-generation task, the authors propose Synthesis Through Simulation (STS). In this paradigm, an LLM agent—the Generalist Populator (GP)—interacts with a simulated enterprise environment solely through its published API tools. Because these APIs already encode business logic and relational dependencies, any data generated through legitimate API calls is guaranteed to be valid. The GP agent operates through three distinct phases:
GP achieves 100% constraint satisfaction (structural validity) across all ten tested domains, including HR, airline reservations, and banking, without requiring access to database schemas. It consistently outperforms statistical synthesizers, which struggle with the 'cold-start' problem when no initial data exists, and even matches or exceeds the performance of schema-privileged agents that have full visibility into the underlying database. The authors demonstrate that training agents on data produced by the STS framework leads to superior performance on downstream evaluation benchmarks compared to models trained on seed data.
Alex: To train an agent on company software, you need company data. And that is exactly the data nobody will hand over. This work's answer is to let an AI agent build the data itself, by using the software's own interface, so every record it creates is valid by design.
Sam: Place it for me before we go further. Whose work is this, and what kind of document?
Alex: It's a method paper from a team at SAP Labs, posted as an arXiv preprint. They call the approach Synthesis Through Simulation, and they've open-sourced the framework, ten test environments, and the generated data.
Sam: I don't build enterprise agents. Why should someone like me care?
Alex: Because anyone training or evaluating tool-calling agents, meaning language models that take actions through APIs, needs realistic databases for them to act on. Real ones contain personal data, and even the database blueprint is often confidential. This offers a way to fill a database starting from the API descriptions alone. So we'll cover why existing synthetic data falls short, the one idea that fixes it, and how far the evidence actually goes.
Sam: Synthetic tabular data is a mature field. What's wrong with just training a generator on some rows?
Alex: Two things, the authors argue. First, enterprise validity is relational, not a property of one row. A time-off request must not exceed the employee's accrued balance. That's checked against the live database at the moment of writing. Generators like CTGAN treat rows independently and have no way to enforce that.
Sam: And the second?
Alex: They need seed data to learn from. In seven of the authors' ten environments there is none, which mimics a fresh deployment. So those generators simply can't run. The other old option is scripted generation, where people hand-code the business rules. That gets validity, but it says what's allowed, not what's typical. And it has to be rewritten for every new domain.
Sam: So one camp gets realistic distributions but breaks the rules, and the other keeps the rules but needs a human per domain.
Alex: That's the tension. The fix is almost embarrassingly simple. Generate the data through the same API layer that enforces the rules. If a call violates a rule, the system rejects it and the database doesn't change. So any sequence of accepted calls leaves a valid database. They state it as a short proof by induction: start valid, and every committed write preserves validity.
The STS framework resolves the conflict between privacy-preserving data access and the need for realistic, relational enterprise data. By shifting the synthesis burden from the model to the API-enforcement layer, this approach provides a scalable way to generate high-fidelity datasets while respecting the architectural boundaries of enterprise systems.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Sam: That sounds true by definition. Isn't the hard part just moved somewhere else?
Alex: Partly, yes, and the authors say so. What it buys is a clean split. Validity is handed to the environment, so the agent only has to worry about realism. That's where their agent comes in, the Generalist Populator. It sees only the tool registry, meaning the list of API functions with their descriptions and inputs. No database schema, no hand-written task list, no per-domain settings.
Sam: How does an agent with only function signatures figure out what a realistic workload looks like?
Alex: It works in three phases, repeated over many runs, with a shared memory store. First it explores, using only read calls. Write tools are deliberately blocked so it can't create sloppy records before it understands the system. It works out which entities depend on which, and guesses how often each operation should happen in real life. Then it plans one concrete task per run. Then it executes that task against the live system, adapting when calls are rejected.
Sam: Does the planning step actually matter, or is it decoration?
Alex: The airline environment shows it nicely. With planning off, the agent is told just to populate the database. It loops over every user and books flight after flight, almost all one-way, because round trips take extra steps. Round-trip bookings drop to about eight percent, against a realistic target of fifty-five. With planning on, it gets tasks like "book a round-trip economy flight for a gold-tier user," and lands close to that target. More writes, worse data.
Sam: Let's get to the evidence. What's the headline number?
Alex: The final databases passed every business-rule check in all ten environments, which the proof predicts. The statistical generators, in the three environments where they could run at all, peaked at about ninety-two percent. On realism, average per-column fidelity was about point eight eight on a zero-to-one scale. Five environments were above point nine three.
Sam: And on realism alone, did it beat those generators where they could compete?
Alex: Mixed. It led on airline, and came in slightly below them on retail and banking. So the clear win is validity and coverage, not distribution matching.
Sam: The obvious objection to schema-free is that you're throwing away information. Did they test against an agent that has the schema?
Alex: They did, against EnvScaler, which gets the full schema, live state, and business rules. On airline it failed badly. Eighty-two of a hundred runs produced no writes at all. Its tasks named specific reservation IDs at planning time, and earlier runs had already cancelled or changed them. So the first write failed. On HR the two were essentially tied. On cloud, the schema-aware baseline did slightly better on cross-table structure, where the schema directly encodes the parent-child links.
Sam: So schema knowledge helps with relationships, but can make execution brittle.
Alex: That's the authors' reading. They also hid all the meaningful names in a scrambled airline variant, with opaque codes instead of words like "business class." The agent performed just as well, and the dependency map it inferred differed by only two edges.
Sam: Where does it fall short?
Alex: Cross-table realism, in a couple of domains. On the forum, the agent creates threads about as often as replies, so most threads get one or two posts instead of a median of three. Banking accounts get too few transactions. The authors blame the language model's guesses about how often operations happen, not an inability to build relationships.
Sam: Does any of this help a downstream agent, which is the whole point?
Alex: They ran one test. They fine-tuned a seven-billion-parameter Qwen model on airline conversations collected in either their populated database or the original seed database. The pass-at-k score rose to point five, versus about point three eight for the seed version and under point three for the untuned model.
Sam: How many training examples is that?
Alex: Nineteen versus eighteen. So in my view it's an encouraging signal, not proof. Two other caveats matter more. The validity guarantee only holds if the API actually enforces every business rule, and the authors say real systems may need significant engineering to get there. These are their own mock environments built on SQLite. And it's slow: about forty seconds to two minutes per run, against seconds for the statistical generators.
Sam: So who should sit down with the full paper, and where should they start?
Alex: Anyone building data or test environments for enterprise agents. Start with the problem formulation, where validity, realism, and scalability are pulled apart. Then read the comparison with the schema-aware baseline, including the airline case study on stale IDs. The planning ablation is a quick, convincing read.
Sam: And for everyone else, the one line to carry around?
Alex: Don't teach the generator the rules. Make it go through the rules, and validity stops being your problem.
Sam: A simple idea, with its hard part honestly flagged.