Choosing Where and How to Moderate: End-to-End Trade-offs in Filter Placement and Response Rewriting

Content moderation is typically evaluated by the accuracy of a standalone classifier. However, in a real-world conversational pipeline, safety teams must decide where to intervene (input vs. response) and what action to take (hard block vs. rewrite). This paper investigates how these architectural choices impact end-to-end customer outcomes, specifically balancing system usefulness against harmful exposure.

The authors define two primary metrics: Usefulness (the fraction of turns resulting in a safe, relevant response) and Harmful Exposure (the fraction of turns resulting in a harmful response). They compare four moderation configurations: Input-only, Response-only, Input + Response, and Response + Rewrite. The study evaluates these configurations using both a human-labeled internal product benchmark and the public ToxicChat dataset, utilizing LLM-based graders for the latter.

The researchers found that Response-only filtering consistently achieves higher usefulness than Input-only filtering, as the generator's internal alignment often handles many inputs that a prompt-side filter would otherwise block. Furthermore, implementing a selective rewrite stage—where flagged responses are regenerated and re-screened—significantly recovers traffic that would have been lost to hard blocking. This approach maintains a comparable level of harmful exposure to hard-blocking configurations while improving the overall user experience. The study also demonstrates that probe-based routing is significantly faster than LLM-based routing for determining when to trigger a rewrite.

These results suggest that there is no universal "best" placement for moderation filters. Instead, safety teams should evaluate moderation configurations based on deployment-specific constraints, such as latency requirements and the desired balance between safety and utility. By shifting the focus from component-level classifier accuracy to end-to-end customer outcomes, developers can build more robust and helpful conversational systems.