Unsafe at any AUC: Unlearned Lessons from Sociotechnical Disasters for Responsible AI

Modern AI safety research is heavily focused on technical metrics, such as Area Under the Curve (AUC), model alignment, and benchmark performance. The authors argue that this "solutionist" approach—treating safety as a problem to be solved by better code or more robust models—is fundamentally flawed. By drawing on decades of research into industrial disasters like the Challenger space shuttle explosion and the Therac-25 radiation accidents, the authors demonstrate that safety is not a component-level property but an emergent one. When systems are complex, they can fail even when every individual component functions exactly as specified.

History shows that major catastrophes are rarely the result of unforeseeable freak accidents. Instead, they are often the result of "unlearned lessons" regarding organizational behavior. The authors identify several recurring patterns that lead to disaster:

• Poor Risk Perception: Organizations often suffer from a "drift into failure," where they systematically underestimate risk by focusing on low-probability component failures while ignoring the systemic interactions that create hazards.
• Incentive Structures: When organizational incentives prioritize speed, growth, or performance metrics over safety, risk management becomes a ritualistic, symbolic activity rather than a functional one.
• Normalization of Deviance: Over time, organizations may begin to accept unsafe practices as standard operating procedure, especially when those practices have not yet resulted in a catastrophic event.

To move toward responsible AI, the authors suggest that the field must shift its focus from purely technical fixes to the broader sociotechnical environment. This includes creating high-level roles for safety personnel, fostering a culture that rewards "curmudgeons" or critical thinkers who challenge the status quo, and ensuring that safety is treated as a first-order engineering concern. The authors emphasize that safety is not a state that can be permanently achieved, but a continuous process of management, repair, and critical self-evaluation.