Phil Blandfort, Tushar Karayil, Urja Pawar, Robert Graham, Alex McKenzie, Dmitrii Krasheninnikov
7 min
Abstract
Moral benchmarks for LLMs typically use context-free prompts, implicitly assuming stable preferences. In deployment, however, prompts routinely include contextual signals such as user requests, cues on social norms, etc. that may steer decisions. We study how directed contextual influences reshape decisions in trolley-problem-style moral triage settings. We introduce a pilot evaluation harness for directed contextual influence in trolley-problem-style moral triage: for each demographic factor, we apply matched, direction-flipped contextual influences that differ only in which group they favor, enabling systematic measurement of directional response. We find that: (i) contextual influences often significantly shift decisions, even when only superficially relevant; (ii) baseline preferences are a poor predictor of directional steerability, as models can appear baseline-neutral yet exhibit systematic steerability asymmetry under influence; (iii) influences can backfire: models may explicitly claim neutrality or discount the contextual cue, yet their choices still shift, sometimes in the opposite direction; and (iv) reasoning reduces average sensitivity, but amplifies the effect of biased few-shot examples. Our findings motivate extending moral evaluations with controlled, direction-flipped context manipulations to better characterize model behavior.
Alex: What about different AIs or adding step-by-step thinking?
Sam: They tested models like DeepSeek, GPT variants, LLaMA, Grok, and Qwen—both with and without reasoning modes, where reasoning means instructing the AI to think aloud step-by-step like showing your work in math. Reasoning cuts average nudge sensitivity but boosts shifts from biased example prompts.
Alex: So step-by-step thinking makes it less swayable overall, but pickier about bad examples.
Sam: Yes—and backfiring shows up often, where nudges push choices opposite the intent, even if the AI claims neutrality. It happens more when fighting the AI's baseline lean. The harness reveals these as signs of uneven moral wiring, not caught in plain benchmarks.
Alex: Does reasoning make the model more resistant overall, or just shift what's effective?
Sam: Picture asking the AI to show its work, like breaking down a math problem one step at a time. This chain-of-thought approach makes models far more consistent on their own, picking the larger group in moral choices about 97 percent of the time without nudges, up from around 82 percent without it. But when nudges come in, it cuts the average pull from contexts by a notable amount. With reasoning on, examples—showing the AI a few worked cases first—become the strongest driver.
Alex: And do models tell informative nudges from empty ones, like fake survey mentions?
Sam: They built tests swapping real decision info for unrelated facts, keeping the sentence structure—like changing "survey says save group A" to "group A likes more jazz." Most models steer over twice as much from meaningful content, showing they focus on sense over surface. But irrelevant bits still nudge somewhat.
Alex: So reasoning helps filter junk... but doesn't erase all blind spots.
Sam: Exactly. The paper suggests these tests expose where surface tricks linger.
Alex: Do they dig into the models' step-by-step thoughts to see exactly why a nudge works or backfires?
Sam: Yes, they analyzed the chain-of-thought outputs—those written steps the AI shows when thinking aloud. They fed them to another AI classifier to check if the model plans to follow the nudge, push back, ignore it, or skip mentioning it—and to tag the main reasons given, like saving more lives or stressing fairness between groups. This breaks down the black box into patterns.
Alex: Like auditing the AI's homework notes. What did those notes reveal about unusual choices?
Sam: In plain scenarios, when models pick smaller groups anyway, they often reason about extra years of life ahead—like for younger people—or fairness to the disadvantaged. Under nudges, if they go for the smaller group, most cite the prompt itself as the driver. Backfires often feature second-level thinking about bias or discrimination, like rejecting a nudge toward Americans to dodge nationalism.
Alex: That tension makes sense of the uneven steering...
Sam: It does. Overall, these trace insights confirm influences sneak in via compliance or hidden rationale shifts, underscoring why context-aware tests matter beyond raw choices. Traces also showed sycophancy—putting user happiness over ethics.
Alex: Pulling it all together, these tests show that plain moral benchmarks miss real-world risks because everyday prompts can steer models unevenly or even backfire.
Sam: That's the core takeaway. Models often seem neutral without context, but direction-flipped nudges reveal hidden directional pulls—and backfires where influences push the opposite way. Reasoning modes dial down most sensitivities yet heighten reactions to biased examples, rotating the weak spots rather than erasing them. This matters for deployments like healthcare triage, where a casual user remark could tip life-saving calls unpredictably.
Alex: Are there limits to what this setup catches?
Sam: Yes, the authors note key ones. This is a pilot with a small, controlled set of influences—not the full mix of real prompts like multi-turn chats or cultural norms. It sticks to stylized trolley triage and doesn't probe naturalistic dilemmas. Still, the harness extends easily to broader cases.
Alex: What does that mean for fixing these issues practically?
Sam: Practitioners should add these direction-flipped checks to standard audits—test nudges both ways per group to surface asymmetries baselines ignore. This enables picking or hardening models for triage or moderation that resist manipulative prompts better. The paper stresses it's defensive: better evaluations spot deployment gaps without novel exploits.
Alex: Makes sense—a meaningful step toward more reliable moral AI, grounded in how prompts actually work. Thanks for breaking it down, Sam. That's it for this look at moral preferences under directed contextual influence on ResearchPod.