Supervising the Visible State: Separating Representation from View Selection in Streaming Video

Sam: Recognizing a push-up does not tell you how many push-ups happened. This study finds that video assistants improve when training asks about history and inference retrieves missing evidence.

Alex: Those sound like different problems. Is the contribution better training, or a better way to look back?

Sam: Both, tested separately. This is a methods preprint by Yihao Hu and colleagues, called “Supervising the Visible State.” It studies streaming video: answering questions using only what the system has observed so far.

Alex: That matters for an assistant watching through camera glasses. The current image might show what someone is doing, but not what happened earlier.

Sam: And if you build those systems, this changes where you look for failures. Better memory access is not the whole story. We'll examine what the training adds, what extra evidence adds, and why a correct answer can still get lost.

Alex: Start with the training problem. Why wouldn't ordinary action recognition teach the model what it needs?

Sam: Because different histories can receive the same label. A person does a push-up once, or repeats it several times. The action label stays “push-up,” so the training target does not require preserving the count.

Alex: The label isn't wrong. It just doesn't distinguish histories that a later question might need to separate.

Sam: The authors call that “state under-specification.” Here, state means what the available observations establish: which events happened, their order, their accumulation, and changed relationships. It also includes whether the evidence supports an answer at all.

Alex: How does that differ from existing work on video memory? Those systems already try to retain earlier observations.

Sam: The authors describe most streaming approaches as context-centric. They compress visual information, maintain memories, or retrieve events. Those methods decide which evidence stays accessible; this study asks what training requires the model to infer from that evidence.

Alex: But training can't recover an event that isn't in the input. How do they keep those explanations apart?

Sam: They begin with the same backbone model and the latest four frames. First, they change training while keeping that evaluation input fixed. Then they freeze the trained model and change only which observations it receives.

Alex: So what does the new training actually ask for?

Sam: They call it Visible-State Supervision. Questions ask which event occurred, whether one event preceded another, how many repetitions finished, or how an object's state changed. Other questions require acknowledging that the supplied evidence is insufficient.

Alex: That's an appealing menu, but synthetic training questions can introduce their own errors. Where do the answers come from?

Sam: The mixture contains about seventy-seven hundred examples, grounded in source annotations or video frames. Where annotations lack a capability, a teacher model proposes questions and answers. Benchmark material and evaluation feedback are excluded from that construction process.

Alex: What checks stop a plausible-looking question from becoming a bad training target?

Sam: They check source membership, timestamps, duplication, and benchmark overlap. They also change answer-option order and retain candidates only when an auditor's answers map back to the same target. But the authors explicitly say this does not prove human unambiguity or that every question needs history.

Alex: The obvious alternative explanation is simply more temporal training. What happens against a control with the same training exposure?

Sam: That's the strongest comparison here. The generic temporal control shares visual inputs, weights, ordering, optimization, and initialization. Against it, visible-state training adds about one-point-seven percentage points on OVO-Bench, a streaming-video benchmark.

Alex: An overall gain can hide a very narrow effect. Where does the extra accuracy come from?

Sam: Mostly hallucination detection: deciding whether the visible evidence supports the claimed answer. That task improves by about eight percentage points against the matched control. Other temporal tasks show smaller, mixed differences.

Alex: Then the result is narrower than “better temporal reasoning everywhere.” Does it survive changing the training seed?

Sam: The OVO advantage stays positive across the paired training seeds. But on StreamingBench, the other benchmark, visible-state and generic training remain comparable. My reading is that the controlled evidence supports a specific benefit, especially for evidence-aware answers, rather than universal superiority.

Alex: Now take the trained model as fixed. How does the system decide when to look further back?

Sam: That stage is called Test-Time Causal View Search. “Causal” here means every frame comes from before the question's time boundary. The system starts with recent frames, adds a sparse historical view when confidence is low, and consults broader history when the answers disagree.

Alex: What does “confidence” mean in that decision? Is it asking the model whether it feels sure?

Sam: It uses the gap between the leading token probabilities at the first answer position. The thresholds are calibrated on separate, nonbenchmark videos, then fixed. Historical frames are ranked by visual change and recency, not by the question or its answer options.

Alex: Give me a case where looking back changes what the model can actually know.

Sam: The document asks why Mr. Bean is under a bed. Recent frames prompt an answer about investigating something strange. Earlier frames show a cat taking his teddy bear, and the historical views produce the correct explanation: he's trying to get it back.

Alex: That separates missing evidence from misreading the same evidence. How much does combining the interventions improve the benchmark result?

Sam: On OVO-Bench, the complete system reaches roughly seventy percent, versus roughly fifty-nine percent for the original model with recent frames. Both training and expanded access contribute. The evaluation uses the full benchmark splits, not selected example clips.

Alex: Does broader history improve the same tasks as the new training?

Sam: No, and that division is central. Training particularly helps hallucination detection and causal reasoning. Broader access particularly helps event recall, counting, and forward reasoning, while slightly hurting some recent-state judgments.

Alex: If more history helps, why not always provide it? The adaptive machinery needs to earn its complexity.

Sam: Its clearest benefit is processed-frame cost. On StreamingBench, adaptive access nearly matches a single broad-history call while processing substantially fewer frames. On OVO-Bench, it beats exhaustive largest-confidence selection using fewer frames, but those counts do not establish a universal latency advantage.

Alex: There's another risk: more evidence gives the model more opportunities to change a good answer. Did they examine that directly?

Sam: They replay every available view using the frozen model. An analysis-only oracle then chooses a correct proposal whenever any view produces one. Among cases where that oracle beats the deployed system, roughly seven in ten failures involve replacing an already retained correct answer.

Alex: That's conditional on a correct proposal existing. It doesn't mean answer replacement explains most errors overall, does it?

Sam: It does not. Most errors from the recent view remain unresolved across all the tested views. Those might reflect missing decisive frames, failed interpretation, or ambiguity; the study cannot separate those explanations.

Alex: And does their confidence-based commitment rule demonstrably solve the replacement problem?

Sam: No. Removing that rule produces no consistent accuracy loss across benchmarks. The replay diagnosis identifies a selection problem, not a validated solution. The main scope limitation is also substantial: the controlled experiments use one backbone and two benchmarks.

Alex: Given that scope, who should read the full document, and where should they start?

Sam: Researchers training multimodal models or building streaming-video assistants should read it. Start with Diagnostic Analysis, especially the strict-matched supervision comparison. Then read “Why Correct Alternatives Can Still Be Lost,” which carefully separates unavailable evidence from failures to preserve an available answer.

Alex: And what should everyone else carry away?

Sam: Training must ask for the distinction, inference must expose the evidence, and revision must not discard the right answer. Those are different problems, and this study gives researchers a practical way to test them separately.

Alex: Thanks for listening.