An assistant monitoring a person performing a procedure is most useful when it can detect mistakes while there is still time to recover. This requires judging actions against task instructions as the video unfolds, identifying the attempted step, locating it in time, and assessing its execution. Existing online detectors do not use the instruction text, while instruction-conditioned methods operate offline. In this paper, we study online, instruction-conditioned mistake detection in egocentric videos. Using only observed video, task instructions, and prior predictions, our model identifies and localizes completed action attempts and classifies them as correct, modified, slip, corrections, or additions; omitted steps are determined after the video ends. We compare a modular pipeline, with separate step identification, temporal localization, and execution verification, to a joint model that predicts all outputs in a single step. To support supervised learning from mistakes, we repartition EgoPER to include erroneous videos in all splits. The approaches achieve similar step-segmentation F1@0.5 (93.6% and 92.1%), while the joint model achieves higher binary mistake-detection F1@0.5 (84.1% vs. 80.7%), with its largest gain on slips (56.4% vs. 37.5%). Component-level analysis suggests that error propagation between stages contributes to the modular approach's lower detection score. On test videos containing mistakes, both designs outperform released AEM and ESTANet checkpoints under our evaluation protocol.
Alex: Spotting an action is not the same as knowing whether it follows the instructions. In this study, a video assistant detected mistakes better when it judged the attempted instruction, action timing, and execution together rather than separately.
Sam: That distinction matters if an assistant is supposed to help someone recover, rather than just describe what happened. Where does this result come from?
Alex: It’s a methods preprint by Amit Feldman, Ilan Shimshoni, and Rotem Dror at the University of Haifa. They study mistakes in videos recorded from the person’s own viewpoint.
Sam: For researchers building assistants, the useful question is whether recognizing a procedure also means knowing when it has gone wrong.
Alex: And this study suggests those are different capabilities. The intended applications include monitoring cooking, assembly, or laboratory procedures, although the experiments cover cooking only. We’ll unpack what the system judges, why combining its decisions helps, and how far the evidence supports an assistant people could actually use.
Sam: Start with the gap in previous work. Were existing systems missing the instructions, or missing the chance to intervene?
Alex: Different systems missed different pieces. Instruction-conditioned methods, including EgoOops and ZeProM, compared actions with procedural text, but used the complete recording. Online methods make decisions using only video observed so far, but the detectors discussed here did not judge actions against explicit instruction text.
Sam: So a system could recognize the expected action and still overlook that it was performed incorrectly.
Alex: The authors give a useful example. The instruction says to pour water into the bowl containing oats. Pouring water into an empty bowl still looks like pouring water, but it violates that instruction.
Sam: How does the system represent that distinction? If the person used the wrong bowl, does it say the instructed step never happened?
Alex: It separates an instruction from an attempt to perform it. The wrong-bowl action still counts as an attempt at the pouring instruction. A completed attempt means the action ended, not that its goal was achieved.
Sam: That also leaves room to distinguish changing a recipe from failing to carry out a step. How detailed are the labels?
Alex: A modification changes how an instructed step is performed, such as stirring with a knife instead of a spoon. A slip fails to achieve the step’s goal and requires corrective action. Pouring into the wrong bowl is their example of a slip.
Sam: And transferring that water into the oats bowl would be a recovery, rather than another mistake?
Alex: That receives a correction label. The system also labels correct attempts and additions, meaning unnecessary actions outside the procedure. Adding an unrequested ingredient is different from substituting an ingredient within an instructed step.
Sam: What makes the monitoring online in practice? I’d want to know whether it really avoids looking ahead.
Alex: It begins with a five-second video window. If it cannot identify a completed attempt, it extends that window and tries again. Once it accepts an action, the next window starts at the predicted end of that action.
Sam: So it waits for an action to finish. That sounds different from detecting a dangerous movement at the instant it begins.
Alex: It is different. The paper distinguishes online processing from strict real-time response. This system uses no future video, but that does not establish instantaneous alerts or sufficient processing speed for live use.
Sam: Now to the central comparison. What changes between the separate-stage system and the combined one?
Alex: The modular system first selects the attempted instruction. Another component locates its start and end. A verifier then sees that cropped interval and judges execution. The joint system predicts the instruction, boundaries, and execution label in a single response.
Sam: Does the joint version get a different foundation model, or is this mainly a comparison of how the decisions are organized?
Alex: Both use the same underlying vision-language model, which processes images and text, with task-specific training. The modular version adapts separate components; the joint version adapts one model for the combined output. Both retain earlier predictions, which can help connect a correction to an earlier slip.
Sam: The obvious risk in the pipeline is that choosing the wrong instruction could contaminate everything downstream. What evidence says that matters here?
Alex: The authors score components across their calls, then separately when upstream predictions were correct. Localization improves substantially with the correct instruction. Execution classification also improves with correct upstream outputs, especially for slips. That supports error propagation as an explanation, though it does not prove it accounts for the entire joint-model advantage.
Sam: Before the headline score, what did they train and test on? The range of procedures seems central to interpreting this.
Alex: They use EgoPER, a dataset covering five cooking tasks, including oatmeal and tea. Its official training and validation sets contain only normal videos. The authors repartition it so their supervised models can also learn from annotated mistakes.
Sam: That makes sense for learning mistake categories, but it changes the benchmark. How much held-out evidence do they have?
Alex: Their test set contains fifty-eight videos, with both normal and erroneous procedures. On that full set, the joint model scores about eighty-four percent on their main mistake-detection measure, compared with about eighty-one percent for the modular pipeline.
Sam: What does that measure reward? A system could look accurate just because most actions are normal.
Alex: They use an F-one score that balances precision and recall, then average the scores for normal and erroneous actions equally. A predicted segment must also overlap the annotated action sufficiently in time. So getting the label right somewhere vaguely nearby is not enough.
Sam: Does combining the decisions mainly help locate actions, or does it help recognize what went wrong?
Alex: Their step-segmentation scores are close. The clearest category-level gain is for slips: the joint model reaches about fifty-six percent, compared with about thirty-eight percent for the pipeline. That supports the authors’ interpretation that the benefit goes beyond better action boundaries.
Sam: But that is not a win across every mistake category. Where does the combined model struggle?
Alex: It performs worse on corrections, and the overall improvement across fine-grained mistake types is modest. Corrections are particularly uncertain because the test set contains only a handful. Also, the binary metric groups corrections with errors for scoring, even though the authors explicitly describe them as recovery actions.
Sam: And against existing detectors, how fair is the comparison after changing the training split?
Alex: Both proposed designs outperform released AEM and ESTANet checkpoints on the twenty-five erroneous test videos. But those baselines were trained on correct videos, while the new systems also learned from annotated mistakes. The authors acknowledge that this is not a fully controlled comparison.
Sam: There’s another boundary case: a skipped instruction has no visible action to classify. Can this approach actually detect omissions while someone is working?
Alex: It finalizes omissions only after the recording ends, by checking which instructions were never detected. An incorrectly executed attempt still counts as observed. During the procedure, an apparent omission can only be provisional, because the person might perform the step later.
Sam: So the supported result is improved detection in recorded cooking procedures, not proof that an assistant prevents failures. What should determine whether someone reads the full document?
Alex: Researchers building procedural video models or monitoring assistants should read it. Start with the modular-versus-joint methodology, then the module-level performance analysis. Those sections connect the design choice to the proposed explanation. Read the dataset split and limitations before interpreting the baseline comparisons.
Sam: And what should someone outside that area carry away, given that live assistance remains untested?
Alex: The experiments do not establish generalization beyond these cooking tasks, useful live alerts, or successful recovery. But they illustrate a useful distinction: recognizing what someone did is not the same as checking whether they did what was asked.
Sam: Keep that distinction in mind when judging the next procedural assistant.