Harnessing LLMs for Reliable Academic Supervision: A Comparative Study

This paper investigates the gap between the fluent, single-shot performance of large language models (LLMs) and the requirements for reliable, production-grade domain systems. The author introduces the concept of "harness engineering," which involves wrapping an LLM in deterministic scaffolding—such as symbolic filters, schema-typed I/O, retrieval systems, and human-in-the-loop (HITL) gates—to ensure reliability, traceability, and institutional consistency.

To test this, the author developed two systems for academic supervision: a baseline (ASA) using a powerful GPT-5 model without scaffolding, and an engineered system (ASuS) using a smaller GPT-4o-mini model wrapped in a LangGraph harness. The systems were evaluated by ten raters across six dimensions: grounding, explainability, consistency, process integrity, cognitive load, and constraint adherence.

Despite using a significantly smaller base model, the ASuS system outperformed the ASA baseline on every evaluated dimension. The pooled mean score for ASuS was 4.08 compared to 1.23 for ASA. The study confirms that the structural benefits of the harness are largely model-invariant, suggesting that for high-stakes institutional workflows, the design of the surrounding system architecture is more impactful than the size of the underlying language model.

The research identifies seven recurring design patterns that contribute to system reliability:

• Symbolic-before-semantic filtering: Using metadata to narrow search spaces before applying semantic similarity.
• Schema-typed module outputs: Ensuring modules communicate via structured data rather than prose.
• LLM-as-judge with bounded retry: Automating quality control with iterative refinement.
• HITL as a hard gate: Requiring human intervention at critical process transitions.
• Deterministic core with LLM narration: Using fixed logic for decisions and the LLM only for explanation.
• Audit-by-construction: Automatically logging system state at every node.
• Intent routing: Simplifying complex user inputs into specific, manageable pipelines.