Auto-DSM Under the Lens: A Black-Box Evaluation Framework for LLM-Based DSM Generation

This paper introduces a black-box evaluation framework to systematically assess the performance of Large Language Models (LLMs) in generating Design Structure Matrices (DSMs) from unstructured technical documentation. The framework addresses the lack of transparency in current Auto-DSM pipelines by benchmarking generated matrices (GEN-DSMs) against manually validated ground-truth matrices (GT-DSMs). The authors employ a multi-run evaluation strategy, utilizing metrics such as Selective Accuracy, Entropy, and a Composite Quality Score (Q) to quantify structural correctness, reproducibility, and the reliability of abstention behavior (i.e., the model's ability to say 'I don't know').

The experiments reveal that LLMs are highly sensitive to the consistency and phrasing of input data. While bi-directional dependency definitions significantly improve symmetry and reduce entropy, the models frequently exhibit 'confident misclassification'—where they provide incorrect dependencies with high certainty rather than abstaining. The study also identifies a 'first-entity bias' in multi-subsystem tasks, where the model prioritizes the first mentioned system and often fails to integrate multiple subsystems into a single, coherent DSM. Furthermore, the results suggest that when input context is weak or ambiguous, the LLM tends to default to pretrained knowledge, leading to hallucinations that are difficult to detect without external verification.

Automating system decomposition is a critical bottleneck in Model-Based Systems Engineering (MBSE). By providing a reproducible, model-agnostic benchmark, this research enables engineers to audit LLM-driven tools before integrating them into high-stakes design workflows. The findings highlight that while LLMs show potential for rapid prototyping, they currently lack the robustness required for safety-critical applications, necessitating better prompt engineering, subsystem-aware architectures, and improved traceability mechanisms.