HABIB_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective Optimization

Large Language Models (LLMs) often struggle with "content effects," where their ability to judge the logical validity of an argument is compromised by whether the conclusion aligns with real-world plausibility. The authors participated in SemEval-2026 Task 11 to develop a system capable of disentangling formal logical structure from semantic content across 12 languages, including scenarios with irrelevant distractor premises.

The team utilized mDeBERTa-v3 backbones and addressed the scarcity of high-quality logical training data by building a rule-based engine to generate synthetic syllogisms covering all 256 Aristotelian moods and figures. To ensure the model learned formal structure rather than semantic shortcuts, they implemented a multi-objective loss function consisting of three key components:

• Adaptive Group Distributionally Robust Optimization (DRO): Focuses training on low-performing subgroups to ensure consistent performance across different logic/plausibility quadrants and languages.
• Differentiable Bias Penalty: A custom loss term that penalizes the model when its confidence scores vary significantly based on the plausibility of the premises.
• KL-Divergence Regularization: Forces the model to maintain consistent output distributions between simple syllogisms and their complex, rephrased counterparts, anchoring the model to the underlying logical form.

The system achieved perfect Ranking Scores (100.0) on English and multilingual subtasks (ST1, ST2, and ST3) with zero measurable bias. On the most complex subtask (ST4, noisy multilingual), the system achieved a 6th-place ranking. The authors identified that while the model struggled with multilingual premise structures, applying KL-Divergence consistency regularization significantly bridged the performance gap between native multilingual data and translated English versions.