ProgramTab: Boosting Table Reasoning of LLMs via Programmatic Paradigm

Overview

Table-based reasoning tasks, such as question answering and fact verification, are challenging for Large Language Models (LLMs) due to input length limitations and the messy, inconsistent nature of web-based tabular data. Existing methods often struggle to handle large tables or perform accurate mathematical operations on poorly structured data. ProgramTab addresses these issues by introducing a programmatic paradigm that combines Python-based data preprocessing with SQL-based information extraction.

The ProgramTab Approach

The framework operates through a five-step pipeline designed to minimize noise and maximize reasoning accuracy:

1. Row Retrieval: Instead of feeding an entire table into the LLM, the system uses an embedding model to identify and extract the top K most relevant rows based on the user's question.
2. Column Extraction: The LLM identifies which specific columns are necessary to answer the query, reducing the context window size.
3. Data Redefinition: The LLM generates Python code to standardize inconsistent data formats (e.g., converting mixed date strings into a uniform format) and explicitly defines data types (int, float, string, datetime) to ensure mathematical operations are valid.
4. SQL Generation: Using the cleaned data, the LLM generates SQL queries via a Chain-of-Thought (CoT) approach, decomposing complex questions into simpler, executable sub-steps.
5. Answer Generation: The final result is derived by having the LLM synthesize the SQL execution output with the original question.

By shifting the burden of data cleaning from the LLM's internal reasoning to explicit, executable Python code, ProgramTab significantly improves performance on benchmarks like WikiTQ and TabFact. This approach effectively bypasses the common pitfalls of LLMs when dealing with "dirty" data, such as calculation errors or hallucinations caused by inconsistent table formatting. It demonstrates that combining symbolic execution (Python/SQL) with neural reasoning is a robust strategy for handling structured data at scale.