Counterfactual Analysis via Large Language Models

This paper investigates whether Large Language Models (LLMs), specifically GPT-3.5, can be utilized to perform counterfactual analysis—predicting outcomes under hypothetical scenarios—within the context of online lending. Specifically, the author explores if an LLM can accurately estimate the Return on Investment (ROI) for loans under alternative interest rate schemes, a task that is critical for platform decision-making but inherently difficult because counterfactual outcomes are never observed.

The researcher uses a dataset of nearly 900,000 loans from LendingClub. To evaluate the LLM, the study first tests its ability to predict actual, observed loan outcomes. The author employs various prompt engineering techniques, including role-playing (e.g., acting as a credit risk expert), chain-of-thought prompting, and tree-of-thought prompting (simulating a panel of experts). These prompts incorporate loan characteristics, borrower profiles, and predictions from traditional machine learning (ML) models like XGBoost. The quality of the LLM's output is assessed through predictive R-squared values and a forecast encompassing test to determine if the LLM provides information beyond what is captured by standard ML algorithms.

The study finds that basic prompts yield poor results, but sophisticated prompt engineering significantly enhances performance. By using a "tree-of-thought" approach with multiple simulated experts, the LLM's R-squared for ROI prediction increases from 1.97% to 2.84%, approaching the 3.48% achieved by gradient-boosted regression. Crucially, the forecast encompassing test indicates that the LLM provides incremental predictive value that the ML model misses. Furthermore, the LLM demonstrates the ability to provide logically coherent, causal reasoning when generating counterfactual scenarios, suggesting that LLMs are a viable tool for complex decision-making tasks.