From Financial Sentiment Classification to Return Predictability: A QLoRA Benchmark of Large Language Models

This study investigates the relationship between two distinct objectives in financial natural language processing: the ability of Large Language Models (LLMs) to accurately classify financial sentiment and the ability of those same models to generate profitable, tradable signals in the stock market. While sentiment classification is a common benchmark, it remains unclear whether higher classification accuracy necessarily leads to superior economic performance.

The author conducts two experiments. First, a unified benchmark is created from five financial datasets to compare various models, including traditional TF-IDF Naive Bayes, domain-specific encoders (FinBERT, Financial-RoBERTa), and several LLMs (Qwen2.5, LLaMA3, Mistral) adapted via QLoRA. Second, the study evaluates the economic validity of these models using a 2019 Benzinga news sample. The models' sentiment outputs are converted into continuous scores and tested for their ability to predict next-session stock returns across multiple time horizons, using rigorous statistical methods like Newey-West inference and false-discovery-rate correction.

The study finds that QLoRA is highly effective for adapting LLMs to financial sentiment tasks, with Mistral-7B achieving the highest classification accuracy and QLoRA significantly boosting the performance of the Qwen2.5 backbone. However, when these models are applied to real-world market data, the results are disappointing. None of the models demonstrate statistically significant return predictability after adjusting for multiple testing. The findings highlight a clear decoupling between linguistic performance—how well a model understands text—and economic utility—how well a model predicts price movements.

This research serves as a cautionary tale for quantitative researchers and practitioners. It demonstrates that optimizing for standard NLP metrics like F1-score or accuracy does not guarantee success in financial trading. The results suggest that the "signal" extracted by LLMs from news headlines may be too weak or too quickly incorporated by the market to be exploited, emphasizing that semantic accuracy is not a proxy for economic value.