Do Sequential Recommendation Benchmarks Really Require Higher-Order Sequence Modelling?

Recent advances in sequential recommendation have increasingly adopted complex language-model architectures, such as Transformer encoders and generative retrieval models, to capture intricate user interaction histories. However, it remains unclear whether widely used benchmark datasets actually require this advanced modelling capacity. Performance gains might instead be driven by simpler factors such as item popularity, recency, and pairwise transitions. This paper investigates whether current sequential recommendation benchmarks genuinely necessitate higher-order sequence modelling or if strong, non-neural baseline models can achieve comparable results.

To test the necessity of higher-order interactions, the authors evaluate two simple, recency-weighted pairwise models that do not learn higher-order sequence representations: Sequential Rules (SeqRules) and a newly introduced Probabilistic Collaborative Transition Model (PCTM). PCTM treats each historical item as an independent source of evidence for the next item, estimates directional next-item distributions using distance-weighted causal counts, applies Bayesian smoothing toward a uniform prior, and aggregates evidence across history items in log space. These models act as capacity probes against reproductions of popular Transformer architectures, specifically SASRec and eSASRec, using their exact evaluation protocols and data splits across five benchmark datasets.

Across three Amazon datasets and MovieLens-1M, simple recency-weighted pairwise probes either outperform or closely match Transformer-based models, exceeding eSASRec reproductions by 15% to 38% on Amazon datasets and by 4.4% on MovieLens-1M. On four out of the five evaluated benchmarks, the pairwise envelope formed by SeqRules and PCTM exceeds or is competitive with complex neural models. Only on the larger MovieLens-20M dataset does the eSASRec reproduction maintain a substantial advantage, leaving 27.3% of its NDCG unexplained by pairwise probes. This indicates that most standard academic benchmarks are poorly suited for measuring genuine gains from higher-order sequence modelling.