PRAGMA: Revolut Foundation Model | Maxim Ostroukhov et al. | ResearchPod

Transcript: PRAGMA: Revolut Foundation Model

[[RP_SECTION:pragma-model-overview|PRAGMA model overview]]

Alex: A team at Revolut has trained a Transformer called PRAGMA directly on banking event streams, with no hand-built features. It scales to a billion parameters and outperforms task-specific baselines across six banking domains.

Sam: That's the claim that matters for anyone who has built gradient-boosted trees on aggregated transaction features. But "outperforms" can hide a lot. Does this actually retire the bespoke pipelines, or just beat them on the paper's chosen tasks?

Alex: The second reading is the safer one. The evidence is the comparison against task-specific baselines in those six domains. The paper also frames this as a general representation of financial behavior, and that goes further than a set of benchmark wins supports. I'd want the margins and the strength of the baselines before agreeing it replaces anything. [[RP_SECTION:tokenization-and-architecture|Tokenization and architecture]]

Sam: Then let's look at the mechanism. A transaction amount and an in-app navigation event are very different objects. How do you put both into one sequence model?

Alex: Through key-value-time tokenization. The authors treat banking data as a structured, temporal graph and split every event into three parts. Think of a library catalog. The key is the metadata, the value is the content, and the timestamp is the coordinate. The model can organize the library without reading every book in full.

Sam: So the semantic identity of an event type is separated from its value and from its timing.

Alex: Yes, and that separation is what lets one vocabulary cover heterogeneous events. The type of event is learned independently of the particular amount or attribute attached to it. Architecturally, there are two branches, one encoding the customer's profile state and one encoding the event stream. A history encoder then fuses them so each prediction sees both. [[RP_SECTION:temporal-data-representation|Temporal data representation]]

Sam: Time is the awkward part of this kind of data. Banking behavior has strong daily and weekly rhythms, and irregular gaps on top of that. How does the model represent it?

Alex: Two mechanisms, covering the two scales. For local sequencing, it computes log-seconds since the previous event. The log compresses gaps that range from seconds to weeks into a usable scale. For calendar structure, it embeds calendar features with periodic functions, so cycles are encoded explicitly rather than left for the model to infer from raw timestamps.

Sam: That's a design choice, not a regularizer. It makes the cycles easy to represent, but it doesn't show the model avoids overfitting to them.

PRAGMA: Revolut Foundation Model

Maxim Ostroukhov, Ruslan Mikhailov, Vladimir Iashin, Artem Sokolov, Andrei Akshonov, Vitaly Protasov, Andrey Goncharov, Dmitrii Beloborodov, Vince Mullin, Roman Y. Enzmann, Georgios Kolovos, Jason Renders, Pavel Nesterov, Anton Repushko

Apr 9, 2026·4:14·arxiv·Original Paper

Foundation ModelsBanking Event SequencesMasked ModellingRepresentation LearningPRAGMAKey-Value-Time Tokenisation

Listen

4 min

Briefing

Research Question

Modern financial institutions generate massive, heterogeneous event streams—ranging from card transactions to in-app navigation—that are difficult to model with standard language-model architectures. The authors investigate whether a single, large-scale foundation model can learn universal representations of these banking user histories that transfer effectively across multiple, disparate downstream financial tasks.

Approach

The researchers introduce PRAGMA, an encoder-only Transformer architecture designed specifically for multi-source banking data. Unlike standard LLMs that treat all data as text, PRAGMA uses a specialized tokenization scheme that preserves the structure of tabular financial records by decomposing them into semantic keys, values (numerical, categorical, or textual), and temporal coordinates. The model employs a two-branch design: a Profile State Encoder for static user attributes and an Event Encoder for sequential interactions, which are then fused by a History Encoder. The model is pre-trained using a masked modeling objective on a large-scale corpus of 24 billion events. For downstream tasks, the authors utilize either frozen embedding probes or parameter-efficient fine-tuning via Low-Rank Adaptation (LoRA).

Main Findings

PRAGMA demonstrates superior performance across six diverse downstream tasks, including credit scoring, fraud detection, and product recommendation, consistently outperforming strong, task-specific baselines. The authors show that scaling the model from 10 million to 1 billion parameters yields significant performance gains, particularly in high-impact tasks like credit scoring. Furthermore, LoRA fine-tuning of the pre-trained backbone consistently matches or exceeds the performance of training models from scratch, confirming the effectiveness of the learned representations. The architecture also proves robust to event staleness, making it suitable for real-time applications.

Why It Matters

This work establishes that banking event sequences, despite their irregular timing and heterogeneous structure, can benefit from the foundation model paradigm. By consolidating multiple independent, high-maintenance pipelines into a single shared backbone, financial institutions can reduce engineering overhead while simultaneously improving predictive accuracy. The findings suggest that PRAGMA provides a general-purpose representation layer that can be adapted to a wide variety of financial use cases with minimal task-specific training.

Key Terms and Definitions

Banking Event Sequences — Chronological records of user interactions with a financial platform, including transactions, app navigation, and communications.
Profile State — Static or slowly changing contextual attributes of a user, such as account tenure, membership plan, or service region, used to provide background for sequential events.
Masked Modeling — A self-supervised training objective where a subset of input tokens is hidden, and the model is tasked with reconstructing the original values based on the surrounding context.
Low-Rank Adaptation (LoRA) — A parameter-efficient fine-tuning technique that updates only a small fraction of a model's weights, allowing for task-specific specialization without full retraining.
Embedding Probe — A lightweight linear model trained on top of frozen, pre-extracted representations from the foundation model to evaluate the quality of those representations for a specific task.

Abstract

Modern financial systems generate vast quantities of transactional and event-level data that encode rich economic signals. This paper presents PRAGMA, a family of foundation models for banking event sequences. Our approach pre-trains a Transformer-based architecture with masked modelling on a large-scale, heterogeneous banking event corpus using a self-supervised objective tailored to the discrete, variable-length nature of financial records. The resulting model supports a wide range of downstream tasks such as credit scoring, fraud detection, and lifetime value prediction: strong performance can be achieved by training a simple linear model on top of the extracted embeddings and can be further improved with lightweight fine-tuning. Through extensive evaluation on downstream tasks, we demonstrate that PRAGMA achieves superior performance across multiple domains directly from raw event sequences, providing a general-purpose representation layer for financial applications.

PRAGMA: Revolut Foundation Model: Research Paper Podcast