jina-reranker-v3.5: An Efficient Listwise Reranker with Hybrid Attention and Self-Distillation

As retrieval systems become more agentic, rerankers must balance three competing demands: high accuracy on complex, semi-structured data; domain robustness across professional fields; and low latency for production environments. The authors seek to improve upon the previous jina-reranker-v3 by optimizing the model's attention architecture and training methodology to meet these requirements without increasing the parameter count.

The researchers introduce jina-reranker-v3.5, which retains the 'last-but-not-late' (LBNL) interaction mechanism of its predecessor but introduces three key innovations:

1. Hybrid Attention Schedule: The model replaces uniform global attention with a '3L2G' schedule—three sliding-window layers followed by two global layers. This reduces computational complexity while pinning the final layer to global attention, which is essential for the LBNL readout to maintain cross-document awareness.
2. Multi-Domain Training: The model is trained on a curated mixture of legal, medical, financial, and structured data, specifically targeting failure modes identified in benchmarks like RTEB and STARK.
3. Three-Stage Self-Distillation: To bridge the gap between full-attention and sparse-attention architectures, the authors use a three-stage distillation process. A full-attention teacher model guides the sparse-attention student through attention realignment and multi-level alignment of scores, hidden states, and embeddings.

jina-reranker-v3.5 demonstrates significant improvements over its predecessor and competitive performance against much larger models. It achieves 63.20 nDCG@10 on the BEIR benchmark, matching the performance of a 4B-parameter model while using roughly 7x fewer parameters. The model shows its strongest gains in semi-structured retrieval, where it improves nDCG@10 by 9.6 points over the previous version. Additionally, the hybrid attention schedule reduces listwise inference latency by up to 1.56x.