Transformer-Based Language Models Across Domain Verticals: Architectures, Applications and Critical Assessment

Overview

This paper provides a structured review of transformer-based language models, designed to help practitioners navigate the rapid pace of AI development. The authors move beyond the "noise" of monthly model releases by organizing the field into a functional taxonomy based on architectural design and intended use cases. They also evaluate the practical shift from simple pretraining to complex pipelines involving instruction tuning, preference optimization, and retrieval augmentation.

Architectural Taxonomy

The authors categorize models by their structural properties to guide deployment decisions:

• Encoder-only (e.g., BERT): Best for classification, extraction, and retrieval tasks where input is bounded.
• Decoder-only (e.g., GPT, Llama): The standard for open-ended generation and dialogue, now optimized via instruction tuning.
• Encoder-decoder (e.g., T5): Ideal for tasks requiring structured rewrites, such as translation or summarization.
• Mixture-of-Experts (MoE) (e.g., Mixtral, DeepSeek): A strategy to decouple model capacity from inference cost by activating only a subset of parameters per token.

The Post-2023 Landscape

The paper highlights that modern deployment is no longer just about the base model, but about the surrounding "recipe." Key developments include:

• Alignment: Techniques like RLHF and DPO have become essential for steering models toward user intent and reducing harmful outputs.
• Efficiency: Methods like LoRA and QLoRA allow for parameter-efficient fine-tuning, enabling high-performance customization on consumer hardware.
• Reliability: Retrieval-Augmented Generation (RAG) is identified as the primary solution for mitigating hallucinations and incorporating private, domain-specific data.