BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Traditional language models often rely on unidirectional architectures (left-to-right or right-to-left), which limit the model's ability to capture deep contextual information. This paper asks whether pre-training a deep bidirectional Transformer can create more powerful, general-purpose language representations that can be fine-tuned for a wide variety of downstream tasks with minimal architectural changes.

To achieve deep bidirectionality, the authors introduce BERT (Bidirectional Encoder Representations from Transformers). Unlike previous models that concatenate independently trained unidirectional LMs, BERT uses a masked language model (MLM) objective. This involves randomly masking 15% of input tokens and training the model to predict them based on their surrounding context. Additionally, the authors include a 'next sentence prediction' (NSP) task to help the model learn relationships between sentences, which is critical for tasks like question answering and natural language inference.

BERT demonstrates that bidirectional pre-training is significantly more effective than unidirectional alternatives. The model achieved new state-of-the-art results on eleven NLP tasks, including the GLUE benchmark, MultiNLI, and SQuAD (question answering). The authors show that BERT's unified architecture allows it to be fine-tuned for both sentence-level and token-level tasks with minimal task-specific modifications, outperforming heavily engineered, task-specific architectures.

This work represents a paradigm shift in NLP by demonstrating that a single, pre-trained model can be adapted to a broad range of tasks with high performance. By eliminating the need for complex, task-specific architectures, BERT simplifies the development of NLP systems and establishes a new standard for transfer learning in language understanding.