Energy and Policy Considerations for Deep Learning in NLP | Emma Strubell et al. | ResearchPod

Transcript: Energy and Policy Considerations for Deep Learning in NLP

[[RP_SECTION:hidden-costs-of-development|Hidden Costs of Development]]

Alex: [even, measured pace] One natural language processing pipeline took 4,789 training jobs over six months to develop. That came to nearly 10,000 GPU days of compute. Emma Strubell, Ananya Ganesh, and Andrew McCallum used it as a case study, and their argument is that the cost of a model sits in the tuning and search behind it, not in the final training run.

Sam: [leaning in, analytical] So the final run is the tip of the iceberg, and the bulk is thousands of failed jobs and grid searches that never appear in a methods section. But this is one pipeline. How far can a single case study carry a claim about the field as a whole?

Alex: [steady, precise] Not very far, and that's the first thing a referee would raise. It's an existence proof, not a sampled estimate. It shows the R&D lifecycle can dwarf the cost of training one model, but it can't tell you the typical ratio across labs. What it does establish is that reporting only the final training cost leaves out most of what a project consumed. [[RP_SECTION:measuring-environmental-impact|Measuring Environmental Impact]]

Sam: [thoughtful] How did they measure it, though? Tracking power across a long, fragmented development cycle sounds hard.

Alex: [slower, for clarity] They sampled power draw from the GPU, CPU, and DRAM during active training. Then they extrapolated using total training time and a standard power usage effectiveness coefficient to account for data center overhead like cooling. Energy converts to carbon dioxide equivalent through a carbon intensity factor.

Sam: [nodding] So it's a proxy audit, not a metered one. The extrapolation assumes the sampled draw is representative of every job in those six months.

Alex: [analytical edge] Yes, and that assumption is part of the limitation. For this pipeline, the R&D process came to over 78,000 pounds of carbon dioxide. The exact figure matters less than the accounting move. It shifts the question from what one model costs to what a research program costs. [[RP_SECTION:barriers-to-research-access|Barriers to Research Access]]

Sam: [quietly] That has consequences for who can participate. A graduate student or small lab trying to reproduce or extend this work isn't only competing on ideas. They're competing on compute budget.

Alex: [measured] That's the practical concern the authors raise. Access to large, centralized compute increasingly determines who can work at the frontier, so the barrier is financial as well as environmental.

Sam: [probing] Isn't that just the price of scaling? If we want better models, we need more compute. Or can the R&D process itself become more efficient without hurting final performance? [[RP_SECTION:improving-compute-efficiency|Improving Compute Efficiency]]

Energy and Policy Considerations for Deep Learning in NLP

Emma Strubell, Ananya Ganesh, Andrew McCallum

Jan 1, 2019·6:18·arxiv·Original Paper

Computational Cost of NLPEnvironmental ImpactEquitable Access to ResourcesModel Efficiency

Listen

6 min

Briefing

Research Overview

This paper investigates the hidden costs of the recent trend toward increasingly large neural network models in Natural Language Processing (NLP). While these models achieve state-of-the-art accuracy, they require massive computational resources. The authors quantify the financial and environmental impact of training these models, arguing that the current trajectory of NLP research is becoming prohibitively expensive and environmentally unsustainable.

Approach

The researchers measured the energy consumption of several prominent NLP models, including Transformer, ELMo, BERT, and GPT-2. They sampled GPU and CPU power usage during training and extrapolated total energy requirements based on reported training times. Furthermore, they conducted a detailed case study of the development process for a specific NLP pipeline, which included thousands of hyperparameter tuning experiments, to illustrate that the true cost of research is not just the final training run, but the entire iterative development cycle.

Main Findings

The study reveals that training a single large model can produce carbon emissions equivalent to a trans-American flight, and when accounting for the extensive hyperparameter tuning required for research, these costs multiply by thousands. The authors highlight that the "rich get richer" dynamic in NLP research—where only well-funded industry labs can afford the compute required for state-of-the-art results—stifles academic creativity and limits equitable access to the field.

Why It Matters

As NLP models continue to grow in size, the community must address the sustainability and accessibility of its research practices. The authors advocate for three major shifts: (1) researchers should report training time and hyperparameter sensitivity to allow for better cost-benefit analysis; (2) funding agencies should invest in shared, centralized academic compute resources to democratize access; and (3) the field must prioritize the development of more computationally efficient algorithms and hardware.

Key Terms and Definitions

Carbon Footprint — The total amount of greenhouse gases, specifically CO2 equivalents, emitted into the atmosphere as a result of the energy consumed by training neural networks.
Hyperparameter Tuning — The iterative process of searching for the optimal configuration settings for a model, which often requires training the model hundreds or thousands of times.
Power Usage Effectiveness (PUE) — A ratio that describes how efficiently a data center uses energy, specifically how much energy is used by the computing equipment versus cooling and other overhead.
Neural Architecture Search (NAS) — An automated method for finding the best neural network architecture for a given task, which is notoriously computationally expensive.
Cloud Compute Cost — The financial expense incurred by renting hardware resources from providers like AWS or Google Cloud, which is the primary barrier for many academic researchers.

Abstract

Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data.These models have obtained notable gains in accuracy across many NLP tasks.However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption.As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware.In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP.Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.

Energy and Policy Considerations for Deep Learning in NLP: Research Paper Podcast