Redefining data engineering in the age of AI | Denis McCauley | ResearchPod

Transcript: Redefining data engineering in the age of AI

Sam: AI can make data engineers more productive without making their workloads lighter. A survey-based report argues that their work is also becoming more strategic: less confined to moving data, more involved in deciding how businesses use AI.

Alex: That creates a planning problem. If automation helps a team deliver more, leaders might assume it also frees up capacity. Whose report is this?

Sam: It’s an MIT Technology Review Insights report, sponsored by Snowflake, called “Redefining data engineering in the age of AI.” It combines an executive survey with interviews. For people building or studying enterprise AI, it challenges the idea that data engineering is merely a support function.

Alex: So let’s separate the changing job from the automation promise, then ask how much the evidence supports either.

Sam: Start with the old job description. The report describes data engineers as maintaining pipelines: sequences that move and prepare data. Their traditional work involved extracting data, transforming it, and loading it into another system. The emerging role includes designing reusable platforms, overseeing data, and helping assess business uses of AI.

Alex: That sounds like a move toward architecture, not just better tools for the same job. How did the report look for that change?

Sam: The researchers surveyed 400 senior data and technology executives in June 2025. They asked about engineers’ activities, workloads, productivity, and influence over decisions. Interviews with executives and experts provide explanations for those patterns.

Alex: Before we generalize, what kinds of organizations are those executives describing? A corporate data team may face different pressures from a university research group.

Sam: Every organization had annual revenue of at least 500 million dollars. They spanned several industries and countries, but this remains a large-enterprise sample. I would use it to understand those organizations, not assume it describes academic labs or small companies.

Alex: What is the clearest sign that daily work has changed, rather than just the language executives use?

Sam: Respondents estimated that engineers spent 19 percent of their workday on AI projects two years before the survey. At the survey date, that estimate was 37 percent. Examples include building and monitoring model pipelines, or cleansing data for AI projects.

Alex: Those are estimates from executives looking backward, rather than time logs collected throughout that period. Does the report also project where the shift goes next?

Sam: It does: respondents expected AI projects to occupy 61 percent of engineers’ time two years later. That is a forecast, not an observed result. The report’s argument is that AI is becoming a central responsibility rather than an additional task at the margins.

Redefining data engineering in the age of AI

Denis McCauley

Oct 19, 2026·8:08·user_upload

Data EngineeringAgentic AIData Engineering EvolutionAI-powered EngineeringAI MaturityTech Stack Fragmentation

Listen

8 min

Summary

The Evolving Role of Data Engineering

The integration of AI into enterprise operations has fundamentally altered the role of the data engineer. Once relegated to "backend plumbing" and routine ETL (extract, transform, load) tasks, data engineers are now being recognized as essential strategic partners. This shift is driven by the realization that AI's effectiveness is entirely dependent on the quality, accessibility, and governance of the underlying data. Consequently, organizations—particularly large enterprises in financial services and manufacturing—are increasingly relying on data engineers to guide AI strategy and business decision-making.

Shifting Workloads and AI Integration

Data engineering teams are experiencing a significant pivot in their daily activities. As of 2025, engineers report spending 37% of their time on AI-related projects, up from 19% in 2023, with expectations that this will climb to 61% within two years. This transition has led to heavier workloads as teams manage not only traditional structured data but also complex, real-time, and unstructured data streams required for modern AI models. While AI has simultaneously improved productivity and output quality, it has also introduced new challenges related to technical debt, integration complexity, and the need for robust data governance to support evolving AI tools.

The Rise of Agentic AI

Beyond simple automation, organizations are increasingly turning to agentic AI—autonomous systems capable of goal-driven reasoning and self-correction. Although current adoption is limited, a majority of surveyed organizations plan to deploy these systems soon. Data engineers see promise in these tools for automating pipeline debugging, data integration, and orchestration. However, this shift requires a new level of expertise, as teams must navigate the risks of fragmented tech stacks and the heightened security implications of autonomous agents acting on organizational data.

Building Strategic Influence

To succeed in this new environment, the report suggests that data engineers must bridge the gap between technical architecture and business outcomes. This involves adopting software engineering practices, such as version control and infrastructure-as-code, while simultaneously developing the communication skills necessary to influence stakeholders and leadership. The convergence of the data engineer and data architect roles is becoming increasingly apparent, as engineers now regularly influence decisions on vendor selection, data strategy, and AI feasibility.

Key Terms and Definitions

Agentic AI — Autonomous systems that go beyond standard generative AI by using advanced reasoning and context to make and execute decisions on behalf of users.
Data Engineering — The backend function of designing and managing data pipelines, which has expanded to include architecting reusable platforms and managing unstructured data for AI.
ETL (Extract, Transform, Load) — The traditional process of pulling data from various sources, cleaning it, and moving it into a data warehouse, which is now increasingly being automated.
Multi-modal AI — AI models capable of processing and analyzing diverse data types, such as images, video, audio, and text, which places greater demands on data ingestion and storage capabilities.
Tool Sprawl — The inefficiency caused by maintaining a growing, fragmented array of disconnected data and AI tools, often requiring consolidation to ensure security and governance.

Redefining data engineering in the age of AI: Research Paper Podcast