ResearchPod Summary
The integration of AI into enterprise operations has fundamentally altered the role of the data engineer. Once relegated to "backend plumbing" and routine ETL (extract, transform, load) tasks, data engineers are now being recognized as essential strategic partners. This shift is driven by the realization that AI's effectiveness is entirely dependent on the quality, accessibility, and governance of the underlying data. Consequently, organizations—particularly large enterprises in financial services and manufacturing—are increasingly relying on data engineers to guide AI strategy and business decision-making.
Data engineering teams are experiencing a significant pivot in their daily activities. As of 2025, engineers report spending 37% of their time on AI-related projects, up from 19% in 2023, with expectations that this will climb to 61% within two years. This transition has led to heavier workloads as teams manage not only traditional structured data but also complex, real-time, and unstructured data streams required for modern AI models. While AI has simultaneously improved productivity and output quality, it has also introduced new challenges related to technical debt, integration complexity, and the need for robust data governance to support evolving AI tools.
Beyond simple automation, organizations are increasingly turning to agentic AI—autonomous systems capable of goal-driven reasoning and self-correction. Although current adoption is limited, a majority of surveyed organizations plan to deploy these systems soon. Data engineers see promise in these tools for automating pipeline debugging, data integration, and orchestration. However, this shift requires a new level of expertise, as teams must navigate the risks of fragmented tech stacks and the heightened security implications of autonomous agents acting on organizational data.
Sam: AI can make data engineers more productive without making their workloads lighter. A survey-based report argues that their work is also becoming more strategic: less confined to moving data, more involved in deciding how businesses use AI.
Alex: That creates a planning problem. If automation helps a team deliver more, leaders might assume it also frees up capacity. Whose report is this?
Sam: It’s an MIT Technology Review Insights report, sponsored by Snowflake, called “Redefining data engineering in the age of AI.” It combines an executive survey with interviews. For people building or studying enterprise AI, it challenges the idea that data engineering is merely a support function.
Alex: So let’s separate the changing job from the automation promise, then ask how much the evidence supports either.
Sam: Start with the old job description. The report describes data engineers as maintaining pipelines: sequences that move and prepare data. Their traditional work involved extracting data, transforming it, and loading it into another system. The emerging role includes designing reusable platforms, overseeing data, and helping assess business uses of AI.
Alex: That sounds like a move toward architecture, not just better tools for the same job. How did the report look for that change?
Sam: The researchers surveyed 400 senior data and technology executives in June 2025. They asked about engineers’ activities, workloads, productivity, and influence over decisions. Interviews with executives and experts provide explanations for those patterns.
Alex: Before we generalize, what kinds of organizations are those executives describing? A corporate data team may face different pressures from a university research group.
Sam: Every organization had annual revenue of at least 500 million dollars. They spanned several industries and countries, but this remains a large-enterprise sample. I would use it to understand those organizations, not assume it describes academic labs or small companies.
Alex: What is the clearest sign that daily work has changed, rather than just the language executives use?
Sam: Respondents estimated that engineers spent 19 percent of their workday on AI projects two years before the survey. At the survey date, that estimate was 37 percent. Examples include building and monitoring model pipelines, or cleansing data for AI projects.
To succeed in this new environment, the report suggests that data engineers must bridge the gap between technical architecture and business outcomes. This involves adopting software engineering practices, such as version control and infrastructure-as-code, while simultaneously developing the communication skills necessary to influence stakeholders and leadership. The convergence of the data engineer and data architect roles is becoming increasingly apparent, as engineers now regularly influence decisions on vendor selection, data strategy, and AI feasibility.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Alex: Those are estimates from executives looking backward, rather than time logs collected throughout that period. Does the report also project where the shift goes next?
Sam: It does: respondents expected AI projects to occupy 61 percent of engineers’ time two years later. That is a forecast, not an observed result. The report’s argument is that AI is becoming a central responsibility rather than an additional task at the margins.
Alex: What makes that responsibility more demanding? Is it simply that organizations want more AI projects?
Sam: More projects are part of it, but the interviews emphasize complexity too. Models consume unstructured data, meaning material such as images, audio, and free text. George Westerman at MIT describes the challenge as deciding how to tag, organize, and store it, alongside handling faster data flows.
Alex: So even if a tool accelerates coding, it does not remove the decisions about organizing the data. Do executives actually report heavier workloads?
Sam: Yes. Seventy-seven percent said their data engineers’ workloads were becoming increasingly heavy. That sits alongside reported productivity improvements, not instead of them. The report presents expanding demand and greater efficiency as things happening together.
Alex: Then we need to be careful with “productivity.” Did the researchers measure how much work engineers completed per hour?
Sam: Not directly. Seventy-four percent of respondents reported that AI had increased their teams’ output over the previous two years. Output could mean projects delivered or new code generated. Executives also reported quality improvements, with data freshness given as an example.
Alex: That supports a claim about what leaders perceive. It cannot establish how much AI itself caused those improvements, or whether engineers experienced the same benefits.
Sam: That is the central evidence limit. The survey is not a controlled comparison, and it does not directly track engineers’ work. My reading is that it offers useful evidence about perceived organizational change, but weaker evidence for a causal productivity claim.
Alex: The interviews also describe changes in engineering practice. What is one concrete example, rather than another forecast?
Sam: Snowflake’s Chris Child points to infrastructure as code: defining computing infrastructure through code. He also describes keeping engineering code in version control and using automated deployment processes. His argument is that adopting software engineering practices helps data teams manage complex projects and change systems with less fear of breaking them.
Alex: That makes the proposed shift more concrete. Engineers are managing systems that can be changed and reused, rather than treating every data flow as an isolated task.
Sam: And the report finds influence beyond those systems. Respondents describe engineers contributing to tool investment, vendor selection, and judgments about whether an AI use case is feasible. Influence over overall AI strategy is less widespread, so this is not a claim that engineers now direct everything.
Alex: The next promise is autonomous agents. What does the report mean by agentic AI, and what would it change for these teams?
Sam: Agentic AI means systems that pursue goals and take actions autonomously. Respondents saw potential benefits in debugging and optimizing pipelines, combining data from different systems, and coordinating work. Interviewees anticipated engineers spending more time on architecture and business problems as agents take on operational tasks.
Alex: But those are potential benefits. If agents act rather than merely suggest, the cost of bad data or inappropriate access could also increase.
Sam: The report makes that concern explicit. Security and privacy were respondents’ leading challenge as AI capabilities advance. New engineering tools also bring integration problems and data governance demands: the rules and oversight for managing data. Those tools still need clean inputs, secure access, and ways to trace where data came from.
Alex: There is another tension here. Adding tools to simplify individual tasks can leave the whole team maintaining a more fragmented system.
Sam: Tool sprawl is a prominent concern in the survey. The report recommends reviewing and consolidating the existing tool stack before buying more AI-enabled tools. Also keep the sponsorship visible: Snowflake sponsored the report, which states that its research was editorially independent.
Alex: Who should read the full document, and where should they start?
Sam: Data engineering leaders, and researchers studying enterprise AI adoption, should read it. Start with “Growing workloads and a shifting focus,” keeping estimates separate from forecasts. Then read “AI-powered data engineering” for the productivity claims and tool problems, and “Building influence” for the changing role. Its conclusion recommends AI expertise, business understanding, and communication skills—not just more coding.
Alex: And for someone who only needs the main lesson?
Sam: The report suggests that data engineers are gaining influence while absorbing more work. Remember the distinction: making a team more productive is not the same as making its workload lighter.
Alex: Thanks for listening.