5. Generative AI Will Be Used by Data Engineers on Nearly Every Project, Improving Productivity by a Third
Prediction: “Generative AI will become an integral tool for data engineers, improving productivity by automating data tasks and driving user interaction through AI-powered interfaces.”
What Happened: This prediction also has largely materialized. Generative AI has transformed the way data teams work, especially with tasks like semantic search, data discovery, and pipeline taiwan rcs data automation. in data tools has led to significant productivity boosts, and as expected, AI has become embedded in data workflows. The foundation for self-service data products, which generative AI helps power, is also paving the way for more democratized and accessible data in organizations. That said, we have no idea if teams have seen a full 30% productivity gain, as many teams are not yet thinking “AI-first” in their projects and pipeline work.
Measuring Up
In aggregate, I’d give us a B, maybe a B+, on last year’s predictions. They were useful to the market, and if used on the starting lineup, we would have won. For those that went all in and established an Iceberg lakehouse and leveraged DataOps best practices to generate data products with GenAI, chances are they would win the 2024 championship.