The Data Engineering Summit 2024 held on May 30-31 in Bangalore brought together experts from diverse sectors to discuss and anticipate the future skills necessary for data engineers. This particular panel discussion, “The Future Skills of Data Engineers: What Will Be Essential in 5 Years?”, moderated by Chiranjeev Singh Sabharwal, Senior Director at Axtria, featured prominent voices like Puneet Pandhi, Vice President at American Express; Jaya Murugan Muthu Manickam, Senior Architect at Adobe; Pramod Rawat, Executive Director of Generative AI/ML, Data Analytics & Cloud Product Management at Wells Fargo; and Raghavendra Prasad Munikrishna, Vice President of Data Engineering at J.P. Morgan.
Setting the Context
The panel kicked off with Chiranjeev Singh Sabharwal highlighting the rapid evolution in data engineering and the need for professionals to anticipate future demands. He set the stage by reminiscing about the earlier days of data engineering, which predominantly involved ETL (Extract, Transform, Load) processes, and emphasized the need to move beyond this traditional view.
The Evolving Role of Data Engineers
From ETL to Comprehensive Data Management
Raghavendra Prasad Munikrishna reflected on the transition in data engineering roles over the past decade. He noted that the skills required today are significantly different from those needed a few years ago. The evolution from traditional ETL processes to more sophisticated automation, integration of AI, and real-time data processing is evident. Prasad emphasized the importance of staying updated with technological advancements to remain relevant in the field.
Puneet Pandhi expanded on this by categorizing data roles into three primary areas within his organization: data stewards, BI developers, and data quality engineers. He explained that data stewards act as pseudo business owners with deep knowledge of data, BI developers create insightful data pipelines using tools like PowerBI and Tableau, and data quality engineers ensure the integrity of data through anomaly detection frameworks. This breakdown underscores the diversity and complexity of data engineering roles beyond the simplistic view of ETL.
The Crucial Role of Data Governance
Jaya Murugan Muthu Manickam emphasized the growing importance of data governance. He likened data engineers to pit crew members in a Formula One race, supporting business stakeholders (the drivers) to achieve their goals. Jaya pointed out that data governance, encompassing data quality, lineage, and management, is essential for leveraging data as a strategic asset. This analogy highlighted the supportive yet critical role data engineers play in ensuring data integrity and value.
Pramod Rawat added that data privacy and classification within organizations are becoming increasingly important. He stressed the need for masking sensitive data, especially personal identifiable information (PII), to ensure compliance with regulations and protect customer data.
The Impact of Generative AI and ML
The conversation naturally shifted towards the impact of generative AI and machine learning (ML) on data engineering. Pramod provided a historical perspective, tracing the journey from mainframes and C++ programming to the advent of structured and unstructured data handling, big data, and eventually, cloud computing. He highlighted the importance of understanding business use cases to effectively leverage generative AI, cautioning against the high costs associated with its infrastructure.
Jaya shared a real-world example of how generative AI enhances customer experience through real-time personalization. He described a scenario where an e-commerce platform offered a personalized discount based on the user’s purchasing behavior, illustrating the potential of AI to drive customer engagement and sales.
Puneet provided a contrasting view, cautioning against over-relying on generative AI for every problem. He emphasized the importance of selecting use cases with significant ROI and the necessity of balancing traditional methods with new technologies.
Looking Ahead: Essential Skills for Data Engineers
Cloud Computing and Real-Time Data Processing
The panel unanimously agreed on the importance of cloud computing skills. Puneet suggested that understanding cloud offerings and obtaining relevant certifications are crucial for future data engineers. He also highlighted the need for expertise in real-time data processing frameworks like Apache Flink and messaging-based processes.
MLOps and Data Governance
MLOps emerged as another critical skill area. Prasad explained that the integration of machine learning operations within data pipelines will become increasingly important as AI and ML applications continue to grow. Puneet emphasized the significance of data governance and data quality, advising data engineers to stay informed about permissible practices and manage data effectively across on-premises and cloud environments.
Domain Knowledge and Business Acumen
Pramod and Jaya stressed the necessity of domain knowledge. Pramod shared his personal journey of evolving from a coder to understanding the broader business context of his work. Jaya echoed this sentiment, emphasizing the importance of understanding client needs and the “as-is” model before building solutions. The ability to translate business problems into technical solutions and communicate effectively with stakeholders was identified as a key competency.
The Future Landscape: Data Engineer as a Service
Prasad introduced the concept of “Data Engineer as a Service,” predicting that specialized firms will offer outsourced data engineering solutions. This trend will necessitate that data engineers within organizations become specialists, capable of managing comprehensive solutions end-to-end. He highlighted the increasing automation of traditional data pipelines and the integration of machine learning within data platforms, such as Snowflake’s Artic, as indicators of this shift.
Conclusion: Embracing Change and Lifelong Learning
The panel concluded with a unanimous call for continuous learning and adaptation. Puneet underscored the rapid pace of change in the industry, urging data engineers to stay updated with new technologies and frameworks. Pramod highlighted the importance of reflecting on recent learnings and maintaining a proactive approach to upskilling.
As data engineering continues to evolve, professionals must embrace a multifaceted approach, combining technical expertise, domain knowledge, and soft skills. The future will demand data engineers who are not only technically proficient but also capable of understanding and solving complex business problems. By staying informed and adaptable, data engineers can ensure their relevance and success in the ever-changing technological landscape.
