In May 2024, Bengaluru played host to India’s biggest generative AI conference focused on data engineering innovation—the Data Engineering Summit 2024. Among the many experts who graced the event was Ashish Singh, the Senior Director of Data Engineering at Idexcel. With nearly two decades of experience, Ashish is a data strategist whose expertise spans across fintech, investment banks, retail, and more. Known for his technical depth in big data ecosystems and data warehousing, Ashish has a proven track record of aligning data strategies with business needs to drive impactful innovation.
The Evolution of Data Architecture
The Historical Context
Ashish Singh commenced his session with a brief historical overview of data architecture, emphasizing the importance of understanding the past to avoid repeating mistakes. He traced the origins back to the 1990s when Bill Inmon and Ralph Kimball proposed the Snowflake and Star Schema centralized repositories. This era marked the transition from simple databases to comprehensive data warehouses, which became the centralized “Center of Truth” for organizational data.
The Emergence of ETL Processes
In tandem with the evolution of data warehousing, the ETL (Extract, Transform, Load) processes emerged, further evolving into ELT (Extract, Load, Transform). These processes revolutionized how data was transferred from file systems to databases, enhancing the efficiency and accuracy of data warehousing.
The Challenge of Big Data
With the explosion of internet speed and global customer service, data challenges grew exponentially, characterized by the “Three Vs”—volume, velocity, and variety—soon expanding to five or more Vs. Companies like Yahoo and Google pioneered solutions like MapReduce and Big Data systems to handle these burgeoning data challenges. However, the complexity of coding in MapReduce, particularly in Java, often posed significant hurdles for data engineers.
The Advent of Lambda Architecture
Solving Multi-V Challenges
To address the growing complexities, the Lambda architecture was introduced around 2010. This architecture initially discussed and partially published in academic papers, aimed to manage all data challenges within a single framework by combining batch and real-time processing. Despite its revolutionary approach, Lambda architecture was not without its shortcomings, such as its complexity, high implementation costs, and the necessity of managing two distinct processing layers.
A Closer Look at Lambda Implementation
Ashish presented one implementation of Lambda architecture, which involves separate layers for real-time and batch processing. The real-time layer handles short-duration processing, while batch processing occurs over longer intervals, typically ranging from four hours to a day. Despite its benefits in integrating batch and real-time views, this dual-layer approach often led to increased costs and complexity for data engineers.
The Rise of Kappa Architecture
Simplifying Real-Time Data Processing
In 2014, Jay Kreps introduced the Kappa architecture, simplifying the complexities of Lambda by unifying batch and real-time processing into a single layer. Kreps, known for co-founding Kafka and Confluent, leveraged Kafka’s capabilities to streamline data processing. This new architecture has since been adopted by major companies like Twitter, Uber, and Netflix, which handle trillions of events daily.
Implementing Kappa Architecture
The Kappa architecture revolves around immutable logs and stream processing. Data is ingested through various event producers and processed in real time using engines like Apache Flink and Apache Spark. This architecture allows both batch and real-time applications to access the same storage, reducing redundancy and cost. However, storing large volumes of data in Kafka can be expensive, prompting companies to adopt hybrid storage solutions that balance cost and performance.
Building Blocks of Kappa Architecture
Event Producers and Messaging Systems
Kappa’s architecture relies heavily on event producers and sophisticated messaging systems. Kafka, as the central messaging system, supports real-time data ingestion and ensures that events are processed efficiently. Other messaging systems have tried to compete, but Kafka remains the most effective solution for handling high-frequency, low-latency event processing.
Stream Processing and Data Storage
Stream processing engines like Apache Flink and Apache Spark play crucial roles in transforming and enriching data in real time. These engines apply various logic for validation, filtering, and data quality governance. Data storage solutions, whether on-premises or cloud-based, store processed data, balancing cost and accessibility.
Real-World Applications and Case Studies
Case Study: Twitter’s Transition
Ashish highlighted Twitter’s journey from Lambda to Kappa architecture. Initially, Twitter managed 400 billion events daily using a Lambda-based system. Over time, they transitioned to a hybrid Kappa system, integrating Google Cloud for storage and leveraging Kafka for real-time processing. This transition significantly reduced costs and improved scalability.
Industry Adoption
Other notable adopters of Kappa architecture include Uber, Shopify, Disney, and LinkedIn. These companies benefit from Kappa’s ability to handle large-scale, real-time data processing, which is crucial for their operational efficiency and customer service.
Future Directions: Beyond Kappa
Exploring GAA Architecture
Ashish concluded his talk by hinting at the next evolution in data architecture: the GAA architecture. This architecture, though not new, has been utilized by companies like Google for over a decade. GAA stands for Generalized Architecture for Analytics, encompassing seven layers that democratize each component of data processing. This modular approach ensures resilience, scalability, and flexibility, allowing systems to plug and play components as needed.
The Promise of Serverless Architectures
The trend towards serverless architectures is gaining momentum, with companies like Confluent offering managed Kappa services. Serverless solutions reduce the management overhead for organizations, allowing them to focus on data analytics rather than infrastructure maintenance. These solutions offer a self-driving car analogy, where users set the destination, and the system handles the journey.
Conclusion
Ashish Singh’s talk at the Data Engineering Summit 2024 provided a comprehensive overview of the evolution of data architecture, from the early days of data warehousing to the advanced Kappa and GAA architectures. His insights into real-world applications and future directions underscored the importance of continuous innovation in data engineering. As organizations strive to handle ever-growing volumes of data, the lessons from these architectural advancements will be crucial in shaping the future of data-driven decision-making.
