At the Data Engineering Summit 2024 in Bangalore, Vishal Singh, Senior Director of Engineering at Slice, delivered an enlightening session on data observability. With extensive experience spanning roles at Razorpay, Flipkart, Intuit, and Teradata, Vishal brought a wealth of practical insights to the table regarding the evolving landscape of data observability. This article speaks about Vishal’s session on “Data Observability – The nth Pillar”.
Defining Data Observability
Vishal began by setting the stage, emphasizing that while the concept of observability has long been integral to system management, its application to data has become increasingly critical in today’s distributed and open-source-driven data environments. He highlighted that data observability goes beyond mere monitoring and alerting; it encompasses ensuring the health, quality, timeliness, and integrity of data across complex, interconnected systems.
Key Challenges and Trends
He pointed out a significant trend: while data and systems have evolved significantly over the past decade with the advent of Big Data and open-source technologies, the challenges of ensuring seamless data flow and quality have also grown. Vishal underscored that data observability tools must now contend with distributed data platforms and the need for real-time insights amidst myriad interconnected components.
Practical Application and Framework
Moving beyond theoretical definitions, Vishal shared how Slice approaches data observability practically. For instance, within Slice’s ecosystem encompassing products like Slice borrow, Slice UPI, and Slice account, data quality is intricately defined and monitored across various schemas and tables. Each component’s health is assessed based on predefined business-critical metrics, ensuring that data-driven decisions are reliable and actionable.
Business and Platform Observability
Vishal stressed the importance of aligning data observability with business objectives. He introduced concepts like business observability, which focuses on correlating key business metrics with underlying data health indicators. This approach enables stakeholders to understand how changes in data quality or timeliness impact broader business outcomes, such as customer experience or operational efficiency.
Moreover, platform observability plays a crucial role in managing costs and optimizing resource utilization within Slice’s data infrastructure. Vishal highlighted how leveraging comprehensive metrics and logs enables proactive cost management and rapid anomaly detection across their diverse tech stack, which includes Kafka, Spark, Flink, and more.
Security and Compliance
Lastly, Vishal addressed the non-negotiable aspects of compliance and security in fintech, emphasizing Slice’s efforts to integrate observability with stringent regulatory requirements. This includes monitoring access patterns, data usage, and anomalies to ensure data integrity and regulatory compliance at all times.
Conclusion
In conclusion, Vishal Singh’s session at DES 2024 provided a comprehensive overview of data observability’s evolution, challenges, and practical implementations. His insights underscored the critical role of observability in modern data ecosystems, where robust monitoring, real-time insights, and alignment with business objectives are essential for driving value and maintaining trust in data-driven decision-making.
As data environments continue to evolve, Vishal’s perspectives serve as a guiding light for organizations navigating the complexities of data observability in an increasingly interconnected world.
