Unlocking the Power of Data Observability

Explore vital insights on enhancing data observability from the latest DES 2024 event for actionable strategies.

At the Data Engineering Summit 2024 in Bangalore, Vishal Singh, Senior Director of Engineering at Slice, delivered an enlightening session on data observability. With extensive experience spanning roles at Razorpay, Flipkart, Intuit, and Teradata, Vishal brought a wealth of practical insights to the table regarding the evolving landscape of data observability. This article speaks about Vishal’s session on “Data Observability – The nth Pillar”.

Defining Data Observability

Vishal began by setting the stage, emphasizing that while the concept of observability has long been integral to system management, its application to data has become increasingly critical in today’s distributed and open-source-driven data environments. He highlighted that data observability goes beyond mere monitoring and alerting; it encompasses ensuring the health, quality, timeliness, and integrity of data across complex, interconnected systems.

He pointed out a significant trend: while data and systems have evolved significantly over the past decade with the advent of Big Data and open-source technologies, the challenges of ensuring seamless data flow and quality have also grown. Vishal underscored that data observability tools must now contend with distributed data platforms and the need for real-time insights amidst myriad interconnected components.

Practical Application and Framework

Moving beyond theoretical definitions, Vishal shared how Slice approaches data observability practically. For instance, within Slice’s ecosystem encompassing products like Slice borrow, Slice UPI, and Slice account, data quality is intricately defined and monitored across various schemas and tables. Each component’s health is assessed based on predefined business-critical metrics, ensuring that data-driven decisions are reliable and actionable.

Business and Platform Observability

Vishal stressed the importance of aligning data observability with business objectives. He introduced concepts like business observability, which focuses on correlating key business metrics with underlying data health indicators. This approach enables stakeholders to understand how changes in data quality or timeliness impact broader business outcomes, such as customer experience or operational efficiency.

Moreover, platform observability plays a crucial role in managing costs and optimizing resource utilization within Slice’s data infrastructure. Vishal highlighted how leveraging comprehensive metrics and logs enables proactive cost management and rapid anomaly detection across their diverse tech stack, which includes Kafka, Spark, Flink, and more.

Security and Compliance

Lastly, Vishal addressed the non-negotiable aspects of compliance and security in fintech, emphasizing Slice’s efforts to integrate observability with stringent regulatory requirements. This includes monitoring access patterns, data usage, and anomalies to ensure data integrity and regulatory compliance at all times.


In conclusion, Vishal Singh’s session at DES 2024 provided a comprehensive overview of data observability’s evolution, challenges, and practical implementations. His insights underscored the critical role of observability in modern data ecosystems, where robust monitoring, real-time insights, and alignment with business objectives are essential for driving value and maintaining trust in data-driven decision-making.

As data environments continue to evolve, Vishal’s perspectives serve as a guiding light for organizations navigating the complexities of data observability in an increasingly interconnected world.

Transform your team into AI powerhouses

Targeted suite of solutions for enterprises aiming to harness the power of AI. MachineHack is your partner in building a future-ready workforce adept in artificial intelligence.

Online AI Hackathons to accelerate innovation

With MachineHack you can not only find qualified developers with hiring challenges but can also engage the developer community and your internal workforce by hosting hackathons.