Join top executives in San Francisco on July 11-12, to hear how leaders are integrating and optimizing AI investments for success<\/em>. Learn More<\/a><\/em><\/p>\n

\n<\/div>\n
The year is 1999 and the internet has begun to hit its stride. Near the top of the list of its most trafficked sites, eBay suffers an outage \u2014 considered to be the first high-profile instance of downtime in the history of the world wide web as we know it today.\u00a0<\/p>\n
At the time, CNN described eBay\u2019s response<\/a> to the outage this way: \u201cThe company said on its site that its technical staff continues to work on the problem and that the \u2018entire process may still take a few hours yet.\u2019\u201d\u00a0<\/p>\n
It almost sounds like a few folks in a server room pushing buttons until the site comes back online, doesn\u2019t it?\u00a0<\/p>\n
Now, nearly 25 years later and in a wildly complex digital landscape with increasingly complex software powering business at the highest of stakes, companies rely on software engineering teams to track, resolve \u2014 and most importantly prevent \u2014 downtime issues. They do this by investing heavily in observability solutions like Datadog<\/a>, New Relic<\/a>, AppDynamics<\/a> and others.\u00a0<\/p>\n
<\/p>\n
\n
Event<\/h3>\n
\n
Transform 2023<\/span><\/p>\n
\n
Join us in San Francisco on July 11-12, where top executives will share how they have integrated and optimized AI investments for success and avoided common pitfalls.<\/p>\n<\/div>\n<\/div>\n

\n Register Now <\/a>\n <\/p>\n<\/div>\n
<\/body><\/p>\n
Why? In addition to the engineering resources it takes to respond to a downtime incident, not to mention the trust that is lost among the company\u2019s customers and stakeholders, the economic impact of a downtime incident can be financially catastrophic<\/a>.<\/p>\n
Preventing data downtime<\/h2>\n
As we turn the page on another year in this massive digital evolution, we see the world of data analytics primed to experience a similar journey. And just as application downtime became the job of massive teams of software engineers to tackle with application observability solutions, so too will it be the job of data teams to track, resolve, and prevent instances of data downtime.\u00a0<\/p>\n
Data downtime refers to periods of time where data is missing, inaccurate or otherwise \u201cbad,\u201d and can cost companies millions of dollars per year in lost productivity, misused people hours and eroded customer trust.\u00a0<\/p>\n
While there are plenty of commonalities between application observability and data observability, there are clear differences, too \u2014 including use cases, personas and other key nuances. Let\u2019s dive in.\u00a0<\/p>\n
What is application observability?<\/h2>\n
Application observability refers to the end-to-end understanding of application health across a software environment to prevent application downtime.\u00a0<\/p>\n
Application observability use cases<\/h3>\n
Common use cases include detection, alerting, incident management, root cause analysis, impact analysis and resolution of application downtime. In other words, measurements taken to improve the reliability of software applications over time, and to make it easier and more streamlined to resolve software performance issues when they arise.<\/p>\n
Key personas<\/h3>\n
The key personas leveraging and building application observability solutions<\/a> include software engineer, infrastructure administrator, observability engineer, site reliability engineer and DevOps engineer.<\/p>\n
Companies with lean teams or relatively simple software environments will often employ one or a few software engineers whose responsibility it is to obtain and operate an application observability solution. As companies grow, both in team size and in application complexity, observability is often delegated to more specialized roles like observability managers, site reliability engineers or application product managers.\u00a0<\/p>\n
Application observability responsibilities<\/h3>\n
Application observability solutions monitor across three key pillars:<\/p>\n
\n
Metrics:<\/strong> A numeric representation of data measured over intervals of time. Metrics can harness the power of mathematical modeling and prediction to derive knowledge of the behavior of a system over intervals of time in the present and future.<\/li>\n
Traces:<\/strong> A representation of a series of causally related distributed events that encode the end-to-end request flow through a distributed system. Traces are a representation of logs; the data structure of traces looks almost like that of an event log.<\/li>\n
Logs: <\/strong>An immutable, timestamped record of discrete events that happened over time.\u00a0<\/li>\n<\/ul>\nCore functionality<\/h3>\n
High-quality application observability possesses the following characteristics<\/a> that help companies ensure the health of their most critical applications:<\/p>\n
\n
End-to-end coverage across applications (particularly important for microservice architectures).<\/li>\n
Fully automated, out-of-the-box integration with existing components of your tech stack \u2014 no manual inputs needed.<\/li>\n
Real-time data capture through metrics, traces and logs.<\/li>\n
Traceability\/lineage to highlight relationships between dependencies and where issues occur for quick resolution.<\/li>\n<\/ul>\n
What is data observability? <\/h2>\n
Like application observability, data observability also tackles system reliability<\/a> but of a slightly different variety: analytical data.\u00a0<\/p>\n
Data observability is an organization\u2019s ability to fully understand the health of the data in its systems. Tools use automated monitoring, automated root cause analysis, data lineage and data health insights to detect, resolve and prevent data anomalies. This leads to healthier pipelines, more productive teams and happier customers.<\/p>\n
Use cases<\/h3>\n
Common use cases for data observability include detection, alerting, incident management, root cause analysis, impact analysis and resolution of data downtime. <\/p>\n
Key personas<\/h3>\n
At the end of the day, data reliability is everyone\u2019s problem, and data quality is a responsibility shared by multiple people on the data team. Smaller companies may have one or a few individuals who maintain data observability solutions<\/a>; however, as companies grow both in size and quantity of ingested data, the following more specialized personas tend to be the tactical managers of data pipeline and system reliability.<\/p>\n
\n
Data engineer:<\/strong> Works closely with analysts to help them tell stories about that data through business intelligence visualizations or other frameworks. Data designers are more common in larger organizations and often come from product design backgrounds.\u00a0<\/li>\n<\/ul>\n
\n
Data product manager:<\/strong> Responsible for managing the life cycle of a given data product and is often in charge of managing cross-functional stakeholders, product road maps and other strategic tasks.<\/li>\n<\/ul>\n
\n
Analytics engineer: <\/strong>Sits between a data engineer and analysts and is responsible for transforming and modeling the data such that stakeholders are empowered to trust and use that data.\u00a0<\/li>\n<\/ul>\n
\n
Data reliability engineer:<\/strong> Dedicated to building more resilient data stacks through data observability, testing and other common approaches.\u00a0<\/li>\n<\/ul>\n
Responsibilities<\/h3>\n
Data observability solutions monitor across five key pillars:\u00a0<\/p>\n
\n
Freshness:<\/strong> Seeks to understand how up-to-date data tables are, as well as the cadence at which they are updated.\u00a0<\/li>\n
Distribution:<\/strong> In other words, a function of data\u2019s possible values and if data is within an accepted range.\u00a0<\/li>\n
Volume:<\/strong> Refers to the completeness of data tables and offers insights on the health of data sources.\u00a0<\/li>\n
Schema:<\/strong> Changes in the organization of your data often indicate broken data.\u00a0<\/li>\n
Lineage:<\/strong> When data breaks, the first question is always \u201cwhere?\u201d Data lineage provides the answer by telling you which upstream sources and downstream ingestors were impacted, as well as which teams are generating the data and who is accessing it.\u00a0<\/li>\n<\/ul>\n
Core functionalities<\/h3>\n
High-quality data observability solutions possess the following characteristics that help companies ensure the health, quality and reliability of their data and reduce data downtime:\u00a0<\/p>\n
\n
The data observability platform connects to an existing stack quickly and seamlessly and does not require modifying data pipelines, writing new code or using a particular programming language.\u00a0<\/li>\n
Monitors data at rest and does not require extracting data from where it is currently stored.\u00a0<\/li>\n
Requires minimal configuration and practically no threshold-setting. Data observability tools should use machine learning (ML) models to automatically learn an environment and its data.\u00a0<\/li>\n
Requires no prior mapping of what needs to be monitored and in what way. Helps identify key resources, key dependencies and key invariants to provide broad data observability with little effort.<\/li>\n
Provides rich context that enables rapid triage, troubleshooting and effective communication with stakeholders impacted by data reliability issues.\u00a0<\/li>\n<\/ul>\n
The future of data and application observability<\/h2>\n
Since the Internet became truly mainstream in the late 1990s,\u00a0we\u2019ve seen the rise in importance, and the corresponding technological advances, in application observability to minimize downtime and improve trust in software.\u00a0<\/p>\n
More recently, we\u2019ve seen a similar boom in the importance and growth of data observability as companies put more and more of a premium on trustworthy, reliable data.\u00a0Just as organizations were quick to realize the impact of application downtime a few decades ago, companies are coming to understand the business impact that analytical data downtime incidents can have, not only on their public image, but\u00a0also on their bottom line.\u00a0<\/p>\n
For instance, a May 2022 data downtime incident<\/a> involving the gaming software company Unity Technologies sank its stock by 36% percent when bad data had caused its advertising monetization tool to lose the company upwards of $110 million in lost revenue.\u00a0<\/p>\n
I predict that this same sense of urgency around observability will continue to expand to other areas of tech, such as ML and security. In the meantime, the more we know about system performance across all axes, the better \u2014 particularly in this macroeconomic climate.\u00a0<\/p>\n
After all, with more visibility comes more trust. And with more trust comes happier customers.<\/p>\n
Lior Gavish is CTO and cofounder of Monte Carlo. <\/em><\/p>\n
\n
DataDecisionMakers<\/h3>\n
Welcome to the VentureBeat community!<\/p>\n
DataDecisionMakers is where experts, including the technical people doing data work, can share data-related insights and innovation.<\/p>\n
If you want to read about cutting-edge ideas and up-to-date information, best practices, and the future of data and data tech, join us at DataDecisionMakers.<\/p>\n
You might even consider\u00a0 contributing an article<\/a>\u00a0of your own!<\/p>\n
Read More From DataDecisionMakers<\/a><\/p>\n<\/div><\/div>\n
Read More<\/a>
\n Lior Gavish, Monte Carlo<\/p>\n","protected":false},"excerpt":{"rendered":"
Join top executives in San Francisco on July 11-12, to hear how leaders are integrating and optimizing AI investments for success. Learn More The year is 1999 and the internet has begun to hit its stride. Near the top of the list of its most trafficked sites, eBay suffers an outage \u2014 considered to be<\/p>\n","protected":false},"author":1,"featured_media":622457,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3428,74017,46],"tags":[],"class_list":{"0":"post-622456","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-difference","8":"category-observability","9":"category-technology"},"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/posts\/622456","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/comments?post=622456"}],"version-history":[{"count":0,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/posts\/622456\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/media\/622457"}],"wp:attachment":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/media?parent=622456"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/categories?post=622456"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/tags?post=622456"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}

Event<\/h3>\n\nTransform 2023<\/span><\/p>\n

What is application observability?<\/h2>\nApplication observability refers to the end-to-end understanding of application health across a software environment to prevent application downtime.\u00a0<\/p>\n

What is application observability?<\/h2>\n
Application observability refers to the end-to-end understanding of application health across a software environment to prevent application downtime.\u00a0<\/p>\n