{"id":596427,"date":"2023-01-11T05:49:47","date_gmt":"2023-01-11T11:49:47","guid":{"rendered":"https:\/\/news.sellorbuyhomefast.com\/index.php\/2023\/01\/11\/how-observability-designed-for-data-teams-can-unlock-the-promise-of-dataops\/"},"modified":"2023-01-11T05:49:47","modified_gmt":"2023-01-11T11:49:47","slug":"how-observability-designed-for-data-teams-can-unlock-the-promise-of-dataops","status":"publish","type":"post","link":"https:\/\/newsycanuse.com\/index.php\/2023\/01\/11\/how-observability-designed-for-data-teams-can-unlock-the-promise-of-dataops\/","title":{"rendered":"How observability designed for data teams can unlock the promise of DataOps"},"content":{"rendered":"
\n

Check out all the on-demand sessions from the Intelligent Security Summit here<\/a><\/em>.<\/p>\n

\n<\/div>\n

These days, it\u2019s no exaggeration to say that every company is a data company. And if they\u2019re not, they need to be. That\u2019s why more organizations are investing in the modern data stack (think: Databricks and Snowflake, Amazon EMR, BigQuery, Dataproc). <\/p>\n

However, these new technologies and the increasing business-criticality of their data initiatives<\/a> introduce significant challenges. Not only must today\u2019s data teams deal with the sheer volume of data being ingested on a daily basis from a wide array of sources, but they must also be able to manage and monitor the tangle of thousands of interconnected and interdependent data applications.\u00a0<\/p>\n

The biggest challenge comes down to managing the complexity of the intertwined systems that we call the modern data stack. And as anyone who has spent time in the data trenches knows, deciphering data app performance, getting cloud costs under control and mitigating data quality issues is no small task.\u00a0<\/p>\n

When something breaks down in these Byzantine data<\/a> pipelines, without a single source of truth to refer back to, the finger-pointing begins with data scientists blaming operations, operations blaming engineering, engineering blaming developers \u2014 and so forth and so on in perpetuity.\u00a0<\/p>\n

<\/p>\n
\n

Event<\/h3>\n
\n

Intelligent Security Summit On-Demand<\/span><\/p>\n

Learn the critical role of AI & ML in cybersecurity and industry specific case studies. Watch on-demand sessions today.<\/span><\/p>\n<\/div>\n


\n Watch Here <\/a>\n <\/p>\n<\/div>\n

<\/body><\/p>\n

Is it the code? Insufficient infrastructure resources? A scheduling coordination problem? Without a single source of truth for everyone to rally around, everybody uses their own tool, working in silos. And different tools give different answers \u2014 and untangling the wires to get to the heart of the problem takes hours (even days).<\/p>\n

Why modern data teams need a modern approach<\/h2>\n

Data teams today are facing many of the same challenges that software teams once did: A fractured team working in silos, under the gun to keep up with the accelerated pace of delivering more, faster, without enough people, in an increasingly complex environment.\u00a0<\/p>\n

Software teams successfully tackled those obstacles via the discipline of DevOps. A big part of what enables DevOps teams to succeed is the observability provided by the new generation of application performance management (APM). Software teams are able to accurately and efficiently diagnose the root cause of problems, work collaboratively from a single source of truth, and enable developers to address problems early on \u2014 before software goes into production \u2014 without having to throw issues over the fence to the Ops<\/a> team.\u00a0<\/p>\n

So why are data teams struggling when software teams aren\u2019t? They\u2019re using basically the same tools to solve essentially the same problem. <\/p>\n

Because, despite the generic similarities, observability for data teams is a completely different animal than observability for software teams.\u00a0<\/p>\n

Cost control is critical<\/h2>\n

First off, consider that in addition to understanding a data pipeline\u2019s performance and reliability, data teams must also grapple with the question of data quality \u2014 how can they be assured that they are feeding their analytics engines with high-quality inputs? And, as more workloads move to an assortment of public clouds, it\u2019s also vital that teams are able to understand their data pipelines through the lens of cost.<\/p>\n

Unfortunately, data teams find it difficult to get the information they need. Different teams have different questions they need answered, and everybody is myopically focused on solving their particular piece of the puzzle, using their own particular tool of choice, and different tools yield different answers.<\/p>\n

Troubleshooting issues is challenging. The problem could be anywhere along a highly complex and interconnected application\/pipeline for any one of a thousand reasons. And, while web app observability<\/a> tools have their purpose, they were never intended to absorb and correlate the performance details buried within a modern data stack\u2019s components or \u201cuntangle the wires\u201d among a data application\u2019s upstream or downstream dependencies.\u00a0<\/p>\n

Moreover, as more data workloads migrate to the cloud, the cost of running data pipelines can quickly spiral out of control. An organization with 100,000-plus data jobs in the cloud has innumerable decisions to make about where, when, and how to run these jobs. And each decision carries a price tag.\u00a0<\/p>\n

As organizations cede centralized control over infrastructure, it\u2019s essential for both data engineers and FinOps to understand where the money is going and identify opportunities to reduce\/control costs.<\/p>\n

To get fine-grained insight into performance, cost, and data quality, data teams are forced to cobble together information from a variety of tools. And, as organizations scale their data stacks, the vast amount of information (and sources) makes it extraordinarily difficult to see the entirety of the data forest when you\u2019re sitting in the trees.\u00a0<\/p>\n

Most of the granular details needed are available \u2014 unfortunately, they\u2019re often hidden in plain sight. Each tool provides some of the information required, but not all. What\u2019s needed is observability that pulls together all these details and presents them in a context that makes sense and speaks the language of data teams. <\/p>\n

Observability that is designed from the ground up specifically for data teams allows them to see how everything fits together holistically. And while there is a slew of cloud-vendor-specific, open-source, and proprietary data observability tools that provide details about one layer or system in isolation, ideally, a full-stack observability solution can stitch it all together into a workload-aware context. Solutions that leverage deep AI are further able\u00a0to show not just where and why an issue exists but how it affects other data pipelines \u2014 and, finally, what to do about it.<\/p>\n

Just like DevOps<\/a> observability provides the foundational underpinnings to help improve the speed and reliability of the software development lifecycle, DataOps observability can do the same for the data application\/pipeline lifecycle. But \u2014 \u200aand this is a big but<\/em> \u2014 \u200aDataOps observability as a technology has to be designed from the ground up to meet the different needs of data teams.<\/p>\n

DataOps observability cuts across multiple domains:<\/p>\n