{"id":617005,"date":"2023-03-12T08:57:24","date_gmt":"2023-03-12T13:57:24","guid":{"rendered":"https:\/\/news.sellorbuyhomefast.com\/index.php\/2023\/03\/12\/3-ways-data-teams-can-avoid-a-tragedy-of-the-cloud-commons\/"},"modified":"2023-03-12T08:57:24","modified_gmt":"2023-03-12T13:57:24","slug":"3-ways-data-teams-can-avoid-a-tragedy-of-the-cloud-commons","status":"publish","type":"post","link":"https:\/\/newsycanuse.com\/index.php\/2023\/03\/12\/3-ways-data-teams-can-avoid-a-tragedy-of-the-cloud-commons\/","title":{"rendered":"3 ways data teams can avoid a tragedy of the cloud commons"},"content":{"rendered":"
\n
\n

\n

\"Humorous<\/p>\n

\n

Image Credit: gremlin\/Getty<\/em><\/p>\n<\/div><\/div>\n<\/p><\/div>\n

\n
\n
\n

In 1833, British economist William Forster Lloyd coined the term \u201cTragedy of the Commons\u201d to describe a situation in which individual users who have open access to a collective resource, unimpeded by formal rules that govern access and use, will act according to their own self-interest and contrary to the common good.<\/p>\n

In Lloyd\u2019s famous hypothetical, a group of individual herders share a public pasture for grazing their cattle. As each herder seeks to optimize his or her own economic gain by giving more of his or her cows access to graze, the commons eventually becomes depleted to the detriment of all.\u00a0<\/p>\n

In other words, when an infinite and seemingly \u201cfree\u201d resource is offered up to be used with little consideration of cost or consequence, it becomes unsustainable.\u00a0<\/p>\n

There\u2019s a similar phenomenon happening in today\u2019s cloud-first data operations (dataops) environment. The \u201ccommons\u201d in this case is the public cloud<\/a>, a shared resource that appears to be free to the data teams using it since they have little visibility into what their cloud usage actually costs.\u00a0<\/p>\n

Crisis in the cloud<\/h2>\n

Industry analysts estimate that at least 30% of<\/a> <\/a>cloud spend is \u201cwasted\u201d<\/a> each year \u2014 some $17.6 billion. For modern data pipelines in the cloud, the percentage of waste is significantly higher, estimated at closer to 50%.<\/p>\n

It\u2019s not hard to understand how we got here. Public cloud services like AWS and GCP have made it easy to spin resources up and down at will, as they\u2019re needed. Having unfettered access to a \u201climitless\u201d pool of computing resources has truly transformed how businesses create new products and services and bring them to market.\u00a0<\/p>\n

For modern data teams, this \u201cdemocratization of IT\u201d facilitated by the public cloud has been a game-changer. For one thing, it\u2019s enabled them to be far more agile as they don\u2019t need to negotiate and justify a business case with the IT department to buy or repurpose a server in the corporate data center. And as an operational expenditure, the pay-by-the-drip model of the cloud makes budget planning seem more flexible.<\/p>\n

However, the ease with which we can spin up a cloud instance doesn\u2019t come without a few unintentional consequences \u2014 forgotten workloads, over-provisioned or underutilized resources \u2014 with results including spiraling and unpredictable costs. Near-infinite cloud resources make it easy to simply throw additional compute resources at inefficient queries.<\/p>\n

The practice of FinOps has emerged in part as a response to this democratization of IT. The unifying principle of FinOps is that by bringing finance, engineering and business teams together to make better decisions around cost and performance, they will act in a more responsible manner \u2014 provided they have access to the right data to inform their decision-making.<\/p>\n

According to the 2022 State of FinOps <\/a>report<\/a>, the biggest challenge facing organizations trying to establish a FinOps culture is \u201cgetting engineers to take action on cost optimization.\u201d The authors go on to say that with so many data projects on their backlog and nearly unlimited cloud resources at their disposal, it\u2019s understandable that data engineers naturally prioritize new data pipeline creation and timely data delivery over resource optimization.<\/p>\n

While this is sound advice, this type of generalized guidance glosses over just how difficult a task this can be, and begs the question: How can data engineers be accountable if they can\u2019t capture accurate and easy-to-understand metrics about actual usage requirements? Moreover, how do you encourage this type of accountability without sacrificing cloud agility?<\/p>\n

Empowering data teams via feedback loops<\/h2>\n

One powerful mechanism to change behavior is providing people with information about their actions in real time so they can alter their behavior accordingly. This is the fundamental premise of a positive feedback loop.<\/p>\n

For instance, think about the black box that is residential electricity consumption. Few of us have real-time access to utility pricing or a sense of how much it really costs us to run a household appliance. But connect a smart meter to an outlet and suddenly you can just look at an app on your phone and understand at a much more granular level precisely how much energy each device that\u2019s plugged in is using and therefore what it\u2019s costing you.<\/p>\n

It\u2019s also important to consider the role that behavior theory and incentives play in shaping how we make decisions. In the context of cloud consumption, the incentives at work for a data engineer are quite different from those of the finance director. The data engineer is primarily motivated by and held accountable to metrics related to performance and reliability. They want to know: Are my applications running reliably, on time, every time?<\/p>\n

In the engineer\u2019s calculus, they\u2019ve become conditioned to overestimate the resources an application might require rather than having to \u201cguesstimate\u201d their perceived capacity requirements. It\u2019s not that they are intentionally over-provisioning resources; rather, they simply don\u2019t know exactly how many or what size resources are actually needed, so they guess, erring on the side of too much rather than too little.<\/p>\n

In order for engineers to take action on cost optimization, they need to be given the granular-level usage details that enable them to make informed and defensible choices \u2014 and do so without worrying that they will fall short on their service-level obligations.<\/p>\n

Getting at this information, however, is anything but easy. The data pipelines that feed modern data apps are enormously complex and the sheer size and scale of the data workloads only amplifies the challenge of identifying cost-saving opportunities.<\/p>\n

A flight path to cloud usage observability<\/h2>\n

This is the problem that full stack observability, informed by AI<\/a> algorithms and machine learning<\/a> models, was designed to address. There are a several ways in which the deep visibility that observability enables can help data teams more fully understand their usage costs and nudge their behavior to become more cost-conscious. <\/p>\n