I will talk through the details of how SLOs at Zalando have evolved from the initial implementation ("SLOs for everything!") to the challenge of ensuring SLOs have the organizational power to drive changes in engineering priorities, to the current design of "critical business operations" and SLOs on those operations.

I'll discuss how to address the "fast burn" SLO problem by leveraging distributed tracing to identify regression in the customer experience automatically. When those regressions are identified, automatically identify and page the team best empowered to address them.

I'll discuss how to address the "slow burn" SLO problem through periodic operational review meetings, in which the SLOs are evaluated, and violations to the SLO (or slow burn issues) are allocated to an owner to investigate and address.

Lastly, I'll talk about challenges with the existing approach, including the difficulty of modelling event systems as a reliable flow, difficulty in rolling out more SLOs for non-customer-facing aspects of the organization and returning to service-specific SLOs.

For any large scale production system, the ability to effectively forecast potential capacity issues is crucial for the smooth functioning of the environment. With a reliable prediction, teams can proactively plan ahead, implement necessary scaling changes in a controlled manner and avoid unexpected availability issues that can cause stress and harm to the system.

Before implementing Tamland, the capacity planning process at GitLab.com was ad-hoc, and relied heavily on manual processes and intuition. Unfortunately, this approach often resulted in oversights, with issues going unnoticed until it was too late, sometimes only surfacing when site availability was impacted.

This talk delves into how GitLab leveraged the power of statistical analysis to greatly improve its capacity planning process. The session will be a practical demonstration of how we analyse long-term metrics data using the Meta’s Prophet library to build sophisticated forecast models.

Tamland, the capacity planning tool built by GitLab, is an open-source project and attendees will have access to the source code if they're interested in exploring the implementation in greater detail. This session is for anyone interested in learning about how forecasting libraries such as Prophet, Greykite, or NeuralProphet, and how they can be integrated into an observability system to provide greater insight into the health of a system.

Point of Change compliance is a deployment concept in which we leverage Kubernetes Admission controllers to block/allow deployments into the environment at the boundary of each environment. We can combine this concept with OpenSLO to create a powerful 0-trust architecture for error budgets. Imagine if we enforced error budgets at the boundary of the environment! Teams will begin to take them more seriously as a result.

---

Tome is our internal platform for managing, reporting and alerting on SLOs. A design goal was to enable SLOs to be defined with Configuration as Code. This has become the primary way that SLOs are maintained within Atlassian. Working with SLOs this way has many benefits:

• Enforces consistency in how we organise, define and validate SLOs
• Changes are tracked and attributed through a version control system
• Updates can be deployed as part of existing continuous integration and delivery pipelines

We have written a custom Terraform plugin for provisioning SLOs using Terraform, which interfaces with Tome's backend API.

• Optimizes deployments by tracking deployment state and applying diffs only
• Performs custom validation on configurations

In this talk I'll run through our implementation of the plugin and associated API, and how it compares to other SLO configuration systems.

I'll cover Flexera's journey from our internally developed SLO solution, to partnering with NOBL9, and also include how we engaged teams to help develop SLO's. I'll also cover how SLO's are now part of our engineering goals for 2023.

I'll talk about Grafana's experience using SLOs to help us describe our critical operations for our systems and monitor them. We had to do several things to make this successful:

• Promote consistent metric and objective definitions across teams.
• Create tooling to make it easy to define SLOs.
• Create tooling for an as-code workflow with SLOs.
• Create and nurture a culture that sees the value of SLOs for both the team and the company.

I'll be talking about the history at Grafana and also presenting results of interviews with some Grafana engineers about what SLOs have done for their teams and where they see room for improvement in implementations.

Once you have implemented SLOs for your organization how do you move forward?
At Polar Signals, we have quarterly SLO reviews. We're first doing a retrospective and discussing where we did great and also could have improved. For the upcoming quarter, we discuss the OKRs and from those, we derive SLOs. Sometimes OKRs are easy to derive from and sometimes they need to be rephrased to make sense for SLOs.

Matthias will walk you through an example of an SLO that we implemented quarters ago and how it changed over time. The example will showcase the SLO tracked in the open-source Pyrra project which makes SLOs with Prometheus manageable, accessible, and easy to use for everyone.

I'm pretty bad at time management, so I was looking into ways to improve that part of my life.

I turned to a lo-fi way to use SLOs to manage my day by turning my tasks and other things I need to do during the day into SLOs.

It's increased my productivity and am interested in sharing it with others!

I will be focusing on how we’ve built observability platform that include incident analytics > auto remediations and secrets management which in turn brought down MTTR and other SLOs for our production services.

I will be covering high level architecture of our internal developer platform, observability Platform and their interaction. Will talk through each component involved such as backstage.IO, grafana stack (grafana, mimir, Loki and temp), vector.dev and stackstorm.

Will conclude by showing the northstar metrics we’ve used to track internally and how they helped increasing the platform adoption.

SREs, as the name implies, care about service reliability. But, often, they struggle with having a way to define, measure and assess their services reliability. In practice, they lack a Reliability Framework.

How can SLOs help? They provided an opinionated way to do just that: define, measure and assess service reliability from the users perspective. They provide a common language to talk about reliability and prioritize work. They help fix the anti-pattern of trying to ensure service reliability without clearly defining what it means.

Recording rules is a clever concept introduced by Prometheus for storing results of query expressions in a form of a new time series. This concept is used for SLO calculations. But due to the nature of recording rules they have no retroactive effect. And since SLO objective usually captures a time window no less than 30d, recording rules produce incomplete results until the whole time window is captured.

The talk will cover how this can be fixed in VictoriaMetrics monitoring solution via retroactive rules evaluation on example of rules generated via https://github.com/slok/sloth framework.

While the direction and intent of SRE has been established and is becoming better-understood, the details on how to achieve "SRE" is still an exercise left for the reader.

Steve from Google and Sal from Nobl9 will present two independently developed methods for teaching SRE topics to customers, which we have discovered are actually quite similar. Huzzah!

Steve will present the "reliability map" and Sal will show Nobl9's SLODLC. Both are essentially sets of documentation presented in a way that allows customers to take a *guided* path through the dark and scary woods that is today's SRE.

The "reliability map" provides a detailed accounts of what development, infrastructure, operational, observability, and people/culture activities organizations can expect to be practicing to achieve different eras of reliability (99%, 99.9%, 99.99%, 99.999%). The "reliability map" takes it one step further and provides descriptions and references to additional content for all activities.

SLODLC provides a framework of documentation and templates to allow customers to iterate on improving their SLOs. This is especially necessary when customers have high reliability targets in which rely upon automation to execute predetermined playbooks. (rollback, exponential backoff, autoscale, etc) If SLOs are the indicator the performance of a user journey and decide when actions needs to be taken, then customers need a clear and useable way to explain:

• What they are attempting to measure (golden signals?, something else?)
• Why was it decided to measure X in such a way?
• How is X impactful for the targeted user journey?
• When was the last time the SLO objective, metric, time window, etc was changed?
• When the error budget is in danger of being breeched what actions should be take?

In short both projects are approaching the same topic of helping customers improve their reliability from different points of view and they synergize well together.

My background has mostly been in developing operating systems for data storage companies. In this environment, almost everything is controlled internally. For example, if the SLO of the operating system is five 9s, then the error budget is usually all consumed by software bugs owned internally by the company. As I switched to developing SaaS products in the cloud, this has drastically changed. Below are examples of lessons learned during different phases of working on a product, from development, to production, to support and operation. My goal is sharing these lessons so other developers can learn from my experience.

In my previous role, two main things I relied on while developing operating system products were the suite of testing as well as the release cadence. Shipping a new operating system every 6 months was considered fast. This gave developers a lot of time for their code to soak and be tested internally before being released to customers. Additionally, with a slower release cadence there was a lot of effort invested in creating different layers of testing. After all, a bug fix would take months to reach customers. Even if we released a patch quickly, which in this context means a few weeks, customers would still need to update their operating systems to apply that patch, and who knows how long a customer will wait before applying that patch. After moving to building SaaS products in the cloud, the release cadence became much faster. This required shifting my thought process. Instead of relying on soak time and various levels of testing, activities such as code review and in-build unit tests now take a front row seat. Metrics like code coverage from unit tests mattered more, while metrics like how long it's been since QA found a bug mattered less.

Another difference between my old role and new role is how developers access production and production metrics. In the old role, gathering production metrics was no easy feat.
Harder access to production metrics, implied that changing SLOs internally was harder, took longer, and a lot of developers didn't know how/why SLOs were changing. For a SaaS product running in the cloud, developers have access to overall system performance metrics at the click of a button, while maintaining compliance. This makes it easier for developers to know why/how SLOs are affected by SLAs and also gives them faster reaction times.

From the operation perspective, the old role and the new role were quite different. In the old role developers didn't go on-call. There was a customer support organization that would handle any customer issues first. Developers were brought in occasionally if customer support needed a bug fixed. Developers didn't need to wake up in the middle of the night because there's an outage. In the new role, developers are on-call, which means developers can and do occasionally get paged in the middle of the night. This shift caught me off-guard as I had to think much harder about how my service impacts the quality of life of my colleagues and myself. It made me think about how to change and measure SLOs to eventually avoid waking someone up in the middle of the night.

Another lesson that caught me off guard is not all SLAs can be trusted when working in the cloud. Yes, I knew this lesson theoretically, but learning it in practice is a different story. One example was an outage that was caused by a cloud provider breaching their SLAs for a managed service that we relied on for our product. This resulted in our product breaching its SLAs. When something like this happens, you think hard about how to update your SLOs to prepare for these issues, catch such issues, and react to them. I found that using SLOs that are tighter than SLAs was helpful in that regard.

In conclusion, even with years of professional experience, moving from developing On-Prem products to cloud SaaS offerings changed how I think about SLOs and it's truly not a one-size-fits-all.

Adopting an SLO culture involves identifying the metrics that matter without drowning in noise and alert fatigue. The black box concept lets us aggregate granular metrics into SLIs that focus on the user experience as an indicator of system reliability.

The talk will be based off this article published on Procore's Engineering blog: https://careers.procore.com/blogs/engineering-at-procore/black-box-slis.

By managing Service Level Objectives as code we can co-locate SLO definitions and ownership with the product code and team. This supports horizontal scaling of SLO ownership, while establishing a single source of truth, adding transparency, integrating with the code management process, and creating an audit trail for SLOs.

The talk will be based off this article published on Procore's Engineering blog: https://engineering.procore.com/implementing-slos-as-code-a-case-study-2/ 

In my experience, many teams encounter SLOs as something they've been told to "do" and the flip side is it's been something I've been asked to help them with. Naturally this is not ideal - as engineers we prefer things to be self-evident.

I have a number of strategies to use when communicating the process and the value of creating and using SLOs. I'll give away the thing I say the most right here: "SLOs are for decision-making". They're not magic or even a single thing. They're a tool that helps us do our jobs.

Audience will come away either better-able to articulate the pragmatic usefulness of an SLO mindset, or with one or two real motivations why they should consider developing and using SLOs for their own systems.

While the direction and intent of SRE has been established and is becoming better-understood, the details on how to achieve "SRE" is still an exercise left for the reader.

Steve from Google and Sal from Nobl9 will present two independently developed methods for teaching SRE topics to customers, which we have discovered are actually quite similar. Huzzah!

Steve will present the "reliability map" and Sal will show Nobl9's SLODLC. Both are essentially sets of documentation presented in a way that allows customers to take a *guided* path through the dark and scary woods that is today's SRE.

The "reliability map" provides a detailed accounts of what development, infrastructure, operational, observability, and people/culture activities organizations can expect to be practicing to achieve different eras of reliability (99%, 99.9%, 99.99%, 99.999%). The "reliability map" takes it one step further and provides descriptions and references to additional content for all activities.

SLODLC provides a framework of documentation and templates to allow customers to iterate on improving their SLOs. This is especially necessary when customers have high reliability targets in which rely upon automation to execute predetermined playbooks. (rollback, exponential backoff, autoscale, etc) If SLOs are the indicator the performance of a user journey and decide when actions needs to be taken, then customers need a clear and useable way to explain:

• What they are attempting to measure (golden signals?, something else?)
• Why was it decided to measure X in such a way?
• How is X impactful for the targeted user journey?
• When was the last time the SLO objective, metric, time window, etc was changed?
• When the error budget is in danger of being breeched what actions should be take?

In short both projects are approaching the same topic of helping customers improve their reliability from different points of view and they synergize well together.

The talk tells the story of how conversations around SLOs can be a great trigger to start a shift to a Product Thinking mindset, with practical examples. We'll also take a light dip into constraint mapping from an SLO perspective.

With a Serverless mindset in place, there should no longer be a need to define resource requests or any other kinds of information. Just developed your app or function and run it.

But how can we achieve performance and make sure we run the system most efficiently? This is were we can now let user define what they truly care about - their SLOs. Based on these performance targets our Intent Driven Orchestration Planner will do the rest. No need to define resource requests and limit s on e.g. Kubernetes cluster anymore. The planner will set up the systems in such a way that your app/functions behave as expected, without the need to know anything about the underlying infrastructure – less knowledge needed, fewer errors made!

This is a shift in how we do orchestration and SLO management, that could be of interest to this community. Away from a monitoring & alerting way of looking SLOs, towards a way of using SLOs to manage the system. Furthermore this truly allows for ease-of-use of the user; rather than defining numbers an values based on domain & contextualized knowledge we let them define what they truly care about: their SLOs!

There's been a lot of (really good!) discussion in the last several years about how to think about, monitor, and alert on long-tailed distributed quantities, such as latency. However, I'm worried that in our desire to describe the long tail we may be too eager to abandon tools that are still useful.

In this talk I'll (re)introduce everyone's favorite summary statistic, the average. I'll talk about the difference between a sample average and a random variable's expectation, and how the two are uniquely linked by the law of large numbers. I'll also talk about how the central limit theorem allows us to treat sample averages as draws from a Gaussian distribution, irrespective of the distribution the samples come from, and then I'll talk about the exceptions. We'll finish up by looking at how the properties of expected values can give us insight into the behavior of systems, even at the tail.

Through all of this we'll be looking at examples drawn from real-world latency data, and comparing the insights gleaned from this versus other common summary statistics.

Resiliency is a critical piece to building reliable systems. It allows us to feel safe knowing failure is inevitable. After all, as noted in the OG SRE book, 100% is terrible target for basically everything.

We spend a lot of resources to add in layers of resiliency from redundant multi-region compute stacks to backups on our backups. How do we know this resilience achieved our ultimate outcome of reliability for our customers? We'll discuss the ways to observe your SLOs and error budgets during resiliency events.

The various events we'll look at include, regional failover, chaos experiments (such as latency injection), database recovery, and more! After failing a region, do we know if your customer's experienced a disturbance? When you're running a resiliency test or game day, how do you measure success?

Observing error budgets before, during, and after an event paint a picture of our customer's experience and can ultimately be part of the success criteria. It is critical that we know how our architecture and system changes unfold. What if new resiliency introduces latency that negatively impacts your customer? For example, if there's complexity introduced that makes your release engineering more convoluted, we may see a longer error budget burn while we remediate. On the flip side the partnership of adding resiliency and observing your SLOs can also lead to improving the objectives with newly matured levels of resiliency; raising the bar for performance.

At eBay we use a wrapped version of Prometheus as our centralized time series database. Data residing in Prometheus is used for mission critical functions like detecting issues on the site through anomaly detection, SLOs or simple threshold based alerts. To be able to enforce SLOs across the 1000s of micro services that are deployed inside of eBay, we would need to be able to aggregate commonly instrumented metrics at an application level.

Why are these aggregations necessary? Host level metrics when used for computation of things like burn rates over a 30 day window can become very very expensive. Using a single query across all micro services to generate such aggregations have several scale limitations. Manually onboarding each micro service into such aggregations can also be tedious.

The talk discusses our commonly instrumented metrics, our metrics store architecture, scale challenges seen with SLI/burn rate computation and how we came up with concepts like templated rule groups that automatically generate Prometheus rule groups to be able to do aggregations for every application that is emitting metrics.

This sounds like the start to a terrible bar joke, and you're absolutely right it is. Soooooooooooooo a Data Science Engineer, M&O Engineer and a DevOps Engineer met via a zoom chat from two completely different regions and time zones.

Our organization, Outsystems is a low code development platform that wanted to monitor the reliability of our customers environments. Due to the nature of these environments we were looking to design SLOs that fit unknown and constantly changing architecture. To further complicate matters there were also competing technology stacks the traditional legacy platform and the next-generation. We needed an automated solution that could dynamically fit into their build pipelines to measure the reliability of our customer’s environments at scale.

This is a talk on how three engineers worked thru buy-in, cultural, and technical challenges to come together and use our specialties to create a fully automated SLO creation factories utilizing the OpenSLO framework for our products that allows us to scale at a moments whim.

In this talk I would like to show a utility for generating service level objectives rules for Prometheus/AlertManager based on the OpenSLO v1 specification.

The talk shows a tool that has been developed by Tapico for generating the configuration files for Prometheus and Alertmanager based on OpenSLO specifications.

Every organization typically identifies a “North Star” for use in guiding its business as it navigates its business, technologies, and customer experiences. Navigation via only one celestial body however is inefficient and prone to environmental constraints. Navigation using multiple known celestial bodies, which are known and imbued with context, prevents environmental and operational constraints.

Identifying a contextual North Star for a technology organization provides benefit, but identifying a constellation of service level objectives provides a set of greater benefit. SLOs used as the primary navigational devices offer transformative advantages in navigating the business, technology, and customer experience challenges.

Leading organizational change always presents challenges. Leveraging Service Level Objectives, we are transforming multiple aspects of our business and operational decision making processes. SLOs are providing innovative technical, operational, and business accelerations across our SRE Transformation, SDLC, and operation of both existing and emerging products.