Hello and welcome to the first module of the SRE Practitioner Pathway.

This module will explore more common pitfalls that can be set in SRE shift, the problems they can create, and how to avoid them, in order to allow you to steer your organization towards a good SRE practice.

Many SRE antipatterns exist due to a lack of comprehensive understanding of what SRE actually is. Many people will be inclined to see only those aspects of SRE that pertain directly to the work they do.

So it's vitally important that SREs work collaboratively with others to support full integration of SRE practices and to guide people away from some of these bad practices.

It's also important that SREs themselves embrace the system’s thinking mindset and a holistic vision of how the work they do drives organizational success.

Some of the more commonly observed SRE antipatterns are driven by misunderstandings around what the role of an SRE actually is.

One example of this is the antipattern of simply rebranding existing operational teams as SREs

The biggest driver for SRE adoption within organizations is the inability of traditional operational practices to remain effective at the vast scales required by modern systems.

Keeping these existing toil-heavy practices in place will not lead to organizational success and can lead to a sense that SREs have failed to deliver on their value proposition, making renewed efforts harder to get off the ground.

Another common manifestation of this is the antipattern of allowing SREs to become gatekeepers of production, placing the burden of responsibility entirely on them to reject insufficiently reliable work.

This approach reinvents the DevOps silos of old and creates friction, blame, and mistrust between the developers and SREs.

SREs should instead act as enablers of reliability, working with other disciplines to ensure that reliability is achieved first time.

By having a holistic vision and working collaboratively with other disciplines across the IT value stream, SREs can better share wisdom of production, promote reliability as a shared responsibility, and push for changes to practices when necessary to reduce toil and improve operational efficiency, whilst avoiding these kinds of foundational misunderstandings.

Still more common antipatterns concern a lack of adequate effective measurement.

Measurement in SRE is heavily focused on understanding and improving the customer experience and monitoring observability practices should reflect this.

Always bear in mind the system's thinking adage, if you do not measure what you truly value, you'll end up valuing whatever you measure.

SREs should work closely with the business to ensure that SLOs are tied closely to real customer needs, an ambitious or overly ambitious SLO that did not reflect your customer’s reality, are a costly mistake.

SREs should also work closely with monitoring functions to ensure that the appropriate SLIs to track these targets are in place.

SREs should also ensure that measurements capture the most complete picture of the user experience possible and avoid falling into the trap of measuring to the edge.

That is tracking SLO performance based exclusively on data from internal monitoring within the service and not considering the effect of third-party services between you and your customer.

An issue in these services will still affect users even if it does not show on your dashboards and even if the issue lies within a third party, being aware of these issues without your users having to report them improves your customer confidence in your reliability.

Even with these concerns addressed, issues can still go undetected due to another common challenge, alert noise.

Noisy low-quality alerting not only adds toil for SREs, but can lead to a culture of alert being routinely ignored.

SRE should work to ensure that alerting is good quality with high signal-to-noise ratio to ensure that all alerts are investigated and any issues of knowledge in the most timely manner possible.

Finally, SREs must avoid common antipatterns associated with incident response.

One of the innovations that SRE brings to incident response is the swarming model, to be covered in more detail in a later module.

In this model tiered escalation-based incidents are out and collaborative first touch incident responses in.

SREs should be careful however to ensure that this does not give a rise to dog piling — unstructured, poorly prepared incident response activities which create chaos and confusion and ultimately increase resolution times.

Good swarming still follows pre-planned, well-drilled procedures and implements an effective command framework.

SREs should also view point fixing as only the start of remedial work.

This can be an effective way to restore service in the moment, but simply accruing point fixes without addressing deeper causes leads to ever greater backlogs of toil.

SRE should always push for the ultimate resolution of an issue to be a systemic fix that eliminates the deeper problem.

This requires a culture of long-term strategic thinking and also of thorough blameless investigations of root causes.

Bear in mind that the root cause of an issue is actually quite arbitrary. It depends on where we choose to stop tracing the chain of causality leading to the issue.

Human error is always a bad place to draw this line, as there is always a decision or assumption underlying that error which could be interrogated, or a systemic failure that prevented that error from being caught.

These are far better root causes which can actually be addressed in order to improve the reliability of service long term.

Consider, for example, the July 2024 crowd strike outage. Details are available in the postmortem incident review.

The report outlines actionable improvements to prevent future such outages, with a focus on systemic issues around validation of changes and the roll out of updates.

SRE Practitioner

About