Site Reliability Engineering (SRE) 101 with DevOps vs SRE

Guest post originally published on the MSys Technologies blog by Sunny Raskar

Consider the scenario below

An Independent Software Provider (ISV) developed a financial application for a global investment firm that serves global conglomerates, leading central banks, asset managers, broking firms, and governmental bodies. The development strategy for the application encompassed a thought through DevOps plan with cutting-edge agile tools. This has ensured zero downtime deployment at maximum productivity. The app now handles financial transactions in real-time at an enormous scale, while safeguarding sensitive customer data and facilitating uninterrupted workflow. One unfortunate day, the application crashed, and this investment firm suffered a severe backlash (monetarily and morally) from its customers.

Here is the backstory – application’s workflow exchange had crossed its transactional threshold limit, and lack of responsive remedial action crippled the infrastructure. The intelligent automation brought forth by DevOps was confined mainly to the development and deployment environment. The IT operations, thus, remained susceptible to challenges.

Decoupling DevOps and RunOps – The Genesis of Site Reliability Engineering (SRE)

A decade or two ago, companies operated with a legacy IT mindset. IT operations consisted mostly of administrative jobs without automation. This was the time when the code writing, application testing, and deploying was done manually. Somewhere around 2008-2010, automation started getting prominence. Now Dev and Ops worked concurrently towards continuous integration and continuous deployment – backed by the agile software movement. The production team was mainly in charge of the runtime environment. However, they lacked skillsets to manage IT operations, which resulted in application instability, as depicted in the scenario above.

Thus, DevOps and RunOps were decoupled, paving the way for SRE – a preventive technique to infuse stability in the IT operations.

“Site Reliability Engineering is the practice and a cultural shift towards creating a robust IT operation process that would instill stability, high performance, and scalability to the production environment.”

Software-First Approach: Brain Stem of SRE

“SRE is what happens when you ask a software engineer to design an operations team,” Benjamin Treynor Sloss, Google. This means an SRE function is run by IT operational specialists who code. These specialist engineers implement a software-first approach to automate IT operations and preempt failures. They apply cutting edge software practices to integrated Dev and Ops on a single platform, and execute test codes across the continuous environment. Therefore, they carry advanced software skills, including DNS Configuration, remediating server, network, and infrastructure problems, and fixing application glitches.

The software approach codifies every aspect of IT operations to build resiliency within infrastructure and applications. Thus, changes are managed via version control tools and checked for issues leveraging test frameworks, while following the principle of observability.

The Principle of Error Budget

SRE engineers verify the code quality of changes in the application by asking the development team to produce evidence via automated test results. SRE managers can fix Service Level Objectives (SLOs) to gauge the performance of changes in the application. They should set a threshold for permissible minimum application downtime, also known as Error Budget. If a downtime during any changes in the application is within the scale of the Error Budget, then SRE teams can approve it. If not, then the changes should be rolled back for improvements to fall within the Error Budget formula.

Error Budgets tend to bring balance between SRE and application development by mitigating risks. An Error Budget is unaffected until the system availability will fall within the SLO. The Error Budget can always be adjusted by managing the SLOs or enhancing the IT operability. The ultimate goal remains application reliability and scalability.

Calculating Error Budget

A simple formula to calculate Error Budget is (System Availability Percentage) minus (SLO Benchmark Percentage). Please refer to the System Availability Calculator below.

Illustration.

Suppose the system availability is 95%. And, your SLO threshold is 80%.

Error Budget: 95%-80%= 15%

Error Budget/month: 108 hours. (At 5% downtime, per day downtime hours is 1.2 hours. Therefore for 15% it is 1.2*3 = 3.6. So for 30 days it will be 30*3.6 = 108 hours)

Error Budget/quarter: 108*3 = 324 hours.

Quick Trivia – Breaking monolithic applications lets us derive SLOs at a granular level.

Cultural Shift: A Right Step towards Reliability and Scalability

Popular SRE engagement models such as Kitchen Sink, a.k.a. “Everything SRE” – a dedicated SRE team, Infrastructure – a backend managed services or Embedded – tagging SRE engineer with developer/s, require additional hiring. These models tend to build dedicated teams that lead to a ‘Silo’ SRE environment. The problem with the Silo environment is that it promotes a hands-off approach, which results in a lack of standardization and co-ordination between teams. So, a sensible approach is shelving off a project-oriented mindset and allowing SRE to grow organically within the whole organization. It starts by apprising the teams of customer principles and instilling a data-driven method for ensuring application reliability and scalability.

Organizations must identify a change agent who would create and promote a culture of maximum system availability. He / She can champion this change by practicing the principle of observability, where monitoring is a subset. Observability essentially requires engineering teams to be vigilant of common and complex problems hindering the attendance of reliability and scalability in the application. See the principles of observability below.

The principle of observability follows a cyclical approach, which ensures maximum application uptime.

Step Zero – Unlocking Potential of Pyramid of Observability

Step zero is making employees aware of end-to-end product detail – technical and functional. Until an operational specialist knows what to observe, the subsequent steps in the pyramid of observability remain futile.

Also, remember that this culture shift isn’t achievable overnight – it will be successful when practiced sincerely over a few months.

DevOps vs. SRE

People often confuse SRE with DevOps. DevOps and SRE are complementary practices to drive quality in the software development process and maintain application stability.

Let’s analyze four key the fundamental difference between DevOps and SRE.