Site Reliability Engineering (SRE) & Observability

• The 4 Golden Signals: Latency, Traffic, Errors, and Saturation
• Instrumenting distributed microservices with OpenTelemetry SDKs and Collectors
• Correlating distributed trace spans with application logs and Prometheus metrics

• Calculating practical SLIs, defining realistic SLOs, and tracking Error Budget burn rates
• Designing multi-window multi-burn-rate alert rules in Prometheus and Datadog
• Eliminating alert fatigue and establishing high-fidelity on-call rotation schedules

• Executing controlled chaos experiments (Pod kill, CPU stress, Network delay) with Chaos Mesh
• Canary deployments with automated progressive rollouts and instant rollback (Argo Rollouts)
• Facilitating blameless post-mortems and developing self-healing runbook automations

Q1: What is an Error Budget and how does an SRE use it to balance velocity and reliability?

An Error Budget is the allowable room for failure calculated from an SLO (e.g., a 99.9% SLO allows a 0.1% error budget). If the error budget is healthy, developers can ship features aggressively. If an incident or regressions burn through the error budget, feature releases are temporarily halted, and engineering capacity shifts exclusively to reliability and technical debt reduction.

Q2: Explain the 4 Golden Signals of monitoring defined by Google SRE.

1. Latency: Time taken to service a request (differentiating success vs error latency). 2. Traffic: Demand on system (requests/sec or network IO). 3. Errors: Rate of failed requests (HTTP 500s or protocol exceptions). 4. Saturation: How full the service is (CPU, memory, database connection pool limits).