lep_qq

Most SLO tools treat a user journey as binary: either all services are up, or the whole thing is down. That breaks when traffic doesn't flow uniformly through all your services.

The checkout SLO that lied Three services: checkout-base (99.9%), payments (99.95%), coupon (99.5%). A naive AND composition gives a system SLO of ~99.35%.

But 90% of users never hit the coupon service. Only 10% go through base → coupon → payments. The coupon service drags the number down, but it only affects a tenth of my traffic.

The correct formula is:

e_total = 1 - ( 0.9 × (1 - e_base) × (1 - e_payments) + 0.1 × (1 - e_base) × (1 - e_coupon) × (1 - e_payments) ) Each route is a chain where all services must succeed (multiply success rates). Weights represent traffic share and must sum to 1.

Translating this into PromQL PromQL has no native "product of a set" operator. What it has is scalar(), which collapses a single-element vector into a scalar — exactly what you need when each slok:sli_error_rate recording rule returns one value.

The generated rule for a 5m window:

1 - ( 0.9 * ( (1 - scalar(slok:sli_error_rate:5m{slo_name="checkout-base-slo",...})) * (1 - scalar(slok:sli_error_rate:5m{slo_name="payments-slo",...})) ) + 0.1 * ( (1 - scalar(slok:sli_error_rate:5m{slo_name="checkout-base-slo",...})) * (1 - scalar(slok:sli_error_rate:5m{slo_name="coupon-slo",...})) * (1 - scalar(slok:sli_error_rate:5m{slo_name="payments-slo",...})) ) ) scalar() is load-bearing. Without it you'd be multiplying labeled vectors with different label sets — PromQL would try to join them and fail.

The rule is generated for each evaluation window (5m, 1h, 6h, 3d, 7d, 30d) and stored as slok:sli_error_composition_rate:WINDOW. Everything downstream — burn rate, alerts, status — consumes this single metric without knowing how it was produced.

The YAML interface

kind: SLOComposition spec: target: 99.9 window: 30d objectives: - name: base ref: { name: checkout-base-slo } - name: payments ref: { name: payments-slo } - name: coupon ref: { name: coupon-slo } composition: type: WEIGHTED_ROUTES params: routes: - name: no-coupon weight: 0.9 chain: [base, payments] - name: with-coupon weight: 0.1 chain: [base, coupon, payments] With the composed error rate as a recording rule, the standard multi-window burn rate pipeline works unchanged. The alert fires when the composed journey is burning budget too fast — not when any single service degrades, but when the degradation actually impacts users at the rate the weights predict.

Limitations scalar() assumes each input recording rule returns exactly one series. If a query matches multiple series, scalar() returns NaN and the composition breaks silently. Also: duplicate alias detection in routes isn't enforced yet by the webhook.

This is alpha. Feedback on the API shape welcome.

Repo: https://github.com/federicolepera/slok

I've been building a Kubernetes operator for SLO management (SLOK) and just shipped a feature I haven't seen elsewhere: WEIGHTED_ROUTES composition.

The problem

Most SLO tools treat a user journey as a simple chain: if any service fails, the whole thing fails. But real traffic doesn't work that way. In a checkout flow, 90% of users might skip the coupon service entirely. If the coupon service has 99.5% availability, does that really pull your checkout SLO down to 99.5%? No — because most of your users never touch it.

The model

WEIGHTED_ROUTES lets you describe which percentage of traffic flows through which service chain. Each chain is an implicit AND (all services in the chain must succeed). The composed error rate is:

e_total = 1 - Σ( weight_i × Π(1 - e_j) ) For the checkout example (90% skip coupon, 10% use it):

e_total = 1 - ( 0.9 × (1 - e_base) × (1 - e_payments) + 0.1 × (1 - e_base) × (1 - e_coupon) × (1 - e_payments) ) SLOK translates this formula directly into Prometheus recording rules wired into the standard multi-window burn rate pipeline.

The YAML

kind: SLOComposition spec: target: 99.9 window: 30d objectives: - name: base ref: { name: checkout-base-slo } - name: payments ref: { name: payments-slo } - name: coupon ref: { name: coupon-slo } composition: type: WEIGHTED_ROUTES params: routes: - name: no-coupon weight: 0.9 chain: [base, payments] - name: with-coupon weight: 0.1 chain: [base, coupon, payments] alerting: burnRateAlerts: enabled: true The operator generates the PrometheusRules automatically. You get burn rate alerts on the composed SLO, not just on individual services.

Other things SLOK does

AND_MIN composition (worst-case across services) Built-in SLI templates for http-availability, http-latency, kubernetes-apiserver Automatic error budget tracking exposed in .status Event correlation: when a burn rate spike is detected, SLOK creates an SLOCorrelation resource listing recent Deployments, ConfigMap changes, and cluster events that may have caused it — with an optional LLM-enhanced summary (Llama 3.3 70B via Groq) WEIGHTED_ROUTES is alpha. Feedback on the API shape is welcome.

Repo: https://github.com/federicolepera/slok