v1.8 goes to production "as canary": 10% of requests via weighted Ingress. Team celebrates. Two hours later, support reports timeouts — but only for some clients. Nobody configured a dashboard comparing canary p95 and stable version. Manual rollback takes forty minutes because nobody knows which weight was applied.
A canary release is not a magic percentage. It is a measurement contract: a small share of real traffic, comparing metrics, and automated or scripted rollback in under a few minutes.
Phases: internal, canary, ramp
A mature canary deployment follows an explicit path:
- Dogfood: internal team or
X-Canary: 1header for testers. - Canary: 1–5% real traffic, business and technical metrics in parallel.
- Ramp: 25→50→100% if service level objectives hold.
- Abort: weight to 0% plus canary instance shutdown.
Each phase has minimum duration — for example thirty minutes at 5% with sufficient traffic. Otherwise canary is statistically empty and validates nothing.
5% of traffic on Sunday at 3 AM validates no regression visible on weekdays.
Metrics and guardrails
| Signal | Example threshold | Action |
|---|---|---|
| HTTP 5xx error rate | > 2× baseline | Automatic rollback |
| p95 latency | + 30% vs stable | Pause ramp |
| Checkout success | − 5% vs stable | Abort |
| Canary CPU saturation | Throttling | Investigate before ramp |
Prometheus and Alertmanager, or Argo Rollouts / Flagger tools. Define before deployment who gets paged and which thresholds trigger stop — not after first support ticket.
Routing: mesh, Ingress, feature flags
Service mesh: fine granularity, mutual TLS, observability — with higher operational cost.
Weighted nginx or Traefik Ingress: simple for stateless HTTP, often enough for first canary.
Feature flags (LaunchDarkly, open source): logical canary without infrastructure split — useful for UI, less for deep protocol change.
Start with header or cookie cohort in production-like environment, then random percentage traffic.
For metrics feeding guardrails, see Prometheus: choosing metrics useful for diagnosis.
Version and data compatibility
Canary requires two versions simultaneously in production. Incompatible API change forbids canary without version gateway or homogeneous clients.
Database migrations: "expand only" schema during canary; contract after 100% stabilization. Sessions, cache keys, serialization — verify cross-version compatibility.
The peak: fake canary, real deployment without net
Here is what the "canary" label hides when guardrails are missing.
Guardrails = compared metrics + tested rollback + explicit ramp window — not an Ingress slider forgotten in a wiki.
Decide and move forward without blind spots
Before next "small scale" deployment, lock the process:
- Choose primary rollback signal (errors, p95 latency or checkout conversion).
- Automate or script abort: weight to zero in under two minutes, tested in staging.
- Verify schema migrations stay compatible with old and new version.
- Write one-page runbook: weight commands, dashboard link, thresholds, decider.
- Repeat in game day: simulate canary degradation and measure rollback time.
- Compare mesh, managed Kubernetes and load balancer offers via the directory if scaling up.
Maturity criterion: staging canary abort in under two minutes without meeting — otherwise not ready for production.
Frequently asked questions
Canary vs blue-green?
Canary progresses by fractions; blue-green switches everything at once. Canary limits blast radius; blue-green simplifies full binary rollback.
Which metrics trigger automatic rollback?
Error rate, p95 latency, conversion or payment — compared to stable baseline, with thresholds set before deployment.
How to route canary traffic?
Weighted Ingress, service mesh or feature flags — always measured and reproducible, never "we manually set ten percent."
Should you canary the database?
Schema must stay compatible with both versions during phase. Apply expand/contract; do not deploy canary that silently breaks old version.
A mature canary ends with numbers — not "we didn't see complaints on Slack."