Friday 2 p.m.: monitoring is green. CPU at 35%, memory comfortable, disk far from full. Yet support receives screenshots — checkout takes eight seconds, sometimes more. The team opens Grafana, scrolls through twelve default dashboards — node_exporter, nginx, a few app counters — and finds nothing that rises when complaints spike.
This is not a Prometheus failure. It is a signal selection failure. You measure machine health, not the path an order follows when it slows down.
What Prometheus does well — and what it will not decide for you
Prometheus stores time series and queries them with PromQL. It excels at questions like “did /api/checkout p95 double since the 11 a.m. deploy?” or “has the PostgreSQL connection pool been saturated since yesterday?”
It does not guess which metrics matter to you. Before installing another exporter, ask: if the site is slow tomorrow, which curve must rise so the team knows where to dig? Everything else is noise that reassures.
A metric that cannot answer “slower than yesterday, on which path?” only decorates a dashboard.
RED, USE and business metrics: three frames, one order
Three frameworks come up often. They stack rather than exclude each other.
| Frame | Question | Useful examples |
|---|---|---|
| RED | Does the service respond fast enough without errors? | http_requests_total, 5xx ratio, duration histogram by route |
| USE | Is the underlying resource saturated? | CPU waiting on I/O, disk, DB connections idle = 0 |
| Business | Does the user reach their goal? | checkout_started vs checkout_completed, abandoned cart |
For an API or dynamic site, start with RED on critical routes — not the entire catalogue. One latency histogram on /api/cart and /api/payment beats a hundred counters on rarely used admin endpoints.
USE becomes essential when RED shows slowness without HTTP errors: the server returns 200 in three seconds because the database struggles, not because PHP is slow.
Histograms: calibrate buckets to your SLO, not defaults
A common mistake: copy generic buckets (0.005, 0.01, 0.025…) while your internal SLO says “95% of checkout requests < 500 ms”. Wrong buckets lie about p95: the computed quantile falls between buckets and smooths real degradation.
Example PromQL for p95 by route:
histogram_quantile(
0.95,
sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route)
)
Adapt buckets to your reality: if most requests stay under 200 ms but checkout can reach 2 s, your buckets must cover that range — not only the sub-second zone.
Alert on what breaks experience, not arbitrary thresholds: checkout p95 > 2 s for ten minutes, DB pool with no idle connections, Redis queue lag above a business threshold. A “disk at 70%” alert with no user correlation feeds alert fatigue without protecting checkout.
Labels and cardinality: the silent trap
Prometheus is not a log warehouse. Each label value multiplies time series. http_requests_total{user_id="8842"} or {url="/product/red-chair-42"} can turn a healthy install into an unmanageable TSDB in days.
Pragmatic rules:
- Route template yes (
/api/cart/{id}), full URL no. - HTTP code, method, environment yes; client id no.
- Need per-user or per-order detail? Traces or structured logs, not Prometheus labels.
On a modest VPS, uncontrolled cardinality costs RAM and slow PromQL — exactly when you need it most.
What to instrument first by hosting type
On shared hosting or small VPS, do not replicate a hyperscaler stack:
- Application — duration on critical routes, errors, DB pool saturation if you manage it.
- Nginx or reverse proxy — upstream latency, active connections.
- node_exporter — basic USE (CPU, RAM, disk) to rule out obvious saturation.
OpenTelemetry or a native Prometheus client (Go, Node, PHP with suitable library) at the app core beats a collection of exotic exporters. Fifteen days local retention is often enough; beyond that, Thanos or Mimir only make sense with volume and a dedicated ops team.
Compare offers where you can install your own agents via our directory — strict shared hosting sometimes limits on-host Prometheus; VPS or managed cloud gives more room.
When metrics are not enough
Prometheus aggregates. One slow request among a thousand fast ones can stay invisible in p95. Sporadic slowness, occasional Redis locks, the forgotten SQL query on a rare code path: you need a trace (Tempo, Jaeger) or correlated logs with a common request_id — covered in Centralized logs.
The three pillars complement each other:
- Metrics — trend, alert, SLO.
- Logs — textual context, stack trace, parameters.
- Traces — path of one request across services.
Prometheus alone is not full observability. It is the right tool for aggregated signals — if you chose the right signals.
The peak: the illusion of full dashboards
Here is what “Grafana in ten minutes” tutorials skip.
Choosing metrics means betting on plausible bottlenecks in your architecture — and removing the rest as noise.
Decide and move forward without blind spots
In half a day you can put Prometheus back in service of diagnosis:
- List two critical paths (checkout, login, partner API) — not “the whole site”.
- Apply RED on those routes: counter, errors, histogram with SLO-aligned buckets.
- Add one USE metric tied to your main suspicion (DB pool, Redis, job queue).
- Review labels — remove any high-cardinality dimension.
- Replace a CPU alert with p95 or pool saturation; test under simulated load.
- Document one PromQL query on-call should run first in an incident.
To compare hosts where you keep install control (VPS, cloud, partial PaaS), use the comparison tool and guides. If slowness stays opaque after clean metrics, move to Distributed tracing rather than adding a fifteenth dashboard.
Concrete exercise: simulate degradation (reduced DB pool, artificial latency) and verify one PromQL query identifies the faulty layer in under five minutes. If not, your metrics still tell the wrong story.
Frequently asked questions
Minimum metrics for an API?
RED: rate, errors, duration histogram. Add connection pool or cache saturation if slowness correlates there — not just server CPU.
Why avoid too many labels?
Each label multiplies time series. Route template and HTTP code yes; user_id and full URL no. Individual detail belongs in traces or logs.
Counter, gauge or histogram — which to choose?
Counter with rate() for volumes; gauge for instant state; histogram for latency with SLO-calibrated buckets.
Is Prometheus enough to diagnose slowness?
For aggregated trends and alerts, yes. For sporadic outliers, add correlated logs and traces — Prometheus does not replace line-by-line investigation.
The next time a dashboard is entirely green while users slow down, ask: which metric is missing to tell their story? That is where useful Prometheus begins.