Observability
Observability
Section titled “Observability”Audience: backend, devops
Status: specced
Owns: devops + backend
Depends on: System architecture, Backend stack ADR, CI/CD
Bake metrics, logs, and tracing in from day one via a shared OTel bootstrap in @yourorg/common.
Instrumentation
Section titled “Instrumentation”- SDK:
@opentelemetry/sdk-node(and related instrumentations) in the shared package so every service behaves the same. - Auto-instrument Express, gRPC, Prisma, and amqplib where possible.
- Export OTLP to the observability backend — instrumentation does not care which UI consumes it.
Backend: SigNoz (preferred)
Section titled “Backend: SigNoz (preferred)”| Option | Pros | Cons |
|---|---|---|
| SigNoz (ClickHouse-backed) | Traces + metrics + logs in one UI; lower ops overhead for a small team | Less “standard” hiring familiarity than Grafana stack |
| Prometheus + Grafana + Loki + Tempo | Best-in-class pieces; common in industry | Four systems to run and tune |
Decision: start with SigNoz. The same OTLP data can later feed a Prometheus/Grafana stack without re-instrumenting.
SigNoz supports latency percentiles (p50/p90/p99), Apdex, and burn-rate alerting on error budgets out of the box.
Metrics from day one
Section titled “Metrics from day one”| Category | Examples |
|---|---|
| RED (per service/endpoint) | Rate, Errors, Duration |
| RabbitMQ | Queue depth, consumer lag, unacked, dead-letter volume |
| DB | Pool saturation, slow query count |
| gRPC | Per-method latency and error rate |
| Business | Applications submitted/day; applications stuck per stage; payment success rate; OTP delivery/verify success |
Two SLA layers (do not conflate)
Section titled “Two SLA layers (do not conflate)”1. API / latency SLOs
Section titled “1. API / latency SLOs”Example target: median response time < 200 ms, p99 < 1 s, availability 99.9% on critical endpoints (login, OTP verify, application submit).
- Tracked via OTel histograms → SigNoz dashboards + burn-rate alerts.
- Stakeholder note: 99.9% monthly ≈ 43 minutes allowed downtime per month.
2. Workflow / process SLAs
Section titled “2. Workflow / process SLAs”Example: “review decision within 3 business days,” “IV must not sit unstarted more than a week.”
- Not automatic from HTTP traces.
- Emit gauges from a scheduled job:
now() - stage_history.entered_atfor open applications, dashboarded as “overdue by stage.” - A candidate stuck weeks at “awaiting IV” is a larger operational failure than a slow endpoint — build this early.
Open product parameters for stage time limits remain in RPL policies (POL-03) until business sets numbers; the measurement mechanism above is accepted architecture.
See also
Section titled “See also”- CAP stage history → Workflow abstraction
- Runbook ownership → fill alert thresholds and on-call escalation as SigNoz is stood up