# Observability Runbook

## Scope

Jarvis observability covers:
- persistent telemetry snapshots in `OBSERVABILITY_DB_PATH`
- runtime event stream in `OBSERVABILITY_EVENT_LOG_PATH`
- restart and uptime state in `OBSERVABILITY_STATE_PATH`
- Prometheus/OpenMetrics export at `GET /metrics` (operator server)
- SSE event feed at `GET /events` (operator server)

## Key Signals

- Latency percentiles: STT / LLM-first-sentence / TTS-first-audio (`p50`, `p95`, `p99`)
- Tool rolling rates: per-tool `success_rate` / `error_rate` windows
- Intent success signals: `answer_quality_success_rate`, `completion_success_rate`, `correction_frequency`
- Unified scorecard: `jarvis_scorecard` and `system_status.scorecard` (latency, reliability, initiative, trust)
- Failure burst alerts: generated when recent tool errors exceed `OBSERVABILITY_FAILURE_BURST_THRESHOLD`
- Budget alerts:
  - `latency_budget_exceeded` when LLM first-sentence p95 breaches `OBSERVABILITY_LATENCY_BUDGET_P95_MS`
  - `tokens_budget_exceeded` when estimated hourly tokens breach `OBSERVABILITY_TOKENS_BUDGET_PER_HOUR`
  - `cost_budget_exceeded` when estimated hourly spend breaches `OBSERVABILITY_COST_BUDGET_USD_PER_HOUR`
- Router canary disagreement analytics: `observability.router_canary_analytics` for canary coverage and shadow disagreement rate
- Process lifecycle: `runtime_start`, `runtime_stop`, `restart_count`, `uptime_sec`

## Triage Flow

1. Check current health snapshot:
   - query `system_status`
   - inspect `observability.alerts`
2. Inspect metrics endpoint:
   - `curl http://127.0.0.1:8765/metrics`
   - verify spikes in p95/p99 latency lines
3. Inspect event stream and timeline:
   - `curl http://127.0.0.1:8765/events`
   - filter for `watchdog_reset`, `stt_fallback`, `tts_fallback_text_only`, `failure_burst`, budget alert types
4. Correlate with tool history:
   - query `tool_summary` and `tool_summary_text`
5. Inspect canary disagreement panel:
   - open Operator Console and inspect `Router Canary`
   - review `recent_disagreements` to identify mismatched primary vs shadow route decisions

## Tuning

- Increase snapshot density:
  - lower `OBSERVABILITY_SNAPSHOT_INTERVAL_SEC`
- Reduce noisy alerts:
  - raise `OBSERVABILITY_FAILURE_BURST_THRESHOLD`
  - increase `OBSERVABILITY_ALERT_COOLDOWN_SEC`
- Tune budget sensitivity:
  - adjust `OBSERVABILITY_LATENCY_BUDGET_P95_MS`
  - adjust `OBSERVABILITY_TOKENS_BUDGET_PER_HOUR` and `OBSERVABILITY_COST_BUDGET_USD_PER_HOUR`
  - adjust `OBSERVABILITY_BUDGET_WINDOW_SEC` for shorter/longer smoothing windows
- Tighten degraded-mode detection:
  - combine lower burst threshold with watchdog timeouts (`WATCHDOG_*`)

## SLO Suggestions

- STT p95 < 1200ms
- LLM first sentence p95 < 1800ms
- TTS first audio p95 < 700ms
- Tool error rate < 5% for critical integrations

## Backup / Retention

- Observability DB and event log are local files; include them in host backup policy.
- If disk pressure increases, rotate/trim event log and archive DB snapshots out of band.

## Related Runbooks

- `campaign-execution-runbook.md`
- `integrations-degradation-runbook.md`
- `autonomy-checkpoint-runbook.md`