Redis Availability Runbook

Redis Availability Runbook

Mandible depends on Redis for two things, with different blast radii. Use this runbook when MandibleJwtBlocklistRedisDown or MandibleSseRedisDown fires (see JWT configuration → Redis availability dependency).

Blast radius

Dependency	Metric	Posture	Impact if Redis is down
JWT revocation (blocklist)	mantis_jwt_blocklist_redis_available	Fail-closed	All JWT-authenticated REST returns 401 (Lens/UI down). API-key auth is unaffected.
SSE / server-side sessions	mantis_sse_redis_up	Degrade	SSE streaming + session enforcement degraded.

/health/ready is intentionally DB-only and stays 200 during a Redis outage — do not rely on readiness to detect this; rely on the metrics/alerts above.

When MandibleJwtBlocklistRedisDown fires (critical)

1. Confirm the scope. Check mantis_jwt_blocklist_redis_available (0 = down) and rate(mantis_jwt_blocklist_redis_errors_total[5m]). If 0/elevated, JWT REST is rejecting traffic right now.
2. Check Redis itself.

   redis-cli -h <redis-host> -p 6379 ping        # expect PONG
   systemctl status redis                          # bare-metal
   docker compose ps redis && docker compose logs --tail=100 redis  # compose

3. Restore Redis (restart the service / replace the instance / fix the network path). The gauge (mantis_jwt_blocklist_redis_available) returns to 1 on the next successful JWT-authenticated request — recovery is immediate once traffic resumes. To confirm Redis is back independently of request traffic, check mantis_sse_redis_up, which is updated by a background probe every ~10 s regardless of request load.
4. If Redis cannot be restored quickly, make a conscious tradeoff:
   • Communicate the auth outage to users.
   • Optionally reduce the JWT access-token TTL going forward so that, after recovery, the window in which a revoked token could have been honored is small. (Mantis does not fail-open; tokens are simply rejected while Redis is down.)
5. Post-incident: consider Redis HA (replica + automatic failover) so a single-instance blip cannot reject all JWT traffic.

When MandibleSseRedisDown fires (warning)

SSE streaming and server-side session enforcement are degraded, but API-key and (Redis-up) JWT REST are unaffected. Restore Redis as above; no auth outage.

Why readiness is not gated on Redis

Gating /health/ready on a per-request Redis PING was tried and reverted: the probe is unauthenticated and rate-limit-exempt, so probe-flooding would exhaust the shared Redis pool (DoS amplification), and gating per-pod readiness on a shared Redis would deschedule the whole fleet on a brief blip (cascading outage). The non-gating mantis_sse_redis_up gauge gives operators the same visibility without those failure modes.