Key Rotation Runbook

This runbook covers scheduled rotation of cryptographic keys and certificates in a Mantis deployment. Regular rotation limits the exposure window if a key is compromised.

Rotation Schedule

Key Material	Recommended Interval	Trigger
Encryption master key	Annually	Schedule or compromise
TLS certificates	Annually (or before expiry)	Schedule or expiry
CA certificate	Every 3-5 years	Schedule
JWT signing key (RS256 PEM pair, or HS256 secret)	Annually	Schedule or compromise
API keys	Per organizational policy	User-initiated

Pre-Rotation Checklist

Before performing any rotation:

Verify current backups are recent and valid
Schedule a maintenance window (rotation may cause brief service interruptions)
Notify affected teams
Have rollback plan ready (old keys accessible)
Test the procedure in a non-production environment first

Encryption Master Key Rotation

The master key is used to derive data encryption keys for credentials, secrets, and sensitive variables.

Procedure

# 1. Generate new master key
NEW_KEY=$(openssl rand -base64 32)
echo "New key generated (store securely before proceeding)"

# 2. Back up the current key
echo "$MANTIS_ENCRYPTION_KEY" > /secure/old-master-key.bak
chmod 600 /secure/old-master-key.bak

# 3. Run key rotation (re-encrypts all data in the database)
mantisctl key rotate \
  --old-key "$MANTIS_ENCRYPTION_KEY" \
  --new-key "$NEW_KEY"

# 4. Verify rotation succeeded (uses the new key from MANTIS_ENCRYPTION_KEY, or pass --key)
mantisctl key verify

# 5. Update secrets manager with new key
vault kv put secret/mantis/encryption key="$NEW_KEY"
# or
aws secretsmanager update-secret --secret-id mantis-encryption-key \
  --secret-string "$NEW_KEY"

# 6. Update environment and restart services
export MANTIS_ENCRYPTION_KEY="$NEW_KEY"
systemctl restart mantis-mandible mantis-thorax

# 7. Verify services are healthy
curl -s https://localhost:3000/api/v1/health | jq

# 8. After confirming everything works, securely delete old key backup
shred -u /secure/old-master-key.bak

Rollback

If rotation fails midway:

The rotation processes records in batches. Each batch commits independently, so records rotated before an interruption remain on the new key. If interrupted mid-run, a checkpoint is saved and the process can be resumed with mantisctl key rotate --resume. Individual record failures within a batch are logged but do not abort the remaining records.
If the service fails to start with the new key, revert to the old key
Restart services with the old key and investigate the failure

TLS Certificate Rotation

Component Certificates (Mandible, Thorax, Tarsus)

# 1. Generate new key pair for the component
openssl req -new -newkey rsa:4096 -nodes \
  -keyout /etc/mantis/certs/mandible-new-key.pem \
  -out /etc/mantis/certs/mandible-new.csr \
  -subj "/CN=mandible"

# 2. Sign with the CA
openssl x509 -req \
  -in /etc/mantis/certs/mandible-new.csr \
  -CA /etc/mantis/certs/ca-cert.pem \
  -CAkey /etc/mantis/certs/ca-key.pem \
  -CAcreateserial \
  -out /etc/mantis/certs/mandible-new-cert.pem \
  -days 365

# 3. Back up old certificates
cp /etc/mantis/certs/mandible-cert.pem /etc/mantis/certs/mandible-cert.pem.bak
cp /etc/mantis/certs/mandible-key.pem /etc/mantis/certs/mandible-key.pem.bak

# 4. Replace certificates
mv /etc/mantis/certs/mandible-new-cert.pem /etc/mantis/certs/mandible-cert.pem
mv /etc/mantis/certs/mandible-new-key.pem /etc/mantis/certs/mandible-key.pem
chmod 600 /etc/mantis/certs/mandible-key.pem

# 5. Restart the component
systemctl restart mantis-mandible

# 6. Verify connectivity
curl -s https://localhost:3000/api/v1/health | jq
# /health/grpc discloses the overall status (healthy/degraded/unhealthy) to
# anyone, but the breaker internals (circuit_state, failure_count,
# success_count) are only returned to authenticated callers -- anonymous
# requests get an empty circuit_state and zeroed counts. Pass a token to see
# the breaker detail:
curl -s -H "Authorization: Bearer $TOKEN" \
  https://localhost:3000/api/v1/health/grpc | jq

Rotate All Components

When rotating all certificates (e.g., annual rotation), rotate in this order to minimize disruption:

Tarsus agents — They connect to both Mandible (register/heartbeat) and their assigned Thorax instance (command polling and result submission in poll mode; Thorax connects back to Tarsus in listen mode). Rotate Tarsus certificates before Thorax so the polling connection is re-established against a known-good Thorax.
Thorax — Restart after Tarsus to maintain execution capability
Mandible — Restart last; it coordinates everything

Each component restart briefly interrupts its connections, but the circuit breaker and reconnection logic handle transient failures.

JWT Signing Key Rotation

Mandible signs JWTs with one of two algorithms ([jwt] algorithm):

RS256 (the production default) — an asymmetric RSA key pair on disk (private_key_path / public_key_path). The Ansible deployment generates /etc/mantis/keys/jwt-private.pem and /etc/mantis/keys/jwt-public.pem.
HS256 — a single shared symmetric secret (32+ chars), set via MANDIBLE__JWT__SECRET.

Both support a graceful overlap window via the previous_* settings, so existing tokens stay valid until they expire instead of logging everyone out at once.

RS256 PEM Key-Pair Rotation (production)

# 1. Generate a new RSA private key and derive its public key
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 \
  -out /etc/mantis/keys/jwt-private-new.pem
openssl rsa -in /etc/mantis/keys/jwt-private-new.pem -pubout \
  -out /etc/mantis/keys/jwt-public-new.pem
chmod 600 /etc/mantis/keys/jwt-private-new.pem

# 2. Keep the OLD public key as `previous_public_key_path` so already-issued
#    tokens still validate during the overlap window. In mandible.toml:
#    [jwt]
#    algorithm = "RS256"
#    private_key_path = "/etc/mantis/keys/jwt-private.pem"
#    public_key_path  = "/etc/mantis/keys/jwt-public.pem"
#    previous_public_key_path = "/etc/mantis/keys/jwt-public-old.pem"
#    previous_key_id = "<old-key-id>"

# 3. Promote the new keys (back up the current pair first)
cp /etc/mantis/keys/jwt-public.pem  /etc/mantis/keys/jwt-public-old.pem
cp /etc/mantis/keys/jwt-private.pem /etc/mantis/keys/jwt-private.pem.bak
mv /etc/mantis/keys/jwt-private-new.pem /etc/mantis/keys/jwt-private.pem
mv /etc/mantis/keys/jwt-public-new.pem  /etc/mantis/keys/jwt-public.pem

# 4. Restart Mandible; new tokens are signed with the new key, old tokens still
#    validate against previous_public_key_path until they expire
systemctl restart mantis-mandible

# 5. After the longest token lifetime has elapsed, remove the previous_* settings
#    and delete jwt-public-old.pem / jwt-private.pem.bak

HS256 Secret Rotation

# 1. Generate new JWT secret
NEW_JWT_SECRET=$(openssl rand -base64 64)

# 2. Move the current secret to `previous_secret` for a graceful overlap, then
#    set the new one. In mandible.toml ([jwt] previous_secret = "<old>"), or:
export MANDIBLE__JWT__PREVIOUS_SECRET="$MANDIBLE__JWT__SECRET"
export MANDIBLE__JWT__SECRET="$NEW_JWT_SECRET"

# 3. Restart Mandible
systemctl restart mantis-mandible

# 4. After the longest token lifetime has elapsed, drop previous_secret

CA Certificate Rotation

CA rotation is the most complex procedure because all component certificates must be re-signed.

Procedure

# 1. Generate new CA key pair
openssl req -x509 -newkey rsa:4096 -nodes \
  -keyout /etc/mantis/certs/ca-new-key.pem \
  -out /etc/mantis/certs/ca-new-cert.pem \
  -days 1825 \
  -subj "/CN=Mantis CA"

# 2. Re-sign all component certificates with the new CA.
#    The deployed component certs are mandible, tarsus, and the Thorax-side
#    server/client/dispatch pairs (each <name>-cert.pem + <name>-key.pem).
for component in mandible tarsus server client dispatch; do
  openssl req -new -newkey rsa:4096 -nodes \
    -keyout "/etc/mantis/certs/${component}-new-key.pem" \
    -out "/etc/mantis/certs/${component}-new.csr" \
    -subj "/CN=${component}"

  openssl x509 -req \
    -in "/etc/mantis/certs/${component}-new.csr" \
    -CA /etc/mantis/certs/ca-new-cert.pem \
    -CAkey /etc/mantis/certs/ca-new-key.pem \
    -CAcreateserial \
    -out "/etc/mantis/certs/${component}-new-cert.pem" \
    -days 365
done

# 3. Back up old CA and component certificates
mkdir -p /etc/mantis/certs/old
cp /etc/mantis/certs/ca-*.pem /etc/mantis/certs/old/

# 4. Replace CA certificate on all hosts
cp /etc/mantis/certs/ca-new-cert.pem /etc/mantis/certs/ca-cert.pem
cp /etc/mantis/certs/ca-new-key.pem /etc/mantis/certs/ca-key.pem

# 5. Replace component certificates on each host
for component in mandible tarsus server client dispatch; do
  cp "/etc/mantis/certs/${component}-new-cert.pem" "/etc/mantis/certs/${component}-cert.pem"
  cp "/etc/mantis/certs/${component}-new-key.pem" "/etc/mantis/certs/${component}-key.pem"
  chmod 600 "/etc/mantis/certs/${component}-key.pem"
done

# 6. Restart all services (brief outage expected)
systemctl restart mantis-tarsus mantis-thorax mantis-mandible

# 7. Verify (pass a token to see breaker internals; anonymous callers
#    receive only the overall status with an empty circuit_state)
curl -s -H "Authorization: Bearer $TOKEN" \
  https://localhost:3000/api/v1/health/grpc | jq

Post-Rotation Verification

After any rotation, verify the system is fully operational:

# Health checks
curl -s https://localhost:3000/api/v1/health | jq
curl -s https://localhost:3000/api/v1/health/ready
# /health/grpc returns the overall status to anyone; authenticate to also
# receive the breaker internals (circuit_state, failure_count, success_count).
# Without a token those fields come back empty/zeroed.
curl -s -H "Authorization: Bearer $TOKEN" \
  https://localhost:3000/api/v1/health/grpc | jq

# Verify encrypted data is accessible
mantisctl key verify

# Test a deployment to confirm end-to-end functionality
# (in a non-production environment)

Next Steps

Certificate Backup — Back up after rotation
Database Backup — Back up after encryption key rotation
Disaster Recovery — Recovery procedures
Encryption at Rest — Encryption architecture