Upgrades

Upgrades

Upgrade Mantis components safely with zero-downtime strategies.

Upgrade Planning

Version Compatibility

Before upgrading, verify compatibility:

Component	Compatibility
Mandible ↔ Thorax	Must match major.minor version
Thorax ↔ Tarsus	Backward compatible within major version
Database schema	Managed by migrations

Pre-Upgrade Checklist

• Review release notes for breaking changes
• Backup database
• Backup configuration files
• Test upgrade in staging environment
• Schedule maintenance window (if needed)
• Notify stakeholders
• Verify rollback plan

Upgrade Order

Always upgrade in this order:

Docker Compose Upgrades

Standard Upgrade

# Pull latest images
docker compose pull

# Stop services
docker compose down

# Run migrations using diesel CLI
cd instinct
diesel migration run

# Start with new images
docker compose up -d

# Verify health
docker compose ps
curl https://localhost:3000/api/v1/health

Rolling Upgrade (Zero Downtime)

For high availability setups with multiple replicas:

# Pull new images
docker compose pull

# Scale up with new version
docker compose up -d --scale mandible=4 --scale thorax=4

# Wait for new containers to be healthy
sleep 30

# Scale down to remove old containers
docker compose up -d --scale mandible=2 --scale thorax=2

Rollback

# Stop current version
docker compose down

# Specify previous version
export MANTIS_VERSION=1.2.3
docker compose up -d

Kubernetes Upgrades

Standard Rolling Update

# Update image tag in deployment
kubectl set image deployment/mandible mandible=ghcr.io/mantisplatform/mantis-mandible:v1.3.0 -n mantis
kubectl set image deployment/thorax thorax=ghcr.io/mantisplatform/mantis-thorax:v1.3.0 -n mantis
kubectl set image deployment/lens lens=ghcr.io/mantisplatform/mantis-lens:v1.3.0 -n mantis

# Monitor rollout
kubectl rollout status deployment/mandible -n mantis
kubectl rollout status deployment/thorax -n mantis

Blue-Green Deployment

deployments/mandible-v2.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mandible-v2
  namespace: mantis
spec:
  replicas: 2
  selector:
    matchLabels:
      app: mantis
      component: mandible
      version: v2
  template:
    metadata:
      labels:
        app: mantis
        component: mandible
        version: v2
    spec:
      containers:
        - name: mandible
          image: ghcr.io/mantisplatform/mantis-mandible:v1.3.0
          # ... rest of spec

Switch traffic:

# Deploy new version
kubectl apply -f deployments/mandible-v2.yaml

# Wait for ready
kubectl rollout status deployment/mandible-v2 -n mantis

# Update service selector to new version
kubectl patch service mandible -n mantis -p '{"spec":{"selector":{"version":"v2"}}}'

# Verify traffic flows to new version
kubectl get endpoints mandible -n mantis

# Remove old deployment after validation
kubectl delete deployment mandible-v1 -n mantis

Canary Deployment

# Deploy canary with subset of traffic
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mandible-canary
  namespace: mantis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mantis
      component: mandible
      track: canary
  template:
    metadata:
      labels:
        app: mantis
        component: mandible
        track: canary
    spec:
      containers:
        - name: mandible
          image: ghcr.io/mantisplatform/mantis-mandible:v1.3.0

Rollback

# Rollback to previous revision
kubectl rollout undo deployment/mandible -n mantis

# Or to specific revision
kubectl rollout undo deployment/mandible --to-revision=2 -n mantis

# Check rollback status
kubectl rollout status deployment/mandible -n mantis

Database Migrations

Running Migrations

Migrations must run before deploying new application code:

# Using diesel CLI (recommended)
cd instinct
diesel migration run

# Kubernetes - use a job with diesel CLI
kubectl create job --from=cronjob/mantis-migration mantis-migration-v1.3.0 -n mantis
kubectl wait --for=condition=complete job/mantis-migration-v1.3.0 -n mantis

Migration Best Practices

1. Always backup before migrations

pg_dump -h localhost -U mantis mantis > backup-$(date +%Y%m%d).sql

2. Run migrations in a transaction

Mantis migrations run in transactions by default. If a migration fails, changes are rolled back.

3. Test migrations on a copy first

# Create test database
createdb mantis_upgrade_test
pg_restore -d mantis_upgrade_test backup.sql

# Run migration against test database
cd instinct
DATABASE_URL="postgres://localhost/mantis_upgrade_test" diesel migration run

Handling Migration Failures

If a migration fails:

1. Check the error message in logs
2. Do not retry without investigating
3. Fix the underlying issue
4. Migration state is tracked in __diesel_schema_migrations table

-- Check migration status
SELECT * FROM __diesel_schema_migrations ORDER BY run_on DESC;

-- If needed, manually mark migration as rolled back
-- WARNING: Only do this if you understand the consequences
DELETE FROM __diesel_schema_migrations WHERE version = 'YYYYMMDDHHMMSS';

Tarsus Agent Upgrades

Upgrade Strategies

Strategy	Description	Downtime
Rolling	Upgrade agents one at a time	None
Batch	Upgrade groups by environment	Minimal
Full	Upgrade all agents simultaneously	Brief

Rolling Upgrade Script

upgrade-agents.sh

#!/bin/bash
# List target IDs via the REST API (there is no `mantisctl targets` subcommand)
AGENTS=$(curl -s -H "Authorization: Bearer $TOKEN" \
    "https://api.mantis.example.com/api/v1/targets" | jq -r '.data[].id')
NEW_VERSION="1.3.0"

for agent in $AGENTS; do
    echo "Upgrading $agent..."

    # Deploy upgrade package with mantisctl
    mantisctl deployment create \
        --solution-id "$TARSUS_UPGRADE_SOLUTION_ID" \
        --target "$agent" \
        --var "version=$NEW_VERSION" \
        --follow

    echo "Completed $agent"
done

Manual Agent Upgrade

On each target server:

# Stop agent
sudo systemctl stop tarsus

# Backup current binary
sudo cp /usr/local/bin/tarsus /usr/local/bin/tarsus.backup

# Download new version
curl -L https://releases.mantis.example.com/tarsus/v1.3.0/tarsus-linux-amd64 \
    -o /usr/local/bin/tarsus

# Set permissions
sudo chmod 755 /usr/local/bin/tarsus

# Start agent
sudo systemctl start tarsus

# Verify
tarsus --version

Configuration Changes

Adding New Settings

New configuration options typically have sensible defaults. For example, Thorax exposes a [session] section and Mandible exposes a [client_session] section:

# Thorax session tuning
[session]
heartbeat_timeout_secs = 90

# Mandible client-session tuning
[client_session]
max_concurrent_commands = 10

Deprecated Settings

Check the release notes for any renamed or removed settings before upgrading. The current database pool keys are max_connections, min_idle, and connection_timeout_secs:

[database]
max_connections = 10
min_idle = 2
connection_timeout_secs = 10

Environment Variable Changes

Update environment files:

# Check for new/changed variables
diff .env.example .env

# Update environment
nano .env

Version-Specific Guides

Breaking Changes Log

Version	Breaking Change	Migration Path
v1.0.0	Initial release	N/A

Monitoring Upgrades

Health Checks During Upgrade

# Continuous health monitoring
watch -n 2 'curl -s https://localhost:3000/api/v1/health | jq'

# Check specific component
curl https://localhost:3000/api/v1/health/ready

Key Metrics to Watch

Metric	Normal	Alert
Request latency	< 100ms	> 500ms
Error rate	< 0.1%	> 1%
Active connections	Stable	Spike/drop
Memory usage	Stable	Growing

Log Analysis

# Watch for errors during upgrade
docker compose logs -f --since 5m | grep -i error

# Kubernetes
kubectl logs -l app=mantis -n mantis -f --since=5m | grep -i error

Rollback Procedures

Docker Compose Rollback

# Immediate rollback
docker compose down
export MANTIS_VERSION=previous-version
docker compose up -d

# Database rollback (if needed)
psql -U mantis mantis < backup.sql

Kubernetes Rollback

# Rollback deployments
kubectl rollout undo deployment/mandible -n mantis
kubectl rollout undo deployment/thorax -n mantis
kubectl rollout undo deployment/lens -n mantis

# Database rollback (if needed)
kubectl exec -it postgres-0 -n mantis -- psql -U mantis -f /backup/backup.sql

Database Rollback

Caution

Database rollbacks may cause data loss. Only perform if absolutely necessary.

# Restore from backup
pg_restore -h localhost -U mantis -d mantis --clean backup.dump

# Or from SQL dump
psql -h localhost -U mantis -d mantis < backup.sql

Maintenance Windows

Scheduling

For upgrades requiring downtime:

1. Announce maintenance - Notify users 24-48 hours in advance
2. Freeze deployments - Prevent new deployments during upgrade
3. Backup everything - Database, configs, certificates
4. Perform upgrade - Follow upgrade procedures
5. Validate - Run smoke tests
6. Resume operations - Unfreeze and notify users

Creating Deployment Freeze

# In Lens UI: Freezes → New Freeze (top-level nav, path /freezes/new)
# Or via API:
curl -X POST https://api.mantis.example.com/api/v1/deployment-freezes \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Upgrade Maintenance",
    "start_time": "2024-01-15T02:00:00Z",
    "end_time": "2024-01-15T04:00:00Z"
  }'
# Omitting tenant_id/solution_id/environment_id applies the freeze globally.

Troubleshooting

Upgrade Failed

1. Check logs for specific errors
2. Verify all prerequisites met
3. Ensure database migrations completed
4. Check connectivity between components

Version Mismatch

# Check tarsus version (supports --version via clap)
tarsus --version

# Check mandible and thorax versions via the server info endpoint
curl -s https://localhost:3000/api/v1/server/info | jq '.server_version'

# Verify the API is healthy after the upgrade
curl https://localhost:3000/api/v1/health | jq '.status'

Database Migration Stuck

# Check migration status
psql -U mantis -c "SELECT * FROM __diesel_schema_migrations"

# Check for locks
psql -U mantis -c "SELECT * FROM pg_locks WHERE NOT granted"

Next Steps

• Production Checklist - Verify production readiness
• Backup & Recovery - Backup procedures
• Troubleshooting - Common issues