Common Issues

Common Issues

This page covers frequently encountered issues when operating Mantis, along with diagnosis steps and solutions.

Service Will Not Start

Mandible Fails to Start

Symptom: Mandible exits immediately after launch.

Check configuration:

# Verify config file syntax
cat /etc/mantis/mandible.toml

# Check for required environment variables
echo $MANDIBLE__DATABASE__URL
echo $MANDIBLE__JWT__SECRET

Common causes:

Error Message	Cause	Fix
address already in use	Port 3000 already bound	Stop conflicting process or change port
database connection failed	PostgreSQL unreachable	Verify database URL and connectivity
invalid JWT secret	Secret too short or missing	Set MANDIBLE__JWT__SECRET (min 32 chars)
TLS configuration error	Certificate files missing	Check cert_path and key_path exist

Database Connection Failures

Symptom: /api/v1/health returns 503, logs show connection refused.

# Test database connectivity directly
psql "$DATABASE_URL" -c "SELECT 1"

# Check if PostgreSQL is running
systemctl status postgresql
# or
docker compose ps postgres

# Verify connection pool isn't exhausted
# Look for "pool exhausted" or "timeout" in Mandible logs
journalctl -u mantis-mandible --since "5 minutes ago" | grep -i pool

Authentication Problems

Cannot Log In

Symptom: Login returns 401 or the UI shows “Invalid credentials”.

1. Verify the user exists and is not disabled
2. Check for auth rate limiting (progressive backoff after failed attempts)
3. Confirm RUST_LOG includes mandible::routes::auth=debug and review logs

# Check if rate limited (look for 429 responses)
journalctl -u mantis-mandible | grep "rate_limit" | tail -5

# Rate limits reset after the lockout period expires

Token Refresh Failures

Symptom: Users are logged out unexpectedly, or API calls return 401.

• Check that the mantis_refresh_token cookie domain matches the deployment domain
• Verify JWT secret has not changed (rotating the secret invalidates all tokens)
• Ensure system clocks are synchronized (JWT expiry is time-sensitive)

Deployment Issues

Deployments Stuck in “Pending”

Symptom: Deployments remain in Pending status and never start.

1. Check if any Thorax instance is online (the overall status is shown to any caller; authenticate to also see circuit_state, failure_count, and success_count, which are withheld from anonymous requests):

curl -s -H "Authorization: Bearer $TOKEN" \
  https://mantis.example.com/api/v1/health/grpc | jq

2. Verify the dispatch queue processor is running (check Mandible logs for dispatch_queue entries)
3. Check if the target has an assigned Thorax instance
4. Review RabbitMQ connectivity if using queue-based dispatch

Deployments Fail Immediately

Symptom: Deployments move to Failed within seconds.

• Check the deployment logs in the Lens UI for error details
• Verify the target (Tarsus agent) is online and reachable
• Confirm the action’s command or script payload is valid
• Check that required variables are defined for the target’s environment

Connectivity Issues

Thorax Shows “Offline”

Symptom: Instance cleanup service marks Thorax as offline.

The InstanceCleanupService checks heartbeats every 30 seconds and marks instances offline after 60 seconds of silence.

1. Verify Thorax is running: systemctl status mantis-thorax
2. Check network connectivity between Mandible and Thorax
3. Review Thorax logs for gRPC connection errors
4. Verify mTLS certificates are valid and not expired

Tarsus Cannot Register

Symptom: Tarsus agent fails to connect to Mandible.

See Connection Problems for detailed network troubleshooting, and Certificate Issues for TLS-specific problems.

Performance Problems

Slow API Responses

Symptom: API calls take several seconds to respond.

# Check database query performance
# Enable debug logging for database operations
export RUST_LOG="mandible=info,diesel=debug"
systemctl restart mantis-mandible

Common causes:

• Missing database indexes (check after schema migrations)
• Connection pool exhaustion under high load
• Large result sets without pagination

High Memory Usage

Symptom: Mandible or Thorax memory usage grows over time.

• Check notification queue depth (mantis_notification_queue_depth metric)
• Review rate limit cache size (mantis_rate_limit_cache_entries metric)
• Verify the RateLimiterCleanupService is running (cleans stale cache entries)

Data Issues

Missing Audit Logs

Symptom: Expected audit log entries are not appearing.

The mantis_audit_log_events_total metric records channel_full or channel_closed results when the audit channel is under pressure or has stopped. This family is recorded in Mandible’s in-process registry but is not currently exposed on an HTTP /metrics endpoint, so monitor it via Mandible’s structured logs (audit-channel warnings) rather than a metrics scrape.

Stale Data in UI

Symptom: Lens shows outdated information.

• Hard refresh the browser (Ctrl+Shift+R)
• Check if the API returns current data directly: curl https://mantis.example.com/api/v1/targets
• Verify SSE (Server-Sent Events) connection is active for real-time updates

Getting Help

If these steps do not resolve your issue:

1. Collect relevant logs with debug-level logging enabled
2. Note the exact error messages and HTTP status codes
3. Check the Mantis version (/api/v1/server/info)
4. Review related pages for specific subsystem issues

Next Steps

• Connection Problems — Network and connectivity issues
• Deployment Failures — Deployment-specific troubleshooting
• Queue Issues — RabbitMQ and dispatch queue problems
• Certificate Issues — TLS and mTLS problems