Deployment Failures

Deployment Failures

This guide covers troubleshooting deployments that fail, get stuck, or produce unexpected results. Understanding the deployment lifecycle helps narrow down where failures occur.

Deployment Lifecycle

Pending --> Queued --> Running --> Success/Failed

Stage	Owner	Common Failure Causes
Pending	Mandible (dispatch queue)	No Thorax online, queue processor stalled
Queued	Mandible (dispatch queue)	No eligible Thorax instance, queue entry expired
Running	Tarsus (execution)	Script error, timeout, resource issue
Failed	Any	See specific causes below

Stuck in Pending

Symptom: Deployment stays in Pending status and never progresses.

Check Thorax Availability

# Verify gRPC connectivity. Authenticate to see the breaker internals
# (circuit_state, failure_count); anonymous callers receive only the overall
# status with an empty circuit_state.
curl -s -H "Authorization: Bearer $TOKEN" \
  https://mantis.example.com/api/v1/health/grpc | jq

# Check if any Thorax instances are online
curl -s https://mantis.example.com/api/v1/thorax-instances | jq '.data[] | {id, status, last_heartbeat_at}'

If no Thorax instances are online, the dispatch queue processor has nowhere to send the deployment.

Check Dispatch Queue Processor

The DispatchQueueProcessor polls for pending dispatch requests and forwards them to Thorax. It retries up to 10 times and expires requests after 24 hours.

# Check for dispatch queue activity in logs
journalctl -u mantis-mandible | grep -i "dispatch" | tail -20

# Check queue status via API (if available)
# There is no dispatch-queue REST endpoint; observe queue activity in the logs:
journalctl -u mandible | grep -i dispatch

Check RabbitMQ (If Using Queue Dispatch)

If the deployment uses RabbitMQ-based dispatching, verify queue health:

# Check queue health
journalctl -u mantis-mandible | grep -i "queue_health\|dispatch_mode" | tail -10

If RabbitMQ is unavailable, the system should automatically fall back to polling-based dispatch. Check that the health monitor detects the outage.

Queued But Not Running

Symptom: Deployment shows Queued but Tarsus never executes it.

1. Verify the target Tarsus agent is online and has a recent heartbeat
2. Check that Thorax can reach the Tarsus agent (gRPC connectivity)
3. Review Thorax logs for command delivery errors

# Check target status
curl -s https://mantis.example.com/api/v1/targets | jq '.data[] | select(.name == "my-target") | {status, last_seen_at}'

If the Thorax instance that received the dispatch goes offline, the ThoraxReassignmentService should reassign orphaned targets to another available Thorax instance.

Script Execution Failures

Symptom: Deployment reaches Running but then fails with a script error.

Check Deployment Logs

View execution output in the Lens UI under the deployment’s detail page, or retrieve the full log history via the API:

curl -s -H "Authorization: Bearer $TOKEN" \
  https://mantis.example.com/api/v1/deployments/{id}/logs/history | jq

Note: The /api/v1/deployments/{id}/logs endpoint streams real-time output as Server-Sent Events (text/event-stream) and is not suitable for jq. Use /logs/history for completed deployments.

Common Script Errors

Error	Cause	Fix
command not found	Binary not in PATH on target	Use absolute paths or verify PATH
permission denied	Tarsus user lacks permissions	Check file permissions and user context
exit code 1 (or non-zero)	Script logic error	Review script output in deployment logs
timeout	Script exceeded time limit	Increase timeout or optimize the script
variable not found	Required variable not set	Define the variable in the appropriate scope

Variable Resolution Issues

Deployments can fail if required variables are not defined for the target’s environment:

1. Check that all referenced variables exist in the variable set
2. Verify the variable scope matches the deployment context (global, environment, or target)
3. Look for typos in variable references

Timeout Failures

Symptom: Deployment fails with a Timeout result status.

Timeouts occur when a deployment step takes longer than the configured limit.

• Review whether the timeout is appropriate for the operation
• Check for network latency between Thorax and Tarsus
• Look for long-running processes that block script completion
• Consider breaking long operations into multiple steps

Rollback Failures

Symptom: A rollback deployment fails.

Rollback deployments re-execute a previous successful deployment’s configuration. If the rollback target environment has changed since the original deployment, the rollback may fail due to:

• Changed infrastructure (removed servers, changed ports)
• Missing dependencies that were present during the original deployment
• Modified file permissions or ownership

Deployment Cancellation

Symptom: A deployment cannot be cancelled or stays in a transitional state.

• Pending deployments can be cancelled via the API or Lens UI
• Running deployments require the Tarsus agent to acknowledge the cancellation
• If the Tarsus agent is unreachable, the deployment may remain in a stale state until the agent reconnects or the session times out

Debugging Checklist

1. Check deployment status and result in the Lens UI or API
2. Review deployment logs for specific error messages
3. Verify all components are online (Mandible, Thorax, Tarsus)
4. Confirm gRPC health (/api/v1/health/grpc)
5. Check that required variables are defined
6. Verify target agent is reachable and registered
7. Review component logs with debug-level logging enabled
8. Check for queue issues if using RabbitMQ dispatch

Next Steps

• Queue Issues — RabbitMQ and dispatch queue problems
• Connection Problems — Network connectivity
• Common Issues — General troubleshooting