Health Status

Understand and monitor the health status of your targets.

Health States

Targets can be in one of three health states:

State	Icon	Description
Online	●	Target is connected and responding
Stale	◐	Target missed recent heartbeat
Offline	○	Target is not connected

Health State Diagram

Online State

A target is Online when:

Tarsus agent is running
The agent’s connection to Mandible is active
Heartbeats are received by Mandible within threshold

┌─────────────────────────────────────────────────────────────┐
│  Target: web-prod-01                                        │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Status:     ● Online                                       │
│  Last Seen:  Just now                                       │
│  Mode:       Listen                                         │
│  Uptime:     5 days, 14 hours                               │
│                                                             │
│  ✓ Ready to receive deployments                             │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Online Indicators

Indicator	Value
Last seen	Within heartbeat interval
Connection	Active
Deployable	Yes

Stale State

A target becomes Stale when:

Heartbeat not received within threshold (default: 90 seconds)
Connection may be intermittent
Target might be under heavy load

┌─────────────────────────────────────────────────────────────┐
│  Target: web-prod-03                                        │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Status:     ◐ Stale                                        │
│  Last Seen:  5 minutes ago                                  │
│  Mode:       Listen                                         │
│                                                             │
│  ⚠ Target may be experiencing issues                        │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Stale Indicators

Indicator	Value
Last seen	> heartbeat threshold
Connection	Unknown/Intermittent
Deployable	Yes (with warning)

Common Stale Causes

Cause	Resolution
Network congestion	Wait for recovery
High CPU load	Investigate target performance
Tarsus overloaded	Check Tarsus logs
Firewall issues	Verify connectivity

Offline State

A target is Offline when:

No heartbeat for extended period (default: 5 minutes)
Connection is confirmed lost
Tarsus agent may not be running

┌─────────────────────────────────────────────────────────────┐
│  Target: worker-01                                          │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Status:     ○ Offline                                      │
│  Last Seen:  2 hours ago                                    │
│  Mode:       Listen                                         │
│                                                             │
│  ✗ Target is not available for deployments                  │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Offline Indicators

Indicator	Value
Last seen	> offline threshold
Connection	Disconnected
Deployable	No

Common Offline Causes

Cause	Resolution
Server powered off	Power on the server
Tarsus not running	Start Tarsus service
Network unreachable	Fix network connectivity
Certificate expired	Renew certificates

Health Monitoring

Dashboard Overview

┌─────────────────────────────────────────────────────────────┐
│  Target Health Summary                                      │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         │
│  │   Online    │  │    Stale    │  │   Offline   │         │
│  │     10      │  │      1      │  │      1      │         │
│  │    (83%)    │  │    (8%)     │  │    (8%)     │         │
│  └─────────────┘  └─────────────┘  └─────────────┘         │
│                                                             │
│  Total Targets: 12                                          │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Health Alerts

Per-target health-alert configuration (notification rules that fire when a specific target changes state) is not yet available (planned — not yet available). Today you monitor health state from the dashboard and target detail views.

Health History

View historical health data:

┌─────────────────────────────────────────────────────────────┐
│  Health History: web-prod-03                                │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌───────────────┬──────────┬──────────┬────────────────┐  │
│  │ Time          │ From     │ To       │ Duration       │  │
│  ├───────────────┼──────────┼──────────┼────────────────┤  │
│  │ Today 14:30   │ Stale    │ Online   │ -              │  │
│  │ Today 14:25   │ Online   │ Stale    │ 5 min          │  │
│  │ Yesterday     │ Offline  │ Online   │ 2 hours        │  │
│  │ 3 days ago    │ Online   │ Offline  │ -              │  │
│  └───────────────┴──────────┴──────────┴────────────────┘  │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Heartbeat Configuration

In both Listen and Poll mode, the Tarsus agent sends heartbeats directly to Mandible, which tracks target liveness and updates the health status. The connection mode only affects how commands are delivered for execution, not how health is tracked.

The stale and offline thresholds below are configured server-side in Thorax’s [session] block (thorax.toml); Tarsus itself does not expose stale/offline threshold settings.

Listen Mode Heartbeat

In Listen mode, Tarsus sends heartbeats to Mandible while a Thorax instance connects inbound to deliver commands:

Setting	Default	Description
Heartbeat interval	30 seconds	How often Tarsus reports to Mandible
Stale threshold	90 seconds	When to mark stale (`heartbeat_timeout_secs`)
Offline threshold	5 minutes	When to mark offline (`stale_timeout_secs`)

Poll Mode Heartbeat

In Poll mode, Tarsus also heartbeats directly to Mandible, separately from polling its assigned Thorax for commands:

Setting	Default	Description
Heartbeat interval	30 seconds	How often Tarsus reports to Mandible
Stale threshold	90 seconds	When to mark stale (`heartbeat_timeout_secs`)
Offline threshold	5 minutes	When to mark offline (`stale_timeout_secs`)

Health Checks

Built-in Health Checks

Mantis performs automatic health checks:

Check	Description	Frequency
Connectivity	TCP connection test	Per heartbeat
TLS	Certificate validation	Per connection
Response	Response time	Per heartbeat

Custom Health Checks

Define custom health checks in actions:

#!/bin/bash
# Check application is running
if ! pgrep -x "myapp" > /dev/null; then
    echo "Application not running"
    exit 1
fi

# Check port is listening
if ! nc -z localhost 8080; then
    echo "Port 8080 not listening"
    exit 1
fi

# Check disk space
DISK_USAGE=$(df -h / | awk 'NR==2 {print $5}' | tr -d '%')
if [ "$DISK_USAGE" -gt 90 ]; then
    echo "Disk usage critical: ${DISK_USAGE}%"
    exit 1
fi

echo "All health checks passed"
exit 0

Health-Based Deployment

Deployment Behavior by State

Target State	Deployment Action
Online	Proceeds normally
Stale	Warning shown, user decides
Offline	Skipped or blocked

Configuring Deployment Health Requirements

┌─────────────────────────────────────────────────────────────┐
│  Deployment Health Settings                                 │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Include targets with status:                               │
│  [✓] Online                                                 │
│  [ ] Stale - deployment may fail                            │
│  [ ] Offline - will definitely fail                         │
│                                                             │
│  [ ] Wait for stale targets to recover (timeout: 5 min)     │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Troubleshooting Health Issues

Target Stuck in Stale

Symptoms:

Target shows Stale for extended periods
Never transitions to Offline or Online

Diagnosis:

Check network latency
Verify Tarsus logs
Check system resources

Resolution:

Restart Tarsus if needed
Fix network issues
Increase heartbeat thresholds if appropriate

Frequent Online/Stale Transitions

Symptoms:

Target flaps between Online and Stale
Health history shows many transitions

Diagnosis:

Check network stability
Look for resource contention
Review system logs

Resolution:

Fix underlying network issues
Increase threshold values
Investigate root cause

Target Won’t Come Online

Symptoms:

Target remains Offline
Tarsus appears to be running

Diagnosis:

Verify Tarsus is listening (Listen mode)
Check firewall rules
Validate certificates
Test network connectivity

Resolution:

# Check Tarsus status
systemctl status tarsus

# Test connectivity (Listen mode)
nc -zv target-hostname 9342

# Check certificates
openssl s_client -connect target-hostname:9342

# View Tarsus logs
journalctl -u tarsus -f

Best Practices

1. Monitor Health Proactively

Review dashboard regularly
Set up alerts for offline targets
Investigate stale targets promptly

2. Right-Size Thresholds

Environment	Stale Threshold	Offline Threshold
Low-latency	60 seconds	3 minutes
Standard	90 seconds	5 minutes
High-latency	5 minutes	15 minutes

3. Document Health Issues

Track and document recurring health issues:

Target	Issue	Frequency	Resolution
web-03	Network flapping	Weekly	ISP issue
db-01	High load stale	Monthly	Scale resources

4. Plan for Offline Targets

Have procedures for offline targets
Don’t deploy to offline targets in production
Investigate offline targets before deploying

Next Steps

Managing Targets - Target management
Listen vs Poll Mode - Connection modes
Tagging - Organize targets
Overview - Return to targets overview