Logging

Logging

Mantis uses structured logging via the Rust tracing ecosystem. All components (Mandible, Thorax, Tarsus) share the same logging configuration pattern, controlled through the RUST_LOG environment variable.

Log Levels

Level	Purpose	Example
error	Unrecoverable failures requiring attention	Database connection lost, TLS handshake failed
warn	Degraded conditions that may need investigation	Health check failed, circuit breaker opened
info	Normal operational events	Server started, deployment dispatched
debug	Detailed operational data for troubleshooting	Request details, query timings
trace	Very verbose internal tracing	Individual function calls, state transitions

Configuring Log Levels

RUST_LOG Environment Variable

The RUST_LOG environment variable controls which log messages are emitted. It supports per-crate and per-module filtering.

# Default: debug for Mantis, debug for HTTP layer
export RUST_LOG="mandible=debug,tower_http=debug"

# Production: info for Mantis, warn for everything else
export RUST_LOG="mandible=info,thorax=info,tarsus=info,warn"

# Troubleshooting: debug for specific modules
export RUST_LOG="mandible=info,mandible::routes::deployments=debug,mandible::queue=debug"

# Verbose: trace for a specific area
export RUST_LOG="mandible=info,mandible::services::dispatch_queue_processor=trace"

Filter Syntax

The RUST_LOG filter supports several patterns:

# Set a global default level
RUST_LOG=info

# Per-crate filtering
RUST_LOG="mandible=debug,instinct=info"

# Per-module filtering (use :: separator)
RUST_LOG="mandible::routes::auth=debug"

# Multiple filters (comma-separated)
RUST_LOG="mandible=info,mandible::queue=debug,tower_http=warn"

# Span-based filtering
RUST_LOG="mandible[deployment_dispatch]=trace"

Component Defaults

When RUST_LOG is not set, each component uses its built-in default:

Component	Default Filter
Mandible	mandible=debug,tower_http=debug
Thorax	info (from log_level / MANTIS_LOG_LEVEL; override via RUST_LOG)
Tarsus	info (from log_level; override via RUST_LOG)

Log Output Format

Mandible and Tarsus output structured logs to stdout. Thorax outputs structured logs to stderr. Each log line includes:

2026-01-15T10:30:45.123456Z  INFO mandible::routes::health: Health check passed
2026-01-15T10:30:45.234567Z DEBUG mandible::queue::dispatch: Processing dispatch queue batch_size=10
2026-01-15T10:30:45.345678Z  WARN mandible::services::instance_cleanup: Thorax instance stale instance_id="abc123" last_heartbeat="60s ago"

Field	Description
Timestamp	ISO 8601 with microsecond precision (UTC)
Level	Log level (ERROR, WARN, INFO, DEBUG, TRACE)
Target	Module path that emitted the log
Message	Human-readable description
Fields	Structured key-value data (appended after message)

Docker and Container Logging

When running in Docker, logs go to stdout (Mandible, Tarsus) or stderr (Thorax) and are captured by the Docker logging driver.

View Logs

# Follow logs for a specific service
docker compose logs -f mandible

# Show last 100 lines
docker compose logs --tail 100 mandible

# Filter by time
docker compose logs --since "2026-01-15T10:00:00" mandible

Docker Compose Log Configuration

services:
  mandible:
    environment:
      RUST_LOG: 'mandible=info,tower_http=warn'
    logging:
      driver: json-file
      options:
        max-size: '50m'
        max-file: '5'

Systemd Journal Integration

When running as a systemd service, logs integrate with journalctl:

# Follow Mandible logs
journalctl -u mandible -f

# Show logs since last boot
journalctl -u mandible -b

# Filter by priority (error and above)
journalctl -u mandible -p err

# Export for analysis
journalctl -u mandible --since "1 hour ago" -o json > mandible-logs.json

Log Aggregation

Shipping to External Systems

Since Mantis components log to stdout or stderr in a structured format, you can use standard log shippers:

Tool	Configuration
Fluentd/Fluent Bit	Tail stdout or Docker log files
Vector	Docker source or journald source
Promtail	Scrape container logs for Loki
Filebeat	Docker or journald input

Promtail Example (for Grafana Loki)

scrape_configs:
  - job_name: mantis
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(mandible|thorax|tarsus)'
        action: keep
      - source_labels: ['__meta_docker_container_name']
        target_label: component

Sensitive Data

Mantis is designed to avoid logging sensitive data. The following are never logged:

• Passwords and secrets
• JWT token values (only token metadata like expiry)
• API key values (only key IDs)
• Encryption keys
• Database credentials

If you see sensitive data in logs, report it as a security issue.

Recommended Production Configuration

# Production: concise but informative
export RUST_LOG="mandible=info,thorax=info,tarsus=info,tower_http=warn,instinct=info"

For troubleshooting a specific area, temporarily increase verbosity for that module:

# Debug deployment dispatching
export RUST_LOG="mandible=info,mandible::services::dispatch_queue_processor=debug,mandible::queue=debug"

# Debug authentication issues
export RUST_LOG="mandible=info,mandible::routes::auth=debug,mandible::middleware::auth=debug"

# Debug gRPC connectivity
export RUST_LOG="mandible=info,mandible::grpc=debug,tonic=debug"

Restart the service after changing RUST_LOG for the new filter to take effect.

Next Steps

• Prometheus Metrics — Quantitative monitoring
• tokio-console — Async runtime debugging
• Common Issues — Troubleshoot using logs