Failure Pattern Recognition

Failure Pattern Recognition

Mantis automatically groups recurring deployment failures into patterns — clusters of failures that share the same normalized error signature. Instead of triaging the same broken deployment a dozen times across targets and days, operators see one ranked pattern with its full blast radius (how many deployments, targets, and environments it has hit) and, optionally, an AI-generated cause and remediation.

Detection is on by default. It runs as a background service inside Mandible and requires no external dependencies for the deterministic grouping; the optional LLM labeling reuses your existing AI Analysis provider configuration.

How it works

1. Fingerprinting. Each failed command result’s error text is normalized (variable tokens such as numbers, UUIDs, paths, and timestamps are masked) and hashed, scoped by an exit-code class. Failures with the same fingerprint belong to the same pattern.
2. Clustering batch. A periodic background pass attributes new un-clustered failures to patterns, updating idempotent counts: distinct deployments, targets, and total occurrences. The “across deployments” count is the primary recurrence signal.
3. Regression. A resolved pattern automatically reopens to active when a new occurrence lands, so a failure you thought was fixed resurfaces instead of silently re-accumulating.
4. Alerts. When a pattern crosses the configured thresholds within the alert window, an edge-triggered pattern_emerging notification fires once (not on every occurrence).
5. LLM labeling (optional). If enabled, the configured AI provider produces a human-readable title, likely cause, and remediation for each cluster. Input is strictly bounded so a pathological log can never blow the provider’s context window.

Enabling and configuring

All knobs live under AI Analysis → Settings in Lens (or via PUT /api/v1/admin/analysis/settings). They require the analysis:manage permission.

Setting	Default	Meaning
pattern_detection_enabled	true	Master toggle for clustering.
pattern_window_days	30	Trailing window of failures considered for clustering.
pattern_batch_interval_secs	300	Seconds between background batch passes.
pattern_retention_days	90	Days a resolved/untouched pattern and its occurrences are kept.
pattern_llm_labeling_enabled	true	Label clusters with the AI provider. Only takes effect when AI Analysis is enabled (off by default), so it incurs no cost until you configure a provider.
pattern_alert_enabled	false	Whether emerging-pattern alerts fire. Off by default — enable it once your thresholds are tuned.
pattern_alert_min_deployments	2	Min distinct deployments before a pattern can alert.
pattern_alert_min_occurrences	5	Min total occurrences before a pattern can alert.
pattern_alert_min_targets	1	Min distinct targets before a pattern can alert.
pattern_alert_window_days	7	Recurrence window the alert thresholds are evaluated over.
pattern_label_max_samples	3	Max error samples sent to the LLM per cluster label.
pattern_label_max_input_chars	4000	Max characters of LLM-label input (context + samples).
pattern_relabel_factor	2	Growth factor (×) before a labeled pattern is relabeled.
pattern_max_batch_size	500	Max failures clustered per batch pass.

Invariants

• Keep pattern_retention_days ≥ pattern_window_days. Retention shorter than the window would prune patterns that are still inside the detection window.
• Keep pattern_alert_window_days ≤ pattern_window_days. The alert window is evaluated within the detection window; a larger alert window has no additional data to consider.

Cost note

Deterministic grouping is cheap and always safe to leave on. LLM labeling makes a provider call per new or significantly-grown cluster and is therefore the only part with a per-use cost. It is on by default but only runs when AI Analysis is enabled (which is off by default), so labeling incurs no cost until you configure an LLM provider. The pattern_relabel_factor controls how much a pattern must grow before it is re-labeled (higher = fewer calls), and pattern_label_max_* bound the per-call input size.

Operating

• Recompute on demand. The Patterns view has a Recompute now button; it is tenant-scoped to the caller and runs on the background service (it does not block the request). The recompute queue is bounded — if it is saturated the request is rejected with 429.
• Mute / resolve. Muting a pattern hides it from the default list and stops it alerting; it never auto-reopens. Resolving closes it but allows regression on a new occurrence.

Monitoring

• The metric pattern_overload_dropped_total increments when deployment-linked failures age out of the accelerator window before they could be clustered. A rising value means the batch is not keeping up — lower pattern_batch_interval_secs or raise pattern_max_batch_size.

Permissions

Permission	Grants
patterns:read	View patterns, cluster detail, and dashboard stats.
patterns:manage	Recompute, mute, resolve, and reactivate patterns.

Seeded roles: admin and tenant_admin hold both; operator holds patterns:read only. Tenant users see only their own tenant’s patterns; global/system patterns are visible to global admins only.