Investigation

SRE incident
D
Checkout pod is crash looping
Service: checkoutEnv: prod
IncidentsPredictionsIMP

Checkout pod is crash looping

resolved · critical · checkout · prod

RCA

checkout-6f9c container exits with code 1 within 30s of start. Likely missing config value.

No approval requests yet.

Resolution history

Resolution #1 · 2026-09-01T20:46:01Z · "fixed"

By local-user · source manual_resolve

Chat

Ask Detective about the incident, evidence, approvals, or next diagnostics. Write actions are not executed.

No messages yet. Ask a question to start the incident chat.

Application Context

Deployment-supplied mapping for owners, repositories, pipelines, runtime labels, and infrastructure topology.

not mapped

No application context matched this incident. Add an app context mapping file and set APP_CONTEXT_PATH to enable repo, pipeline, and infrastructure correlation.

Infrastructure Topology

Where to investigate for this application — CDN, load balancers, Kubernetes, data stores, and custom hints from app-context.json.

not mapped

Add an infrastructure block to the matched application in your context file (for example CloudFront, ALB, Kubernetes, Redis, Kafka) so Detective knows where to collect evidence.

Ownership And Escalation

Team routing and escalation hints from resolved application context.

not mapped

No ownership metadata matched this incident. Add owner and escalation fields to the application context mapping.

Runbook Suggestions

Matching local runbooks and Confluence knowledge collected during investigation.

0 suggestion(s)

No runbook evidence yet. Mount `RUNBOOKS_PATH` or configure Confluence to collect runbook guidance during investigation.

Recent Changes

Commits, merged PRs/MRs, and CI/CD runs collected from application context near the alert window.

0 item(s)

No SCM change evidence yet. Enable application context, set GITHUB_TOKEN or GITLAB_TOKEN, and rerun investigation when the incident maps to a repository.

Similar Past Incidents

Vector-ranked historical RCA memory plus related incidents by fingerprint or prior occurrence.

4 match(es)
paymentprodsimilarity 27%

Postmortem: Payment pod is not running

Open incident

fixed

ordersprodsimilarity 10%

Orders API latency p95 above 2s

Open incident

1. Telemetry is absent for the orders service in the incident window. i. Prometheus returns zero series for up{service=~"orders|orders.*"}, so service availability cannot be confirmed [E1]. ii. Loki returns zero log streams for orders/* labels, so error, exception, crash, and panic signals cannot be evaluated [E2]. iii. No matching Prometheus alerting rules or active alerts exist for this incident context [E3]. 2. The incident title reports orders API p95 latency above 2s, but no metric, log, or rule evidence is available to verify which component, deployment, or dependency is the source. 3. Given the complete absence of observability data, the immediate actionable finding is an evidence gap or connector/configuration issue: Prometheus/Loki query selectors, namespaces, or endpoints may be misconfigured, unavailable, or returning no matching streams. Prior postmortems for checkout and payment are low-similarity and do not inform this incident.

Previous action plan
[
  {
    "url": null,
    "title": "Prior verified fix: Fix for Orders API latency p95 above 2s",
    "source": "scm_suggested_fixes",
    "repository": null,
    "risk_level": "low",
    "description": "This service has a stored fix from org memory that may apply again:\ntest and fixed",
    "permissions": [
      "read"
    ],
    "target_system": "orders",
    "rollback_notes": "Review prior fix context before repeating the same remediation.",
    "suggestion_kind": "org_memory_hint",
    "requires_approval": true
  },
  {
    "title": "Review Prometheus connectivity and service scrape configuration for orders",
    "executor": "manual_review_only",
    "risk_level": "low",
    "target_system": "connector: prometheus",
    "rollback_notes": "No rollback required for read-only review.",
    "requires_approval": false,
    "required_permissions": []
  },
  {
    "title": "Review Loki connectivity and log label selectors for orders",
    "executor": "manual_review_only",
    "risk_level": "low",
    "target_system": "connector: loki",
    "rollback_notes": "No rollback required for read-only review.",
    "requires_approval": false,
    "required_permissions": []
  },
  {
    "title": "Verify alert routing and service ownership metadata for the incident",
    "executor": "manual_review_only",
    "risk_level": "low",
    "target_system": "evidence",
    "rollback_notes": "No rollback required for read-only review.",
    "requires_approval": false,
    "required_permissions": []
  }
]
ordersprodsimilarity 6%

Orders API latency p95 above 2s

Open incident

1. Root cause cannot be determined from available telemetry because the orders service has no observable metrics or logs in Detective. i. Prometheus returned no series for service=~"orders|orders.*" [E1], indicating a likely evidence gap or connector/configuration issue rather than confirmed service behavior. ii. Loki returned no log streams matching error patterns for the orders app [E2], and no active Prometheus alerts or rules matched the incident context [E3]. iii. Prior postmortems for checkout and payment have low similarity, so they provide no applicable context for this orders latency incident. 2. The high-severity alert cannot be validated or investigated without obtaining authoritative monitoring data from the source systems.

Previous action plan
[
  {
    "url": null,
    "title": "Prior verified fix: Fix for Orders API latency p95 above 2s",
    "source": "scm_suggested_fixes",
    "repository": null,
    "risk_level": "low",
    "description": "This service has a stored fix from org memory that may apply again:\ntest and fixed",
    "permissions": [
      "read"
    ],
    "target_system": "orders",
    "rollback_notes": "Review prior fix context before repeating the same remediation.",
    "suggestion_kind": "org_memory_hint",
    "requires_approval": true
  },
  {
    "title": "Review Prometheus connector status and query scope for orders service",
    "executor": "manual_review_only",
    "risk_level": "low",
    "target_system": "Prometheus / connector configuration",
    "rollback_notes": "No rollback required for read-only review.",
    "requires_approval": false,
    "required_permissions": []
  },
  {
    "title": "Review Loki connector status and log label selectors for orders namespace",
    "executor": "manual_review_only",
    "risk_level": "low",
    "target_system": "Loki / connector configuration",
    "rollback_notes": "No rollback required for read-only review.",
    "requires_approval": false,
    "required_permissions": []
  },
  {
    "title": "Inspect orders deployment and pod status via read-only API checks",
    "executor": "manual_review_only",
    "risk_level": "low",
    "target_system": "Kubernetes / orders service",
    "rollback_notes": "No rollback required for read-only review.",
    "requires_approval": false,
    "required_permissions": []
  },
  {
    "title": "Review alert source configuration to confirm the latency threshold and data source used",
    "executor": "manual_review_only",
    "risk_level": "low",
    "target_system": "Alerting configuration",
    "rollback_notes": "No rollback required for read-only review.",
    "requires_approval": false,
    "required_permissions": []
  }
]
ordersprodsimilarity 0%

Postmortem: Orders API latency p95 above 2s

Open incident

test and fixed

Connector Policy For This Incident

Select connectors from alert labels, application context, and infrastructure topology.

auto

Selected: loki, prometheus

Investigation Coverage

Connector selection, evidence collection, and gaps for the latest investigation.

0 connector(s)

Coverage will appear after the worker starts an investigation.

Connector Timeline

No connector runs recorded yet.

Evidence

No evidence collected yet.

Alerts