what is happening
Payment webhook retries exhausted
resolved · high · payment · prod
RCA
Stripe webhook deliveries exhausted retries for a subset of tenants.
No approval requests yet.
Resolution history
Resolution #1 · 2026-08-30T20:19:29Z · "Replayed failed webhooks after fixing the expired signing secret."
By Ananya Rao · source slack
Chat
Ask Detective about the incident, evidence, approvals, or next diagnostics. Write actions are not executed.
1. From existing incident context, the incident is resolved: Stripe webhook deliveries exhausted their retry budget for a subset of tenants in the prod payment service, with high severity. 2. No evidence references are attached, so specific affected tenants, failure counts, and root cause are not confirmed. i. Missing evidence includes Stripe delivery logs, payment webhook handler error logs, and affected tenant identifiers. ii. Reviewing those stored sources would clarify scope, but they are not present in the current evidence set.
Application Context
Deployment-supplied mapping for owners, repositories, pipelines, runtime labels, and infrastructure topology.
No application context matched this incident. Add an app context mapping file and set APP_CONTEXT_PATH to enable repo, pipeline, and infrastructure correlation.
Infrastructure Topology
Where to investigate for this application — CDN, load balancers, Kubernetes, data stores, and custom hints from app-context.json.
Add an infrastructure block to the matched application in your context file (for example CloudFront, ALB, Kubernetes, Redis, Kafka) so Detective knows where to collect evidence.
Ownership And Escalation
Team routing and escalation hints from resolved application context.
No ownership metadata matched this incident. Add owner and escalation fields to the application context mapping.
Runbook Suggestions
Matching local runbooks and Confluence knowledge collected during investigation.
No runbook evidence yet. Mount `RUNBOOKS_PATH` or configure Confluence to collect runbook guidance during investigation.
Recent Changes
Commits, merged PRs/MRs, and CI/CD runs collected from application context near the alert window.
No SCM change evidence yet. Enable application context, set GITHUB_TOKEN or GITLAB_TOKEN, and rerun investigation when the incident maps to a repository.
Similar Past Incidents
Vector-ranked historical RCA memory plus related incidents by fingerprint or prior occurrence.
paymentprodsimilarity 47%
Open incidentPostmortem: Payment pod is not running
fixed
checkoutprodsimilarity 9%
Open incidentPostmortem: Checkout pod is crash looping
fixed
ordersprodsimilarity 5%
Open incidentOrders API latency p95 above 2s
1. Root cause cannot be determined from available telemetry because the orders service has no observable metrics or logs in Detective. i. Prometheus returned no series for service=~"orders|orders.*" [E1], indicating a likely evidence gap or connector/configuration issue rather than confirmed service behavior. ii. Loki returned no log streams matching error patterns for the orders app [E2], and no active Prometheus alerts or rules matched the incident context [E3]. iii. Prior postmortems for checkout and payment have low similarity, so they provide no applicable context for this orders latency incident. 2. The high-severity alert cannot be validated or investigated without obtaining authoritative monitoring data from the source systems.
Previous action plan
[
{
"url": null,
"title": "Prior verified fix: Fix for Orders API latency p95 above 2s",
"source": "scm_suggested_fixes",
"repository": null,
"risk_level": "low",
"description": "This service has a stored fix from org memory that may apply again:\ntest and fixed",
"permissions": [
"read"
],
"target_system": "orders",
"rollback_notes": "Review prior fix context before repeating the same remediation.",
"suggestion_kind": "org_memory_hint",
"requires_approval": true
},
{
"title": "Review Prometheus connector status and query scope for orders service",
"executor": "manual_review_only",
"risk_level": "low",
"target_system": "Prometheus / connector configuration",
"rollback_notes": "No rollback required for read-only review.",
"requires_approval": false,
"required_permissions": []
},
{
"title": "Review Loki connector status and log label selectors for orders namespace",
"executor": "manual_review_only",
"risk_level": "low",
"target_system": "Loki / connector configuration",
"rollback_notes": "No rollback required for read-only review.",
"requires_approval": false,
"required_permissions": []
},
{
"title": "Inspect orders deployment and pod status via read-only API checks",
"executor": "manual_review_only",
"risk_level": "low",
"target_system": "Kubernetes / orders service",
"rollback_notes": "No rollback required for read-only review.",
"requires_approval": false,
"required_permissions": []
},
{
"title": "Review alert source configuration to confirm the latency threshold and data source used",
"executor": "manual_review_only",
"risk_level": "low",
"target_system": "Alerting configuration",
"rollback_notes": "No rollback required for read-only review.",
"requires_approval": false,
"required_permissions": []
}
]ordersprodsimilarity 5%
Open incidentOrders API latency p95 above 2s
1. Telemetry is absent for the orders service in the incident window. i. Prometheus returns zero series for up{service=~"orders|orders.*"}, so service availability cannot be confirmed [E1]. ii. Loki returns zero log streams for orders/* labels, so error, exception, crash, and panic signals cannot be evaluated [E2]. iii. No matching Prometheus alerting rules or active alerts exist for this incident context [E3]. 2. The incident title reports orders API p95 latency above 2s, but no metric, log, or rule evidence is available to verify which component, deployment, or dependency is the source. 3. Given the complete absence of observability data, the immediate actionable finding is an evidence gap or connector/configuration issue: Prometheus/Loki query selectors, namespaces, or endpoints may be misconfigured, unavailable, or returning no matching streams. Prior postmortems for checkout and payment are low-similarity and do not inform this incident.
Previous action plan
[
{
"url": null,
"title": "Prior verified fix: Fix for Orders API latency p95 above 2s",
"source": "scm_suggested_fixes",
"repository": null,
"risk_level": "low",
"description": "This service has a stored fix from org memory that may apply again:\ntest and fixed",
"permissions": [
"read"
],
"target_system": "orders",
"rollback_notes": "Review prior fix context before repeating the same remediation.",
"suggestion_kind": "org_memory_hint",
"requires_approval": true
},
{
"title": "Review Prometheus connectivity and service scrape configuration for orders",
"executor": "manual_review_only",
"risk_level": "low",
"target_system": "connector: prometheus",
"rollback_notes": "No rollback required for read-only review.",
"requires_approval": false,
"required_permissions": []
},
{
"title": "Review Loki connectivity and log label selectors for orders",
"executor": "manual_review_only",
"risk_level": "low",
"target_system": "connector: loki",
"rollback_notes": "No rollback required for read-only review.",
"requires_approval": false,
"required_permissions": []
},
{
"title": "Verify alert routing and service ownership metadata for the incident",
"executor": "manual_review_only",
"risk_level": "low",
"target_system": "evidence",
"rollback_notes": "No rollback required for read-only review.",
"requires_approval": false,
"required_permissions": []
}
]ordersprodsimilarity 0%
Open incidentPostmortem: Orders API latency p95 above 2s
test and fixed
Connector Policy For This Incident
Select connectors from alert labels, application context, and infrastructure topology.
Selected: loki, prometheus
Investigation Coverage
Connector selection, evidence collection, and gaps for the latest investigation.
Coverage will appear after the worker starts an investigation.
Connector Timeline
No connector runs recorded yet.
Evidence
No evidence collected yet.