CEX Stuck Node Recovery

Diagnose and recover stuck CEX withdrawal or observe nodes

CEX Stuck Node Diagnosis (Stuck Node Recovery — Diagnosis Runbook)

This is a diagnosis-only document for live CEX withdrawal/observe nodes. The scope of this document is limited to classifying stuck nodes and preserving evidence. Resolution must be escalated manually under the on-call operator’s review. When a follow-up manual-resolve CLI arrives, the automation scope will be redefined at that point. Until then, never force COMPLETED.

1. Header + scope

• Scope: diagnosis only. resolution is performed only under the on-call operator’s manual review.
• Follow-up: the automatic resolution flow is deferred to a future manual-resolve CLI.
• Hard rule: do not overwrite state without re-validating through the exchange API. In particular, forced transitions to COMPLETED are forbidden.
• Lane: any live CEX ↔ CEX withdrawal/observe pair.

2. Marker glossary

Marker	Location	Meaning
__submit_attempt_pending	provider_refs JSON field	After entering submit(), the SUBMITTED/FAILED outcome is still unresolved. This is a pending submit attempt that recover() must inspect after dispatch pre-persist.
__submit_attempt_started_at	provider_refs	ISO8601 timestamp when submit started. This is the reference point for the recovery lookback window.
__legacy_reconciliation_gated	provider_refs	Indicates that recover() blocked matcher entry on the legacy or AUTH-failure path.
__manual_review_required	provider_refs	Manager intervention required. Example values: "reconciliation_auth_failed", "reconciliation_ambiguous"

3. Diagnosis decision tree

1. DB query

   SELECT id, node_state, provider_refs, error_code, error_detail, updated_at
   FROM movement_request_nodes
   WHERE id = '<node-id>';
   

   Markers are stored inside the provider_refs JSON column (src/qtg/infrastructure/db/models.py). Code access also uses the provider_refs["__submit_attempt_pending"] pattern (src/qtg/application/services/recover.py).

2. Primary branch by marker pattern
   Follow the order below exactly.

   Important: the marker pattern alone does not distinguish the C2 sub-cause (lookback expired / 0 matches / ambiguity). Matcher ambiguity only raises RECONCILIATION_AMBIGUOUS in withdrawal_action.py; it does not persist a separate marker. Therefore always inspect the worker logs from the same time window together with the markers.

3. Inspect the log time window

   • C1: the recovery worker will pick it up soon. Candidate matching attempts should appear within the lookback window.
   • C2: look for logs such as RECONCILIATION_AMBIGUOUS, 0 matches, and lookback window exceeded.
   • C3a: look for traces of the pre-§18.4-defer-1 entry path, or traces of manual marker scrubbing.
   • C3b: look for recent RECONCILIATION_AUTH_FAILED and adapter _request() AUTH failures.
   • C5: check whether unknown_code warnings or mapping-miss errors repeat on every tick.
   • C6: look for signs of pre-API failure immediately before/after submit_withdrawal (network drop, 401 before request hit the exchange).
   • C7: confirm that observe pacing logs are normal.

4. Exchange console / inspect-node

   • For C6/C7, first confirm whether exchange_withdrawal_id exists in the exchange console.
   • For the C2/C3 family, capture matcher candidates with inspect-node-style tooling if needed.

4. Per-case sections

C1. Recovery imminent

• Marker pattern: SUBMITTING + __submit_attempt_pending=True, and still within the grace 90 seconds + lookback window.
• Log signals: worker restart immediately after submit, recover pickup after CancelledError/SIGTERM, history match attempts.
• Internal behavior: if the worker stops during submit, it hands the node to the recovery lane while preserving __submit_attempt_pending / __submit_attempt_started_at. After that, the recover path safely matches through the list_withdrawals lookback window (src/qtg/infrastructure/executors/cex/withdrawal_action.py).
• Required action: wait. This is an automatic recovery case.

C3b. AUTH fail then escalate

• Marker pattern: UNKNOWN + __legacy_reconciliation_gated=True + __manual_review_required="reconciliation_auth_failed"; __submit_attempt_pending=True may also be present.
• Log signals: RECONCILIATION_AUTH_FAILED, adapter AUTH failure, credential revoke/401-style logs.
• Internal behavior: if AUTH breaks during inline reconciliation or recover reconciliation, the node is intentionally left stuck while preserving the reservation. The purpose is to prevent double-spend from blind retry (src/qtg/infrastructure/executors/cex/withdrawal_action.py).
• Required action: fix AUTH first, then escalate. Frequency is still unknown.

C2. Lookback or matcher issue

• Marker pattern: UNKNOWN + __submit_attempt_pending=True, but there is no __manual_review_required="reconciliation_auth_failed".
• Log signals: RECONCILIATION_AMBIGUOUS, 0 matches, lookback window exceeded.
• Internal behavior: there are three main causes. (a) lookback window expiration (default 15 minutes), (b) amount/address/memo mismatch, (c) ambiguity because there are two or more candidates. Ambiguity appears only as TemporaryMovementError(... RECONCILIATION_AMBIGUOUS) and does not remain as a marker (src/qtg/infrastructure/executors/cex/withdrawal_action.py).
• Required action: escalate. There is currently no CLI other than manual operator review.

C3a. Legacy entry

• Marker pattern: UNKNOWN + __legacy_reconciliation_gated=True, without __submit_attempt_pending, and without __manual_review_required.
• Log signals: legacy UNKNOWN gate, traces of old-version entry, traces of manual marker scrubbing.
• Internal behavior: the new path enters the matcher only when both __submit_attempt_pending + __submit_attempt_started_at are present. If either is missing, it branches to the legacy gate (src/qtg/infrastructure/executors/cex/withdrawal_action.py).
• Required action: escalate. Preserve an incident artifact so it can later feed audit query improvements.

C5. Perpetual observing

• Marker pattern: OBSERVING persists for a long time and retry_after keeps getting reapplied on every poll.
• Log signals: per-tick unknown_code warning, mapping-miss error, or logs where the observe path absorbs transient errors at a 30-second cadence.
• Internal behavior: the causes are (a) status mapping gap, (b) catalog miss that only emits an emit_unknown_code_event() warning + Prometheus counter, (c) transient httpx-level errors absorbed by observer pacing (src/qtg/infrastructure/executors/cex/adapters/_telemetry.py, src/qtg/infrastructure/executors/cex/withdrawal_observe.py).
• Required action: bug + escalate. Capture a fixture, create a bug file, then connect it to catalog backfill (D4). The stuck node itself is still a manual escalation target.

C6. Not found

• Marker pattern: UNKNOWN + exchange_withdrawal_id is NOT FOUND in the exchange console, or exchange_withdrawal_id itself was never persisted.
• Log signals: submit-time network drop, pre-API 401, timeout, or other signs that the request never reached the exchange.
• Internal behavior: submit likely failed before it reached the exchange.
• Required action: use POST /v3/movements/{movement_id}/actions/cancel. The handlers live in src/qtg/interfaces/api/routes/dashboard/movements.py and src/qtg/interfaces/api/routes/movements.py. Do not assume a qtg-cancel CLI exists.

C7. Processing

• Marker pattern: UNKNOWN or OBSERVING, and the exchange console shows PROCESSING/PENDING.
• Log signals: observe polling repeats at a normal cadence, with no terminal/fatal signal.
• Internal behavior: this is actually a normal in-flight case. The exchange-side processing is simply not finished yet.
• Required action: wait.

5. C3b AUTH-fix sub-procedure

1. Check the adapter env vars (MG_*_ACCESS_KEY, MG_*_SECRET).
2. Run cex_submit_safety_drill preflight --venue <venue-key> --asset <asset>.
3. Confirm exit code 0 and that AUTH errors no longer occur.
4. If it still fails, proceed in this order: rotate the exchange-side key, update the env, restart the worker.
5. Once AUTH is restored, escalate that node manually. There is still no dedicated resolve CLI.

6. Manual escalation flow

• Incident artifact location: operator-local incident note <UTC>-<node-id>.md
• Start from the template below.

# Incident: <node-id> (<UTC>)
## Marker snapshot
## Log keywords (time window)
## Exchange console snapshot
## Matcher candidates (inspect-node output)
## Resolution (manual)

• Escalation chain: the single on-call operator.
• Minimum required artifacts:
  • DB snapshot (node_state, provider_refs, error_code, updated_at)
  • worker log keywords from the same time window
  • exchange console screenshot/note
  • whether matcher candidates exist

7. Forbidden actions

• Do not try a resume/retry while UNKNOWN + provider_refs.__submit_attempt_pending=True. The v3 race guard in retry_movement() (src/qtg/application/services/movements.py) rejects it; the underlying service blocks it regardless of any operator tooling.
• Do not force COMPLETED without re-validating through the exchange API. It can overwrite a double-spend signal.
• Do not skip incident artifact capture.