> ## Documentation Index > Fetch the complete documentation index at: https://jephalabs.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # States and Transitions > Movement and node state machine — valid transitions and terminal states # States and Transitions > **Source of truth**: `src/qtg/domain/states.py` > **State derivation**: `src/qtg/application/services/advance.py` QTG v3 uses a two-layer state model: **RequestState** (the full request lifecycle) and **NodeState** (the progress state of each execution node). The request state is **derived** from the states of its nodes. *** ## Table of Contents 1. [RequestState Enum](#requeststate-enum) 2. [NodeState Enum](#nodestate-enum) 3. [REQUEST\_TRANSITIONS - request-state transition graph](#request_transitions) 4. [NODE\_TRANSITIONS - node-state transition graph](#node_transitions) 5. [State transition diagrams](#state-transition-diagrams) 6. [Terminal vs Recoverable states](#terminal-vs-recoverable-states) 7. [State Derivation — derive\_request\_state()](#state-derivation) 8. [apply\_request\_state\_from\_nodes() - side effects](#apply_request_state_from_nodes) 9. [Effect of the has\_side\_effect flag](#effect-of-the-has_side_effect-flag) 10. [Frontier advancement - advance\_after\_completion()](#frontier-advancement) *** ## RequestState Enum `RequestState` tracks the full lifecycle of a movement request. It is based on `enum.StrEnum` and stored in the DB column `movement_requests.request_state`. | Value | Description | Nature | | ----------------------- | ---------------------------------------------------------------------- | ------------ | | `RECEIVED` | signal/request received, validation not started yet | initial | | `VALIDATED` | input validation passed, plan compilation/binding completed | in progress | | `PENDING_APPROVAL` | waiting for operator approval — every movement enters this state | waiting | | `APPROVED` | approved, ready to execute | in progress | | `EXECUTING` | one or more nodes are running | in progress | | `WAITING_MANUAL_ACTION` | a `manual_gate` node is `READY`, and there are no active running nodes | waiting | | `COMPLETED` | all nodes are `COMPLETED` or `SKIPPED` | **Terminal** | | `FAILED` | unrecoverable failure (without a side effect) | **Terminal** | | `MANUAL_INTERVENTION` | a side-effect node completed, but another node is failed/unknown | **Terminal** | | `REJECTED` | approval rejected | **Terminal** | | `EXPIRED` | expired while waiting for approval | **Terminal** | | `CANCELLED` | cancelled by operator/system | **Terminal** | *** ## NodeState Enum `NodeState` is the execution progress state of an individual node in the plan graph. It is stored in the DB column `movement_request_nodes.node_state`. | Value | Description | Nature | | -------------------- | -------------------------------------------------------- | --------------- | | `BLOCKED` | predecessor nodes are incomplete - cannot run yet | initial | | `READY` | predecessor nodes are complete - ready to run | waiting | | `PREPARING` | `executor.preflight()` / `prepare()` in progress | running | | `AWAITING_SIGNATURE` | `signing_required=True`, waiting for `signer.sign()` | running | | `SUBMITTING` | `executor.submit()` in progress | running | | `SUBMITTED` | submit completed, waiting for external system processing | observing | | `OBSERVING` | `executor.observe()` polling in progress | observing | | `COMPLETED` | node execution completed | **Terminal** | | `FAILED` | deterministic (fatal) failure | **Terminal** | | `UNKNOWN` | unclear whether a side effect happened - recovery target | **Recoverable** | | `CANCELLED` | externally cancelled | **Terminal** | | `SKIPPED` | skipped on a conditional path | **Terminal** | *** ## REQUEST\_TRANSITIONS `REQUEST_TRANSITIONS` is the allowed transition graph of type `Mapping[RequestState, set[RequestState]]`. The `can_transition_request_state()` function checks this dictionary to decide whether a transition is allowed. ```python theme={null} REQUEST_TRANSITIONS: Mapping[RequestState, set[RequestState]] = { RECEIVED: {VALIDATED, FAILED}, VALIDATED: {PENDING_APPROVAL}, PENDING_APPROVAL: {APPROVED, REJECTED, EXPIRED}, APPROVED: {EXECUTING, FAILED}, EXECUTING: {COMPLETED, FAILED, MANUAL_INTERVENTION, WAITING_MANUAL_ACTION, CANCELLED}, WAITING_MANUAL_ACTION: {EXECUTING, CANCELLED}, MANUAL_INTERVENTION: {EXECUTING, FAILED, COMPLETED}, COMPLETED: {}, # terminal FAILED: {}, # terminal REJECTED: {}, # terminal EXPIRED: {}, # terminal CANCELLED: {}, # terminal } ``` ### Main transition paths | From | To | Trigger | | -------------------------------------- | --------------------------------------------------------------------------- | ------- | | `RECEIVED` -> `VALIDATED` | plan compilation + binding succeeded | | | `RECEIVED` -> `FAILED` | validation failed (input error, template not found, etc.) | | | `PENDING_APPROVAL` -> `APPROVED` | approval API called | | | `PENDING_APPROVAL` -> `REJECTED` | reject API called | | | `PENDING_APPROVAL` -> `EXPIRED` | `expires_at` passed | | | `APPROVED` -> `EXECUTING` | first `READY` node dispatch starts | | | `EXECUTING` -> `COMPLETED` | `derive_request_state()` - all nodes `COMPLETED`/`SKIPPED` | | | `EXECUTING` -> `FAILED` | `derive_request_state()` - there is a failed node and no side effect | | | `EXECUTING` -> `MANUAL_INTERVENTION` | `derive_request_state()` - failed node + completed side-effect node coexist | | | `EXECUTING` -> `WAITING_MANUAL_ACTION` | `derive_request_state()` - waiting on a manual gate node | | | `WAITING_MANUAL_ACTION` -> `EXECUTING` | manual gate resolved | | | `MANUAL_INTERVENTION` -> `EXECUTING` | operator retry/resume | | | `MANUAL_INTERVENTION` -> `COMPLETED` | operator force-complete | | | `MANUAL_INTERVENTION` -> `FAILED` | operator force-fail | | *** ## NODE\_TRANSITIONS ```python theme={null} NODE_TRANSITIONS: Mapping[NodeState, set[NodeState]] = { BLOCKED: {READY, SKIPPED}, READY: {PREPARING, OBSERVING, SKIPPED, FAILED}, PREPARING: {READY, AWAITING_SIGNATURE, SUBMITTING, FAILED}, AWAITING_SIGNATURE: {READY, SUBMITTING, FAILED}, SUBMITTING: {SUBMITTED, COMPLETED, FAILED, UNKNOWN}, SUBMITTED: {OBSERVING, COMPLETED, FAILED, UNKNOWN}, OBSERVING: {COMPLETED, FAILED, UNKNOWN}, UNKNOWN: {SUBMITTED, COMPLETED, FAILED}, COMPLETED: {}, # terminal FAILED: {}, # terminal CANCELLED: {}, # terminal SKIPPED: {}, # terminal } ``` ### Main transition paths | From | To | Trigger | | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | ------- | | `BLOCKED` -> `READY` | `advance_after_completion()` after predecessor completion | | | `BLOCKED` -> `SKIPPED` | skipped on a conditional path | | | `READY` -> `PREPARING` | dispatcher starts execution (prepare/submit lane) | | | `READY` -> `OBSERVING` | observe-only node dispatched straight into observation (no prepare/submit side effect) | | | `READY` -> `SKIPPED` | skipped on a conditional path | | | `READY` -> `FAILED` | dispatch-time fatal error (address allowlist, missing destination, ExecutionContext build) fails the node before any prepare step | | | `PREPARING` -> `AWAITING_SIGNATURE` | `signing_required=True` | | | `PREPARING` -> `SUBMITTING` | signing not required or signing completed | | | `PREPARING` -> `READY` | node re-queued to `READY` (re-dispatch / retry) | | | `PREPARING` -> `FAILED` | preflight failed | | | `AWAITING_SIGNATURE` -> `SUBMITTING` | signer returned a signature | | | `AWAITING_SIGNATURE` -> `READY` | node re-queued to `READY` (re-dispatch / retry) | | | `AWAITING_SIGNATURE` -> `FAILED` | signing failed deterministically | | | `SUBMITTING` -> `SUBMITTED` | submit succeeded, follow-up observation required | | | `SUBMITTING` -> `COMPLETED` | sync-terminal submit — submit returns the result with no in-node observation (CEX withdrawal, etc.) | | | `SUBMITTING` -> `UNKNOWN` | network timeout - side effect unclear | | | `SUBMITTING` -> `FAILED` | submit failed deterministically | | | `SUBMITTED` -> `OBSERVING` | first observe call | | | `SUBMITTED` -> `COMPLETED` | submitted side effect confirmed complete without an intermediate `OBSERVING` step | | | `SUBMITTED` -> `UNKNOWN` | a submitted side effect became ambiguous during observation (`observe()` raised or returned UNKNOWN) | | | `SUBMITTED` -> `FAILED` | observation returned a deterministic failure | | | `OBSERVING` -> `COMPLETED` | observe confirmed completion | | | `OBSERVING` -> `FAILED` | observe confirmed a deterministic failure | | | `OBSERVING` -> `UNKNOWN` | temporary RPC issue during observe | | | `UNKNOWN` -> `SUBMITTED` | recovery succeeded - txid confirmed | | | `UNKNOWN` -> `COMPLETED` | recovery confirmed completion directly | | | `UNKNOWN` -> `FAILED` | recovery confirmed deterministic failure | | *** ## State transition diagrams ### RequestState transition diagram ```mermaid theme={null} stateDiagram-v2 [*] --> RECEIVED RECEIVED --> VALIDATED RECEIVED --> FAILED VALIDATED --> PENDING_APPROVAL PENDING_APPROVAL --> APPROVED PENDING_APPROVAL --> REJECTED PENDING_APPROVAL --> EXPIRED APPROVED --> EXECUTING APPROVED --> FAILED EXECUTING --> COMPLETED EXECUTING --> FAILED EXECUTING --> MANUAL_INTERVENTION EXECUTING --> WAITING_MANUAL_ACTION EXECUTING --> CANCELLED WAITING_MANUAL_ACTION --> EXECUTING: resume WAITING_MANUAL_ACTION --> CANCELLED MANUAL_INTERVENTION --> EXECUTING MANUAL_INTERVENTION --> FAILED MANUAL_INTERVENTION --> COMPLETED COMPLETED --> [*] FAILED --> [*] REJECTED --> [*] EXPIRED --> [*] CANCELLED --> [*] ``` ### NodeState transition diagram ```mermaid theme={null} stateDiagram-v2 [*] --> BLOCKED BLOCKED --> READY BLOCKED --> SKIPPED READY --> PREPARING READY --> OBSERVING READY --> SKIPPED READY --> FAILED PREPARING --> AWAITING_SIGNATURE PREPARING --> SUBMITTING PREPARING --> READY PREPARING --> FAILED AWAITING_SIGNATURE --> SUBMITTING AWAITING_SIGNATURE --> READY AWAITING_SIGNATURE --> FAILED SUBMITTING --> SUBMITTED SUBMITTING --> COMPLETED SUBMITTING --> FAILED SUBMITTING --> UNKNOWN SUBMITTED --> OBSERVING SUBMITTED --> COMPLETED SUBMITTED --> FAILED SUBMITTED --> UNKNOWN OBSERVING --> COMPLETED OBSERVING --> FAILED OBSERVING --> UNKNOWN UNKNOWN --> SUBMITTED UNKNOWN --> COMPLETED UNKNOWN --> FAILED COMPLETED --> [*] FAILED --> [*] CANCELLED --> [*] SKIPPED --> [*] ``` ### NodeState linear path (happy path) ```mermaid theme={null} flowchart LR BLOCKED --> READY --> PREPARING --> AWAITING_SIGNATURE --> SUBMITTING --> SUBMITTED --> OBSERVING --> COMPLETED PREPARING -. signing not required .-> SUBMITTING SUBMITTED -. submit completes immediately .-> COMPLETED ``` *** ## Terminal vs Recoverable states ### Terminal states (no further transitions) **Terminal RequestState values**: `COMPLETED`, `FAILED`, `REJECTED`, `EXPIRED`, `CANCELLED` * They have an empty `set()` in `REQUEST_TRANSITIONS`. * When a terminal state is reached, reservation handling (`consumed`/`released`) and callback delivery run. **Terminal NodeState values**: `COMPLETED`, `FAILED`, `CANCELLED`, `SKIPPED` * These states have no transition targets. ### Recoverable states **`NodeState.UNKNOWN`** is the only recoverable node state. * `SUBMITTING` -> `UNKNOWN`: submit result is unclear because of a network timeout, etc. * `OBSERVING` -> `UNKNOWN`: RPC failure during observe `UNKNOWN` nodes are handled by the `recover_unknown_nodes()` worker: * It calls `executor.recover(context)` to check the result. * It transitions to `SUBMITTED` (recovery succeeded) or `FAILED` (deterministic failure), depending on the result. ```python theme={null} # Handle UNKNOWN nodes in `recover.py` next_state = NodeState(result.next_state) await set_node_state(session, node, next_state, actor_type='system', actor_id='recovery') if next_state == NodeState.FAILED: await apply_request_state_from_nodes(...) ``` **`RequestState.MANUAL_INTERVENTION`** is a recoverable request-level state: * The operator can resume it to `EXECUTING`, or force it to `FAILED`/`COMPLETED`. **`RequestState.WAITING_MANUAL_ACTION`** is the waiting state for a manual gate: * It returns to `EXECUTING` after the manual gate is resolved. *** ## State Derivation `derive_request_state()` checks the states of all nodes in a request and **derives** the request state. The system does not set `RequestState` directly from outside. It decides it from the set of node states. ### `derive_request_state()` logic ```python theme={null} def derive_request_state(nodes: list[MovementRequestNode]) -> RequestState | None: ``` #### Decision order (highest priority first) **Step 1: all-complete check** ```python theme={null} if all(node.node_state in (NodeState.COMPLETED, NodeState.SKIPPED) for node in nodes): return RequestState.COMPLETED ``` If all nodes are `COMPLETED` or `SKIPPED`, it returns `COMPLETED` immediately. **Step 2: classify failed/unknown nodes** ```python theme={null} failed_nodes = [node for node in nodes if node.node_state == NodeState.FAILED] unknown_nodes = [node for node in nodes if node.node_state == NodeState.UNKNOWN] completed_with_side_effect = [ node for node in nodes if node.node_state == NodeState.COMPLETED and node.has_side_effect ] ``` **Step 3: decide `MANUAL_INTERVENTION` vs `FAILED`** ```python theme={null} if failed_nodes or unknown_nodes: if completed_with_side_effect: return RequestState.MANUAL_INTERVENTION if failed_nodes: return RequestState.FAILED ``` Core rules: * If there is a failed/unknown node and there is also a **completed side-effect node** -> `MANUAL_INTERVENTION` * If there is a failed node and there is no completed side-effect node -> `FAILED` * If there are only unknown nodes (no failed nodes), and there is no completed side-effect node -> `None` (keep the current state) **Step 4: check manual-gate wait** ```python theme={null} manual_gate_frontier = [ node for node in nodes if node.node_state == NodeState.READY and node.node_key.startswith('manual_') ] if manual_gate_frontier and not any( node.node_state in (PREPARING, SUBMITTING, SUBMITTED, OBSERVING) for node in nodes ): return RequestState.WAITING_MANUAL_ACTION ``` * If there is a `READY` node whose name starts with the `manual_` prefix, * and there are no currently active running nodes, * then -> `WAITING_MANUAL_ACTION` **Step 5: no match** ```python theme={null} return None # no state change (execution continues) ``` ### Summary table of derived results | Node state combination | Derived result | | ---------------------------------------------- | --------------------------- | | all `COMPLETED`/`SKIPPED` | `COMPLETED` | | `FAILED` + completed side-effect node exists | `MANUAL_INTERVENTION` | | `UNKNOWN` + completed side-effect node exists | `MANUAL_INTERVENTION` | | `FAILED` + no completed side-effect node | `FAILED` | | only `UNKNOWN` + no completed side-effect node | `None` (keep current state) | | `manual_` `READY` + no active node | `WAITING_MANUAL_ACTION` | | otherwise (execution still in progress) | `None` (keep current state) | *** ## apply\_request\_state\_from\_nodes() This function writes the result of `derive_request_state()` to the DB and handles side effects (`callback`, `reservation`). ```python theme={null} async def apply_request_state_from_nodes( session: AsyncSession, request: MovementRequest, nodes: list[MovementRequestNode], *, actor_type: str, actor_id: str | None = None, ) -> RequestState | None: ``` ### Side effects by state | Derived state | Reservation | Callback | Frontier | | ------------------------------------------ | ----------- | ------------------- | -------------- | | `COMPLETED` | `consumed` | `request_completed` | `[]` (cleared) | | `FAILED` (only nodes without side effects) | `released` | `request_failed` | preserved | | `MANUAL_INTERVENTION` | preserved | `request_attention` | preserved | | `WAITING_MANUAL_ACTION` | preserved | `request_attention` | preserved | | `None` | no change | none | preserved | The release condition for a reservation is decided by `should_release_reservation()`: * `REJECTED`, `EXPIRED`: always release * `CANCELLED`, `FAILED`: release only when there is no completed side-effect node ```python theme={null} def should_release_reservation(request, nodes): if request.request_state in (REJECTED, EXPIRED): return True if request.request_state in (CANCELLED, FAILED): return not any( node.node_state == NodeState.COMPLETED and node.has_side_effect for node in nodes ) return False ``` *** ## Effect of the `has_side_effect` flag `has_side_effect` is defined on `MovementPlanNode.has_side_effect` and copied to `MovementRequestNode.has_side_effect`. ### Meaning * `True`: the node's executor can change external state (withdrawal submit, on-chain tx send, etc.) * `False`: read-only or observe-only (deposit observe, finality check, etc.) ### Role in state derivation If a node with a side effect is **already `COMPLETED`** and another node fails: * the funds may already have moved, so the system cannot treat it as a simple `FAILED`; * it transitions to `MANUAL_INTERVENTION` and requires operator judgment. **Example: 3-node CEX lane** ``` withdrawal_action (has_side_effect=True) -> withdrawal_observe -> deposit_observe ``` * `withdrawal_action` completed, `deposit_observe` failed -> `MANUAL_INTERVENTION` * the withdrawal completed, but deposit confirmation failed - manual confirmation is required * `withdrawal_action` failed -> `FAILED` * the withdrawal itself failed - the system can fail safely ### Role in reservation release If there is a completed side-effect node, the system does not release the reservation even on `FAILED`/`CANCELLED`. The funds may already be in transit, so the hardcap quota must stay reserved. *** ## Frontier Advancement `advance_after_completion()` changes successor nodes to `READY` after a node completes and updates the frontier. ```python theme={null} async def advance_after_completion( session, request, completed_node, nodes ): successor_keys = await get_successor_keys(session, completed_node.plan_node_id) if not successor_keys: # Final node - derive the request state await apply_request_state_from_nodes(...) return for node in nodes: if node.node_key in successor_keys and node.node_state == NodeState.BLOCKED: await set_node_state(session, node, NodeState.READY, ...) request.current_frontier = successor_keys await enqueue_callback(session, request, event_type='frontier_advanced') ``` ### Frontier advancement order 1. From the completed node's `plan_node`, look up successors along `on_success` edges. 2. Change successor nodes that are in `BLOCKED` state to `READY`. 3. Update `request.current_frontier` with the list of successor keys. 4. Send the `frontier_advanced` callback. 5. If there is no successor (end of the graph), call `apply_request_state_from_nodes()` to decide request termination. *** ## State transition validation functions State transitions can be checked for validity through the validation functions: ```python theme={null} def can_transition_request_state(current: RequestState, new_state: RequestState) -> bool: return new_state in REQUEST_TRANSITIONS.get(current, set()) def can_transition_node_state(current: NodeState, new_state: NodeState) -> bool: return new_state in NODE_TRANSITIONS.get(current, set()) ``` The actual state transitions are performed by `set_node_state()` and `set_request_state()`. Each transition creates a `MovementEvent` audit record. *** ## Cross-References * [types-and-enums.md](/reference/domain/types-and-enums) — graph-construction types such as NodeKind and EdgeType * [error-taxonomy.md](/reference/domain/error-taxonomy) — error categories and NodeState transition mapping * [../executors/overview.md](/reference/executors/overview) — executor lifecycle and state transitions * [../workers/runtime-workers.md](/reference/workers/runtime-workers) — dispatch/observe/recover workers * [../infrastructure/data-model.md](/reference/infrastructure/data-model) — MovementRequest and MovementRequestNode tables