States and Transitions

Movement and node state machine — valid transitions and terminal states

States and Transitions

Source of truth: src/qtg/domain/states.py State derivation: src/qtg/application/services/advance.py

QTG v3 uses a two-layer state model: RequestState (the full request lifecycle) and NodeState (the progress state of each execution node). The request state is derived from the states of its nodes.

---

Table of Contents

1. RequestState Enum
2. NodeState Enum
3. REQUEST_TRANSITIONS - request-state transition graph
4. NODE_TRANSITIONS - node-state transition graph
5. State transition diagrams
6. Terminal vs Recoverable states
7. State Derivation — derive_request_state()
8. apply_request_state_from_nodes() - side effects
9. Effect of the has_side_effect flag
10. Frontier advancement - advance_after_completion()

---

RequestState Enum

RequestState tracks the full lifecycle of a movement request. It is based on enum.StrEnum and stored in the DB column movement_requests.request_state.

Value	Description	Nature
RECEIVED	signal/request received, validation not started yet	initial
VALIDATED	input validation passed, plan compilation/binding completed	in progress
PENDING_APPROVAL	waiting for operator approval — every movement enters this state	waiting
APPROVED	approved, ready to execute	in progress
EXECUTING	one or more nodes are running	in progress
WAITING_MANUAL_ACTION	a manual_gate node is READY, and there are no active running nodes	waiting
COMPLETED	all nodes are COMPLETED or SKIPPED	Terminal
FAILED	unrecoverable failure (without a side effect)	Terminal
MANUAL_INTERVENTION	a side-effect node completed, but another node is failed/unknown	Terminal
REJECTED	approval rejected	Terminal
EXPIRED	expired while waiting for approval	Terminal
CANCELLED	cancelled by operator/system	Terminal

---

NodeState Enum

NodeState is the execution progress state of an individual node in the plan graph. It is stored in the DB column movement_request_nodes.node_state.

Value	Description	Nature
BLOCKED	predecessor nodes are incomplete - cannot run yet	initial
READY	predecessor nodes are complete - ready to run	waiting
PREPARING	executor.preflight() / prepare() in progress	running
AWAITING_SIGNATURE	signing_required=True, waiting for signer.sign()	running
SUBMITTING	executor.submit() in progress	running
SUBMITTED	submit completed, waiting for external system processing	observing
OBSERVING	executor.observe() polling in progress	observing
COMPLETED	node execution completed	Terminal
FAILED	deterministic (fatal) failure	Terminal
UNKNOWN	unclear whether a side effect happened - recovery target	Recoverable
CANCELLED	externally cancelled	Terminal
SKIPPED	skipped on a conditional path	Terminal

---

REQUEST_TRANSITIONS

REQUEST_TRANSITIONS is the allowed transition graph of type Mapping[RequestState, set[RequestState]]. The can_transition_request_state() function checks this dictionary to decide whether a transition is allowed.

REQUEST_TRANSITIONS: Mapping[RequestState, set[RequestState]] = {
    RECEIVED:              {VALIDATED, FAILED},
    VALIDATED:             {PENDING_APPROVAL},
    PENDING_APPROVAL:      {APPROVED, REJECTED, EXPIRED},
    APPROVED:              {EXECUTING, FAILED},
    EXECUTING:             {COMPLETED, FAILED, MANUAL_INTERVENTION,
                            WAITING_MANUAL_ACTION, CANCELLED},
    WAITING_MANUAL_ACTION: {EXECUTING, CANCELLED},
    MANUAL_INTERVENTION:   {EXECUTING, FAILED, COMPLETED},
    COMPLETED:             {},  # terminal
    FAILED:                {},  # terminal
    REJECTED:              {},  # terminal
    EXPIRED:               {},  # terminal
    CANCELLED:             {},  # terminal
}

Main transition paths

From	To	Trigger
RECEIVED -> VALIDATED	plan compilation + binding succeeded
RECEIVED -> FAILED	validation failed (input error, template not found, etc.)
PENDING_APPROVAL -> APPROVED	approval API called
PENDING_APPROVAL -> REJECTED	reject API called
PENDING_APPROVAL -> EXPIRED	expires_at passed
APPROVED -> EXECUTING	first READY node dispatch starts
EXECUTING -> COMPLETED	derive_request_state() - all nodes COMPLETED/SKIPPED
EXECUTING -> FAILED	derive_request_state() - there is a failed node and no side effect
EXECUTING -> MANUAL_INTERVENTION	derive_request_state() - failed node + completed side-effect node coexist
EXECUTING -> WAITING_MANUAL_ACTION	derive_request_state() - waiting on a manual gate node
WAITING_MANUAL_ACTION -> EXECUTING	manual gate resolved
MANUAL_INTERVENTION -> EXECUTING	operator retry/resume
MANUAL_INTERVENTION -> COMPLETED	operator force-complete
MANUAL_INTERVENTION -> FAILED	operator force-fail

---

NODE_TRANSITIONS

NODE_TRANSITIONS: Mapping[NodeState, set[NodeState]] = {
    BLOCKED:            {READY, SKIPPED},
    READY:              {PREPARING, OBSERVING, SKIPPED, FAILED},
    PREPARING:          {READY, AWAITING_SIGNATURE, SUBMITTING, FAILED},
    AWAITING_SIGNATURE: {READY, SUBMITTING, FAILED},
    SUBMITTING:         {SUBMITTED, COMPLETED, FAILED, UNKNOWN},
    SUBMITTED:          {OBSERVING, COMPLETED, FAILED, UNKNOWN},
    OBSERVING:          {COMPLETED, FAILED, UNKNOWN},
    UNKNOWN:            {SUBMITTED, COMPLETED, FAILED},
    COMPLETED:          {},  # terminal
    FAILED:             {},  # terminal
    CANCELLED:          {},  # terminal
    SKIPPED:            {},  # terminal
}

Main transition paths

From	To	Trigger
BLOCKED -> READY	advance_after_completion() after predecessor completion
BLOCKED -> SKIPPED	skipped on a conditional path
READY -> PREPARING	dispatcher starts execution (prepare/submit lane)
READY -> OBSERVING	observe-only node dispatched straight into observation (no prepare/submit side effect)
READY -> SKIPPED	skipped on a conditional path
READY -> FAILED	dispatch-time fatal error (address allowlist, missing destination, ExecutionContext build) fails the node before any prepare step
PREPARING -> AWAITING_SIGNATURE	signing_required=True
PREPARING -> SUBMITTING	signing not required or signing completed
PREPARING -> READY	node re-queued to READY (re-dispatch / retry)
PREPARING -> FAILED	preflight failed
AWAITING_SIGNATURE -> SUBMITTING	signer returned a signature
AWAITING_SIGNATURE -> READY	node re-queued to READY (re-dispatch / retry)
AWAITING_SIGNATURE -> FAILED	signing failed deterministically
SUBMITTING -> SUBMITTED	submit succeeded, follow-up observation required
SUBMITTING -> COMPLETED	sync-terminal submit — submit returns the result with no in-node observation (CEX withdrawal, etc.)
SUBMITTING -> UNKNOWN	network timeout - side effect unclear
SUBMITTING -> FAILED	submit failed deterministically
SUBMITTED -> OBSERVING	first observe call
SUBMITTED -> COMPLETED	submitted side effect confirmed complete without an intermediate OBSERVING step
SUBMITTED -> UNKNOWN	a submitted side effect became ambiguous during observation (observe() raised or returned UNKNOWN)
SUBMITTED -> FAILED	observation returned a deterministic failure
OBSERVING -> COMPLETED	observe confirmed completion
OBSERVING -> FAILED	observe confirmed a deterministic failure
OBSERVING -> UNKNOWN	temporary RPC issue during observe
UNKNOWN -> SUBMITTED	recovery succeeded - txid confirmed
UNKNOWN -> COMPLETED	recovery confirmed completion directly
UNKNOWN -> FAILED	recovery confirmed deterministic failure

---

State transition diagrams

RequestState transition diagram

NodeState transition diagram

NodeState linear path (happy path)

---

Terminal vs Recoverable states

Terminal states (no further transitions)

Terminal RequestState values: COMPLETED, FAILED, REJECTED, EXPIRED, CANCELLED

• They have an empty set() in REQUEST_TRANSITIONS.
• When a terminal state is reached, reservation handling (consumed/released) and callback delivery run.

Terminal NodeState values: COMPLETED, FAILED, CANCELLED, SKIPPED

• These states have no transition targets.

Recoverable states

NodeState.UNKNOWN is the only recoverable node state.

• SUBMITTING -> UNKNOWN: submit result is unclear because of a network timeout, etc.
• OBSERVING -> UNKNOWN: RPC failure during observe

UNKNOWN nodes are handled by the recover_unknown_nodes() worker:

• It calls executor.recover(context) to check the result.
• It transitions to SUBMITTED (recovery succeeded) or FAILED (deterministic failure), depending on the result.

# Handle UNKNOWN nodes in `recover.py`
next_state = NodeState(result.next_state)
await set_node_state(session, node, next_state, actor_type='system', actor_id='recovery')
if next_state == NodeState.FAILED:
    await apply_request_state_from_nodes(...)

RequestState.MANUAL_INTERVENTION is a recoverable request-level state:

• The operator can resume it to EXECUTING, or force it to FAILED/COMPLETED.

RequestState.WAITING_MANUAL_ACTION is the waiting state for a manual gate:

• It returns to EXECUTING after the manual gate is resolved.

---

State Derivation

derive_request_state() checks the states of all nodes in a request and derives the request state. The system does not set RequestState directly from outside. It decides it from the set of node states.

derive_request_state() logic

def derive_request_state(nodes: list[MovementRequestNode]) -> RequestState | None:

Decision order (highest priority first)

Step 1: all-complete check

if all(node.node_state in (NodeState.COMPLETED, NodeState.SKIPPED) for node in nodes):
    return RequestState.COMPLETED

If all nodes are COMPLETED or SKIPPED, it returns COMPLETED immediately.

Step 2: classify failed/unknown nodes

failed_nodes = [node for node in nodes if node.node_state == NodeState.FAILED]
unknown_nodes = [node for node in nodes if node.node_state == NodeState.UNKNOWN]
completed_with_side_effect = [
    node for node in nodes
    if node.node_state == NodeState.COMPLETED and node.has_side_effect
]

Step 3: decide MANUAL_INTERVENTION vs FAILED

if failed_nodes or unknown_nodes:
    if completed_with_side_effect:
        return RequestState.MANUAL_INTERVENTION
    if failed_nodes:
        return RequestState.FAILED

Core rules:

• If there is a failed/unknown node and there is also a completed side-effect node -> MANUAL_INTERVENTION
• If there is a failed node and there is no completed side-effect node -> FAILED
• If there are only unknown nodes (no failed nodes), and there is no completed side-effect node -> None (keep the current state)

Step 4: check manual-gate wait

manual_gate_frontier = [
    node for node in nodes
    if node.node_state == NodeState.READY and node.node_key.startswith('manual_')
]
if manual_gate_frontier and not any(
    node.node_state in (PREPARING, SUBMITTING, SUBMITTED, OBSERVING)
    for node in nodes
):
    return RequestState.WAITING_MANUAL_ACTION

• If there is a READY node whose name starts with the manual_ prefix,
• and there are no currently active running nodes,
• then -> WAITING_MANUAL_ACTION

Step 5: no match

return None  # no state change (execution continues)

Summary table of derived results

Node state combination	Derived result
all COMPLETED/SKIPPED	COMPLETED
FAILED + completed side-effect node exists	MANUAL_INTERVENTION
UNKNOWN + completed side-effect node exists	MANUAL_INTERVENTION
FAILED + no completed side-effect node	FAILED
only UNKNOWN + no completed side-effect node	None (keep current state)
manual_ READY + no active node	WAITING_MANUAL_ACTION
otherwise (execution still in progress)	None (keep current state)

---

apply_request_state_from_nodes()

This function writes the result of derive_request_state() to the DB and handles side effects (callback, reservation).

async def apply_request_state_from_nodes(
    session: AsyncSession,
    request: MovementRequest,
    nodes: list[MovementRequestNode],
    *,
    actor_type: str,
    actor_id: str | None = None,
) -> RequestState | None:

Side effects by state

Derived state	Reservation	Callback	Frontier
COMPLETED	consumed	request_completed	[] (cleared)
FAILED (only nodes without side effects)	released	request_failed	preserved
MANUAL_INTERVENTION	preserved	request_attention	preserved
WAITING_MANUAL_ACTION	preserved	request_attention	preserved
None	no change	none	preserved

The release condition for a reservation is decided by should_release_reservation():

• REJECTED, EXPIRED: always release
• CANCELLED, FAILED: release only when there is no completed side-effect node

def should_release_reservation(request, nodes):
    if request.request_state in (REJECTED, EXPIRED):
        return True
    if request.request_state in (CANCELLED, FAILED):
        return not any(
            node.node_state == NodeState.COMPLETED and node.has_side_effect
            for node in nodes
        )
    return False

---

Effect of the has_side_effect flag

has_side_effect is defined on MovementPlanNode.has_side_effect and copied to MovementRequestNode.has_side_effect.

Meaning

• True: the node’s executor can change external state (withdrawal submit, on-chain tx send, etc.)
• False: read-only or observe-only (deposit observe, finality check, etc.)

Role in state derivation

If a node with a side effect is already COMPLETED and another node fails:

• the funds may already have moved, so the system cannot treat it as a simple FAILED;
• it transitions to MANUAL_INTERVENTION and requires operator judgment.

Example: 3-node CEX lane

withdrawal_action (has_side_effect=True) -> withdrawal_observe -> deposit_observe

• withdrawal_action completed, deposit_observe failed -> MANUAL_INTERVENTION
  • the withdrawal completed, but deposit confirmation failed - manual confirmation is required
• withdrawal_action failed -> FAILED
  • the withdrawal itself failed - the system can fail safely

Role in reservation release

If there is a completed side-effect node, the system does not release the reservation even on FAILED/CANCELLED. The funds may already be in transit, so the hardcap quota must stay reserved.

---

Frontier Advancement

advance_after_completion() changes successor nodes to READY after a node completes and updates the frontier.

async def advance_after_completion(
    session, request, completed_node, nodes
):
    successor_keys = await get_successor_keys(session, completed_node.plan_node_id)
    if not successor_keys:
        # Final node - derive the request state
        await apply_request_state_from_nodes(...)
        return

    for node in nodes:
        if node.node_key in successor_keys and node.node_state == NodeState.BLOCKED:
            await set_node_state(session, node, NodeState.READY, ...)

    request.current_frontier = successor_keys
    await enqueue_callback(session, request, event_type='frontier_advanced')

Frontier advancement order

1. From the completed node’s plan_node, look up successors along on_success edges.
2. Change successor nodes that are in BLOCKED state to READY.
3. Update request.current_frontier with the list of successor keys.
4. Send the frontier_advanced callback.
5. If there is no successor (end of the graph), call apply_request_state_from_nodes() to decide request termination.

---

State transition validation functions

State transitions can be checked for validity through the validation functions:

def can_transition_request_state(current: RequestState, new_state: RequestState) -> bool:
    return new_state in REQUEST_TRANSITIONS.get(current, set())

def can_transition_node_state(current: NodeState, new_state: NodeState) -> bool:
    return new_state in NODE_TRANSITIONS.get(current, set())

The actual state transitions are performed by set_node_state() and set_request_state(). Each transition creates a MovementEvent audit record.

---

Cross-References

• types-and-enums.md — graph-construction types such as NodeKind and EdgeType
• error-taxonomy.md — error categories and NodeState transition mapping
• ../executors/overview.md — executor lifecycle and state transitions
• ../workers/runtime-workers.md — dispatch/observe/recover workers
• ../infrastructure/data-model.md — MovementRequest and MovementRequestNode tables