KMS Signer Rotation

Step-by-step runbook for rotating AWS KMS signing keys

KMS Signer Rotation Runbook

Scope: Scheduled rotation of a production EVM AwsKmsEvmSigner key. Covers preflight → drill → cut-over → KMS key deletion → exception handling.

See the Pro signer lifecycle reference for the lifecycle API and CLI contract.

This runbook covers prod EVM AwsKmsEvmSigner only. For Callback HMAC v3, CEX API keys, or Tier-3 local signer rotation, see § 7 Related rotation procedures.

---

1. Overview & Control Objective Mapping

Control objective:

KMS asymmetric key rotation is a scheduled operator-run procedure. The procedure, actor, asset movement, and completion time leave an evidence chain in the audit log and rotation snapshot.

Evidence item	Storage location
Action record	audit_events row (INSERT-only, PG TRIGGER enforced)
Detailed payload	signer_rotation_events.snapshot (FK audit_event_id NOT NULL)
Actor identity	audit_events.actor_client_id + actor_key_id (RBAC v2 admin role)
Failure/block record	Retirement hard-gate denials only: retire_denied records lifecycle audit/event evidence. Identity and health rotation denials occur before lifecycle evidence and create no lifecycle audit event or SignerRotationEvent; they return only a code-only HTTP error.

Control limit: this covers app-role compromise protection. It does not cover a broad AWS account compromise or IAM root takeover.

Evidence chain quick-look query

SELECT
  ae.created_at,
  ae.entity_type,
  ae.entity_key,
  ae.action,
  ae.outcome,
  ae.actor_client_id,
  ae.actor_key_id,
  ae.namespace,
  ae.new_value,
  sre.event_type,
  sre.snapshot
FROM audit_events ae
LEFT JOIN signer_rotation_events sre ON sre.audit_event_id = ae.id
WHERE ae.entity_type = 'signer'
  AND ae.created_at >= NOW() - INTERVAL '12 months'
ORDER BY ae.created_at;

---

2. Preflight

2-1. Resolve the current signer identity (required — Gate 3 backstop)

The Gate 3 nonterminal-node scan also matches signer_binding['signer_address_snapshot']. Record old_address before rotation. Use the supported synchronous bootstrap method; do not adapt an older async probe:

uv run python -c '
from qtg.infrastructure.signers.aws_kms_evm import AwsKmsEvmSigner
signer = AwsKmsEvmSigner(
    signer_key="<OLD_SIGNER_KEY>",
    kms_key_id="<OLD_KMS_KEY_ID>",
    aws_region="<AWS_REGION>",
    aws_profile=None,
)
print(signer.bootstrap_signer_address())
'

bootstrap_signer_address() is synchronous. Record only the resulting address in the operator note, never KMS credentials or provider response payloads.

2-2. Create a new KMS asymmetric key

aws kms create-key \
  --key-usage SIGN_VERIFY \
  --key-spec ECC_SECG_P256K1 \
  --description "qtg-signer-<NAME>-<YYYY-QN>" \
  --region <AWS_REGION>

Record the output KeyMetadata.KeyId and KeyMetadata.Arn.

Minimum IAM policy (grant to the QTG runtime role)

{
  "Effect": "Allow",
  "Action": ["kms:DescribeKey", "kms:GetPublicKey", "kms:Sign"],
  "Resource": "arn:aws:kms:<REGION>:<ACCOUNT_ID>:key/<NEW_KEY_ID>"
}

2-3. Confirm the new signer EVM address

uv run python -c '
from qtg.infrastructure.signers.aws_kms_evm import AwsKmsEvmSigner
signer = AwsKmsEvmSigner(
    signer_key="<NEW_SIGNER_KEY>",
    kms_key_id="<NEW_KMS_KEY_ID>",
    aws_region="<AWS_REGION>",
    aws_profile=None,
)
print(signer.bootstrap_signer_address())
'

Record the output as candidate_live_address. It is the fresh live identity used by the defense-in-depth verification below.

2-4. Enroll the replacement through runtime bootstrap

Add an aws_kms_evm replacement entry to MG_EXTRA_SIGNERS_JSON using the same signer_key, kms_key_id, aws_region, and optional aws_profile / aws_endpoint_url fields used by the runtime bootstrap. Treat the JSON value as secret-bearing configuration: do not paste it into tickets, logs, transcripts, or the operator evidence packet.

Restart the Pro application. Startup registers the configured signer and runs sync_registry_to_db() to materialize its registry row. Do not create a replacement runtime enrollment with direct SQL: a database row alone neither registers the in-memory signer nor proves its KMS identity.

2-5. Verify roster enrollment and pin expected identity

Keep workers enabled with MG_WORKERS_ENABLED=true and wait for the executor-health worker to complete a successful signer health refresh. That worker persists the EVM observed_signer_address; a roster GET alone does not materialize the observed identity.

Before pinning, confirm GET /v3/signers includes <NEW_SIGNER_KEY> as active with a fresh observed identity and the deliberately unpinned state:

{
  "signer_key": "<NEW_SIGNER_KEY>",
  "status": "active",
  "observed_signer_address": "<CANDIDATE_LIVE_ADDRESS>",
  "identity_state": "missing_expected"
}

This first roster read proves runtime enrollment and health refresh, not identity approval. If the signer is not active, has no observed identity, or does not show identity_state: "missing_expected", STOP and fix enrollment before the expected-identity request. observed_signer_address is diagnostic and non-authoritative; it is not a substitute for the server’s fresh rotation-time comparison.

Pin the independently recorded candidate identity with the shared signed HTTP client. QTG_KEY_ID, QTG_HMAC_SECRET_FILE (or QTG_HMAC_SECRET), and QTG_API_URL must already be configured; do not print any of their values. QtgHttpClient.request("PUT", ...) provides the HMAC-signed transport; there is no dedicated expected-identity CLI command.

from qtg.clients.qtg_http import build_default_client

signer_key = "<NEW_SIGNER_KEY>"
with build_default_client() as client:
    result = client.request("PUT", f"/v3/signers/{signer_key}/expected-identity",
        json_body={
            "expected_address": "<CANDIDATE_LIVE_ADDRESS>",
            "reason": "scheduled enrollment <YYYY-QN>",
        },
    )
print({key: result[key] for key in ("signer_key", "outcome", "identity_state")})

Re-read GET /v3/signers after the successful PUT. Require the pinned expected_signer_address, the fresh observed_signer_address, and identity_state: "match". Any absence, invalid address, or mismatch is a STOP; do not proceed to drill or cut-over.

2-6. Defense-in-depth identity verification (required)

Rotation-time identity enforcement runs inside the Pro server. For every --drill request and every live --ack-impact rotation, the server automatically resolves the replacement signer’s fresh runtime identity and compares it with the persisted expected_signer_address. The CLI sends the rotation request; it does not perform this comparison locally.

The server fails closed before impact calculation or any audit event, rotation event, metadata, or authority-remap write. observed_signer_address is diagnostic and non-authoritative; identity_state: "match" remains useful roster and health-refresh evidence but cannot authorize rotation. Identity denial returns HTTP 409 with one stable code and no address or backend-cause detail:

Error	Meaning
signer_expected_identity_missing	No operator pin exists.
signer_expected_identity_invalid	The persisted pin is malformed or zero.
signer_runtime_identity_unavailable	Fresh identity cannot be resolved or runtime signer continuity is lost.
signer_runtime_identity_invalid	The fresh runtime value is malformed or zero.
signer_identity_mismatch	Canonical expected and fresh identities differ.

Keep the independent KMS probe from Step 2-3 and compare its result with the pinned expected value before drill. This manual probe remains a defense-in-depth check; missing, invalid, or mismatched values are a STOP even though the server repeats the fresh resolution and comparison for the request.

2-7. Check new signer health (drill mode)

uv run qtg signer rotate <OLD_SIGNER_KEY> \
  --replace-with <NEW_SIGNER_KEY> \
  --drill \
  --reason "preflight-health-<YYYY-QN>"

Drill mode runs validation, persists drill evidence, and produces an impact report without changing signer bindings. Before calculating impact or persisting that evidence, the server freshly resolves and verifies the replacement identity. Confirm outcome: "drilled" in the output JSON.

---

3. Run Drill Mode (recommended before cut-over)

uv run qtg signer rotate <OLD_SIGNER_KEY> \
  --replace-with <NEW_SIGNER_KEY> \
  --drill \
  --reason "drill-<YYYY-QN>"

Impact report review checklist

Field	What to check
affected_authority_mappings	Matches the expected number of authorities
remap_action per row	Correct update_in_place vs delete_then_keep_existing branch
collision_existing_row_id	Whether an existing row collision occurred; if present, an idempotent re-run may be possible
existing_account_address_changed	If true, the existing row address is stale and will change after cut-over

Drill mode does not change signer bindings. Record the audit/event identifiers and the impact report in the sanitized operator evidence.

---

4. Live Cut-over

Follow the 9-step sequence. Complete each step before moving to the next.

Step 1 — Re-resolve replacement identity immediately before rotation

Do not reuse candidate_live_address from the earlier drill preparation. Immediately before the state-changing command, resolve the replacement KMS identity again and compare it with the pinned value from the latest roster GET:

uv run python -c '
from qtg.infrastructure.signers.aws_kms_evm import AwsKmsEvmSigner
signer = AwsKmsEvmSigner(
    signer_key="<NEW_SIGNER_KEY>",
    kms_key_id="<NEW_KMS_KEY_ID>",
    aws_region="<AWS_REGION>",
    aws_profile=None,
)
pre_cutover_live_address = signer.bootstrap_signer_address()
pinned_expected_address = "<EXPECTED_SIGNER_ADDRESS_FROM_LATEST_GET>"
if pre_cutover_live_address.lower() != pinned_expected_address.lower():
    raise SystemExit("STOP: fresh KMS identity does not match pinned expected identity")
print(pre_cutover_live_address)
'

An absent, invalid, or mismatched fresh identity is a STOP. Do not rotate until the pinned expected_signer_address and pre_cutover_live_address match; this is a second independent resolution, not a reuse of the drill probe. It is defense in depth: the server performs its own fresh resolution and comparison again when it handles the live rotation request.

From this fresh probe through completion of the rotation response, STOP if any operator or deployment action would repoint or mutate the replacement KMS alias/key or replace the runtime signer object. Do not continue the in-flight request; restore a stable configuration, repeat the fresh probe, and submit a new rotation request.

The automatic gate validates one fresh identity snapshot plus continuity of that same runtime signer object for the request; it does not perform a second in-lock KMS identity resolution. Keep the replacement configuration stable until the rotation response completes.

Step 2 — Rotate (deprecate old, remap authorities)

uv run qtg signer rotate <OLD_SIGNER_KEY> \
  --replace-with <NEW_SIGNER_KEY> \
  --ack-impact \
  --reason "scheduled-<YYYY-QN>"

The server, not the CLI, repeats the fresh replacement-identity resolution and expected-pin comparison before it calculates impact or writes lifecycle evidence.

On success, old_signer_key.rotation_metadata.deprecated_from is set. The OLD key is then excluded from new movement and template-compile candidates. In-flight nodes can continue to sign, observe, and recover with the OLD key. This is expected.

Step 3 — Capture pre-drain inventory

uv run qtg signer inventory <OLD_SIGNER_KEY>

Attach the output snapshot to the incident log. Check native gas, USDC, and USDT balances per chain.

Step 4 — Manual drain (per chain)

For each chain, use this order: (1) provision gas -> (2) move ERC-20 assets -> (3) move native dust last. Use a QTG movement or direct calldata. A retire attempt before drain completion is blocked by Gate 2.

Step 5 — Scan allowances

uv run qtg signer allowances <OLD_SIGNER_KEY>

Record the non-zero allowance list. Revoke per protocol: CCTP -> Circle bridge UI, Stargate -> Stargate UI / Etherscan write, Circle Gateway -> approve(0) calldata.

Step 6 — Manual revoke

Revoke each non-zero allowance from Step 5 in the protocol UI. Record revoke txids.

Step 7 — Recheck post-drain inventory

uv run qtg signer inventory <OLD_SIGNER_KEY>

Confirm all balances are at or below the dust level. If any balance remains, rerun Step 4.

Step 8 — Recheck allowances

uv run qtg signer allowances <OLD_SIGNER_KEY>

If non-zero allowances remain, complete revoke before proceeding. Use --ack-allowances only when there is a reason revoke cannot be completed, and record that reason in the operator note.

Step 9 — Retire after a zero-allowance scan

uv run qtg signer retire <OLD_SIGNER_KEY> \
  --dust-threshold-usd 1.0

Use --ack-allowances only when non-zero allowances deliberately remain and the operator has recorded why they cannot be revoked; then use the exceptional acknowledged-residual command:

uv run qtg signer retire <OLD_SIGNER_KEY> \
  --ack-allowances \
  --dust-threshold-usd 1.0

Unit warning for --dust-threshold-usd

This parameter is not USD; it is token units in the smallest denomination. The Phase 7 v1 implementation compares raw balances without real USD conversion. In production, conservatively drain to actual zero before retiring. 1.0 means “1 token unit” (for example, 0.000001 USDC for USDC).

All four gates must pass for retire to succeed:

Gate	Condition
Gate 1	deprecated_from is not null (confirms Step 2 completed)
Gate 2	All balances <= dust_threshold_usd (token units)
Gate 3	No nonterminal MovementRequestNode references (signer_key + signer_address_snapshot match)
Gate 4	--ack-allowances is required if non-zero allowances remain

There is no --force flag. If a gate fails, see § 6 Exception scenarios.

---

5. Delete the AWS KMS Key (post cut-over, out of band)

This section is outside QTG. Run it after QTG retire completes.

1. Remove the kms:Sign / kms:GetPublicKey policy for the OLD key through the IAM console or CLI.

2. Schedule deletion. It can be canceled with cancel-key-deletion during the waiting period:

# Development/test: 7 days
aws kms schedule-key-deletion --key-id <OLD_KEY_ID> --pending-window-in-days 7

# Production recommendation: 30 days
aws kms schedule-key-deletion --key-id <OLD_KEY_ID> --pending-window-in-days 30

---

6. Exception scenarios

Drain does not complete

Cause	Action
Insufficient gas	Send a small amount of native token to the OLD address
RPC failure	Retry only the affected chain with --chains <CHAIN>
Protocol withdrawal lock	Wait for withdrawal completion, then retry
Stuck allowance	etherscan write → approve(spender, 0)

In-flight node breaks

Deprecated signers can still sign, observe, and recover in-flight nodes — deprecation stops new assignments, it does not revoke an in-flight one. If a node actually breaks, check the recovery worker logs and then run:

SELECT id, state, signer_binding, updated_at
FROM movement_request_nodes
WHERE state NOT IN ('completed','failed','cancelled')
  AND signer_binding::text LIKE '%<OLD_SIGNER_KEY>%';

If recovery is impossible, use the supported movement recovery actions (cancel, retry, or resume) and the ordinary operator recovery process. Do not mutate movement state directly or invent a failure endpoint.

Follow-up checks after drill failure

Drill failure means no state mutation occurred. Check whether the new signer passes health(), whether signer_registry.status = 'active', and whether signer_key has a typo.

Gate 2 dust threshold exceeded

retire denied: Gate 2 — balance exceeds dust_threshold_usd

Rerun Step 4 (Manual drain). After drain completes, rerun Steps 5–8, then retry Step 9 retirement.

Gate 3 finds nonterminal nodes

retire denied: Gate 3 — <N> nonterminal nodes reference this signer

SELECT mrn.id, mrn.state, m.id as movement_id, m.status as movement_status
FROM movement_request_nodes mrn
JOIN movements m ON m.id = mrn.movement_id
WHERE mrn.state NOT IN ('completed','failed','cancelled')
  AND (
    mrn.signer_binding->>'signer_key' = '<OLD_SIGNER_KEY>'
    OR mrn.signer_binding->>'signer_address_snapshot' = '<OLD_EVM_ADDRESS>'
  );

Options:

• Wait: wait until the node reaches a terminal state (recommended).
• Recover: use the supported cancel, retry, or resume action as appropriate, then retry retire only after the node is terminal.

Gate 4 allowance ack incomplete

After revoke completes, retry retire without --ack-allowances. Proceed with --ack-allowances only when there is a reason UI revoke cannot be completed, and record the incomplete-revoke reason in the operator note.

---

7. Related rotation procedures

Procedure	Document
Callback HMAC v3 rotation	docs/runbooks/secret-rotation-checklist.md § Callback HMAC v3
CEX API key rotation	docs/runbooks/secret-rotation-checklist.md § Exchange API keys
Tier-3 local signer rotation	docs/runbooks/secret-rotation-checklist.md § Local signer key material
ApiClientKey HMAC secret rotation	docs/runbooks/secret-rotation-checklist.md § ApiClientKey HMAC secrets

This runbook covers prod EVM AwsKmsEvmSigner only. The Pro KMS Signing Gateway (post-GA Pro spec), including runtime app-role compromise protection, is handled by a separate spec.

The AWS KMS rotation row in docs/runbooks/secret-rotation-checklist.md forward-references this runbook from the secret inventory table.

---

8. SOC2/ISO27001 Evidence Query

8-1. All rotation actions in the last 12 months

SELECT
  ae.created_at,
  ae.entity_type,
  ae.entity_key,
  ae.action,
  ae.outcome,
  ae.actor_client_id,
  ae.actor_key_id,
  ae.namespace,
  ae.new_value,
  sre.event_type,
  sre.snapshot
FROM audit_events ae
LEFT JOIN signer_rotation_events sre ON sre.audit_event_id = ae.id
WHERE ae.entity_type = 'signer'
  AND ae.created_at >= NOW() - INTERVAL '12 months'
ORDER BY ae.created_at;

event_type: rotate / rotate_drill / inventory / allowance_scan / retire / retire_denied. retire_denied is reserved for retirement hard-gate denial evidence. Identity and health rotation denials do not create an event.

8-2. Actor role attribution

Use actor_client_id + actor_key_id to look up the executor’s role:

SELECT ae.*, ack.role
FROM audit_events ae
LEFT JOIN api_clients ac ON ac.id::text = ae.actor_client_id
LEFT JOIN api_client_keys ack
  ON ack.client_id = ac.id
  AND ack.key_id = ae.actor_key_id
WHERE ae.entity_type = 'signer'
ORDER BY ae.created_at;

RBAC v2 alembic 0003 swapped the purpose column to role. Environments migrated from the old schema may have ack.role as NULL.

8-3. Query the full snapshot for a specific signer

SELECT ae.created_at, ae.action, ae.outcome, sre.event_type, sre.snapshot
FROM audit_events ae
JOIN signer_rotation_events sre ON sre.audit_event_id = ae.id
WHERE ae.entity_key = '<OLD_SIGNER_KEY>'
ORDER BY ae.created_at;

sre.snapshot: drill/rotate -> impact report JSON, inventory/retire -> balance snapshot.

---

References

• Pro runtime CLI: qtg signer
• Signer protocol and the local AWS KMS signer: reference/signers/signer-protocol.md
• Control objectives for this rotation: section 1 of this runbook