> ## Documentation Index
> Fetch the complete documentation index at: https://jephalabs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Auth Rollout

> HMAC authentication rollout plan and migration guide

# Auth Enable Rollout Runbook

> Purpose: an operator checklist to close **easy-to-miss preparation items** in order before switching `MG_AUTH_ENABLED=true`

***

## 1. What this document covers

This document organizes the preparation and verification procedure required when enabling v3 inbound HMAC auth **for the first time**.

Covered scope:

* caller inventory cleanup
* role assignment
* client/key bootstrap with the auth seed CLI
* verification of caller-side HMAC signing implementation
* staging enable sequence
* smoke test
* rollback

Out of scope:

* revoke/rotate lifecycle
* mTLS

***

## 2. Final target state

The final state should be the following.

* All `/v3/**` requests except `/healthz` require HMAC signatures
* Only registered callers can send requests
* Each caller is assigned one of three roles (`admin` / `operator` / `agent`)
* Replay nonces are recorded/rejected
* Operators can trace which caller uses which key

## 2-1. Current verification state (2026-03-21)

The following has already been verified.

* staging DB `qtg_v3` created
* five staging callers seeded
  * `dashboard-staging` → `operator`
  * `ops-cli-staging` → `operator`
  * `admin-cli-staging` → `admin`
  * `read-monitor-staging` → `operator`
  * `strategy-engine-staging` → `operator` (or `agent` if it only files template proposals)
* local real HTTP smoke test completed with `MG_AUTH_ENABLED=true`
  * `/healthz` → `200`
  * unsigned `GET /v3/signers` → `401`
  * operator key `GET /v3/signers` → `200`
  * operator key `POST /v3/movements` → `201`
  * `agent` key against an `admin`-only route (e.g. `PATCH /v3/executors/{executor_key}`) → `403`
  * operator key `POST /v3/movements/{id}/approve` → `200`
  * admin key on an admin route -> expected `404` after auth passes (route reached, target row absent)

Local artifacts:

* `.artifacts/auth-staging/staging-auth-seeds.json`
* `.artifacts/auth-staging/staging-auth-smoke-results.json`
* `.artifacts/auth-staging/staging-auth-external-http-smoke.json`

Notes:

* seed artifacts contain real secrets, so they must be treated as local secrets.
* Secret distribution to actual external caller processes (dashboard/ops-cli/strategy-engine) is not yet complete.

***

## 3. Preflight checklist

### A. Caller inventory

First, list every system that actually calls `/v3/**`.

| Caller          | Description       | Assigned role       | Actual route usage                             | Secret storage location | Done |
| --------------- | ----------------- | ------------------- | ---------------------------------------------- | ----------------------- | ---- |
| dashboard       | operator UI       | operator            | `/approve`, `/reject`, `GET /v3/*`             |                         | \[ ] |
| ops-cli         | operator CLI      | operator            | `/actions/*`, `POST /v3/movements`             |                         | \[ ] |
| admin-cli       | admin tasks       | admin               | `PATCH /v3/executors/*`, `PATCH /v3/signers/*` |                         | \[ ] |
| read-monitor    | status monitoring | operator            | `GET /v3/*`                                    |                         | \[ ] |
| strategy-engine | movement creation | operator (or agent) | `POST /v3/movements`, agent template proposals |                         | \[ ] |

> If even one caller is missing, 401/403 errors can occur immediately after enable.

### B. Verify role mapping

The three roles are:

* `admin` — full access, including template / signer / executor mutation
* `operator` — write and operate: create movements, approve/reject/retry/cancel, and all read surfaces
* `agent` — network-scoped automation: read surfaces plus its own agent-authority template proposals and bridge attempts (bound 1:1 to a `AgentAuthority`)

There is no `read` role; read surfaces are reachable by `operator` (and by `admin`/`agent`
where the route allows it). The route → allowed-roles map lives in `ROLE_ALLOWED_ROUTES`
(`src/qtg/interfaces/api/middleware/auth/hmac.py`). Current meanings:

* `POST /v3/movements` → any role (`admin` / `operator` / `agent`)
* `GET /v3/movements`, `GET /v3/movements/{id}`, `GET /v3/movements/{id}/timeline` → any role
* `GET /v3/whoami`, `GET /v3/balances`, `GET /v3/route-catalog` → any role
* `GET /v3/plan-templates/{template_id}` → any role
* `POST /v3/movements/{id}/approve|reject` → `admin` / `operator`
* `POST /v3/movements/{id}/actions/resume|retry|cancel` → `admin` / `operator`
* `GET /v3/executors`, `GET /v3/signers` → `admin` / `operator`
* `POST /v3/plan-templates`, `PATCH /v3/plan-templates/{template_key}` → `admin`
* `PATCH /v3/executors/{executor_key}` → `admin`
* `PATCH /v3/signers/{signer_key}` → `admin`
* `POST /v3/admin/cutover-mode` → `admin`
* `POST /v3/registry/verify-drift` → `admin`
* `POST /v3/agent-authorities/{authority_key}/template-proposals` → `agent`
* `POST /v3/agent-authorities/{authority_key}/bridge-attempt` → `agent`

> An `agent` key is additionally namespace-scoped: `AgentNamespaceMiddleware` rejects
> requests whose path does not match the key's bound `AgentAuthority` namespace with
> `403 agent_namespace_mismatch`.

***

## 4. Key bootstrap

### Seed CLI

Base command:

```bash theme={null}
uv run python -m qtg.interfaces.tools.seed_auth_client \
  --client-name dashboard \
  --key-id dashboard-staging \
  --role operator
```

Options:

* `--database-url` — explicit target DB
* `--hmac-secret` — set the secret directly
* `--cert-fingerprint` — initial client metadata value

Default behavior:

* Reuse the active client if one with the same name exists
* Otherwise create a new `ApiClient`
* Create a new `ApiClientKey`
* Generate a random secret if none is specified
* Print the result once to JSON stdout

### Checks during bootstrap

* [ ] agree on the `client_name` naming rule (`dashboard`, `ops-cli`, `strategy-engine`, etc.)
* [ ] agree on the `key_id` naming rule (`{caller}-{env}` recommended)
* [ ] retrieve `hmac_secret` from JSON stdout safely
* [ ] prevent the secret from remaining in shell history / CI logs / shared paste
* [ ] place the secret into the actual caller's secret store

## 5. Caller implementation checklist

Each caller must implement/verify the following.

* [ ] `X-QTG-Key-Id`
* [ ] `X-QTG-Timestamp`
* [ ] `X-QTG-Nonce`
* [ ] `X-QTG-Signature`
* [ ] canonical string matches the server

Canonical string:

```text theme={null}
METHOD
PATH
QUERY
SHA256(raw_body_bytes)
TIMESTAMP
NONCE
```

* `PATH` is the routed, percent-decoded path the server dispatches on — no origin, no query string.
* `QUERY` is the query **exactly as sent**: no leading `?`, **no sorting**, empty string if absent.
  Sorting before signing transmits bytes you did not sign, and every such request 401s.

Additional checks:

* [ ] nonce is a new value for every request
* [ ] timestamp uses the current epoch seconds
* [ ] body hash is computed from **raw bytes**
* [ ] GET requests also require auth (except `/healthz`)

## 6. Staging enable sequence

### Phase 1 — verify wiring with auth off

* [ ] `MG_AUTH_ENABLED=false`
* [ ] deploy so that each caller can create signed requests
* [ ] prepare a smoke helper that can send signed requests in advance

### Phase 2 — seed complete

* [ ] all required callers are seeded in the staging DB
* [ ] roles match the expected values
* [ ] secret distribution is complete for each caller

### Phase 3 — smoke test

positive test:

* [ ] operator key → `GET /v3/signers`
* [ ] operator key → `POST /v3/movements`
* [ ] operator key → `POST /v3/movements/{id}/approve`
* [ ] operator key -> `POST /v3/movements/{id}/actions/retry` or `/cancel`
* [ ] admin key -> `PATCH /v3/executors/{executor_key}` or `PATCH /v3/signers/{signer_key}`
* [ ] agent key → `POST /v3/agent-authorities/{authority_key}/template-proposals` (within its namespace)

negative test:

* [ ] unsigned request → `401`
* [ ] wrong secret → `401`
* [ ] stale timestamp → `401`
* [ ] replay nonce → `401`
* [ ] role not allowed for the route → `403`
* [ ] agent key outside its bound namespace → `403` (`agent_namespace_mismatch`)

### Phase 4 — enable

* [ ] `MG_AUTH_ENABLED=true` in the staging env
* [ ] app restart / redeploy
* [ ] rerun the smoke test

## 7. Checks immediately after enable

* [ ] `/healthz` is still unauthenticated
* [ ] verify that unsigned `/v3/**` requests are blocked
* [ ] expected callers work normally
* [ ] auth failure reasons are logged
* [ ] verify `NonceRegistry` entries are recorded

Most common operator mistakes:

* a read-only monitor forgets GET auth
* an ad-hoc curl script is unsigned
* timestamp/nonce generation differs by caller, causing canonical mismatch

## 8. Rollback

If a problem occurs after auth enable, the simplest rollback is:

1. `MG_AUTH_ENABLED=false`
2. app restart / redeploy

Notes:

* Seeded `ApiClient` / `ApiClientKey` rows do not need to be deleted during rollback
* After identifying the root cause, fix the signed caller side and retry

## 9. Completion criteria

If all of the following are satisfied, consider "auth enable preparation complete."

* caller inventory documented
* each caller role finalized
* key bootstrap completed with the seed CLI
* secret distribution completed
* staging positive/negative smoke tests passed
* rollback procedure verified

Only after that should `MG_AUTH_ENABLED=true` be enabled.

***

## 10. Key Rotation Procedure

When a key must be replaced after auth enable (periodic rotation, suspected exposure, etc.).

### Principle

* Order: **Seed new -> Deploy -> Verify -> Revoke old**.
* If a new `key_id` is issued for the same `client_name`, it can stay active at the same time as the old key.
* Revoke the old key only after verifying that the caller works normally with the new key.

### Procedure

#### Step 1. Check the current key

```bash theme={null}
uv run python -m qtg.interfaces.tools.list_auth_keys \
  --database-url "$MG_DATABASE_URL"
```

#### Step 2. Issue a new key

```bash theme={null}
uv run python -m qtg.interfaces.tools.seed_auth_client \
  --client-name dashboard \
  --key-id dashboard-staging-v2 \
  --role operator \
  --database-url "$MG_DATABASE_URL"
```

* Retrieve the `hmac_secret` value from the output safely.
* To avoid leaving it in shell history, use `history -d` or `HISTCONTROL=ignorespace`.

#### Step 3. Deploy the new secret

Set the new `key_id` and `hmac_secret` in the caller's environment variables or `.env` file.

```env theme={null}
QTG_KEY_ID=dashboard-staging-v2
QTG_HMAC_SECRET=<new-secret>
```

Restart the caller process.

#### Step 4. Smoke test

Verify that normal requests succeed with the new key (see §6 Smoke test).

#### Step 5. Revoke the old key

```bash theme={null}
uv run python -m qtg.interfaces.tools.revoke_auth_key \
  --key-id dashboard-staging-v1 \
  --database-url "$MG_DATABASE_URL"
```

* Immediately after revoke, all requests using the old key are rejected with `401`.
* The DB row is not deleted and remains with `status=revoked` for auditability.

#### Step 6. Verify

```bash theme={null}
uv run python -m qtg.interfaces.tools.list_auth_keys \
  --database-url "$MG_DATABASE_URL"
```

* old key: `key_status=revoked`
* new key: `key_status=active`

### Emergency revoke (suspected exposure)

If exposure is suspected, run Step 5 immediately, then recover in order from Step 2.
In this case the order is old-key revoke -> new-key issue -> deployment, so API access for that caller is interrupted from the revoke point until deployment of the new key is complete.

***

## 11. Secret deployment model

### Current model: env-based deployment

QTG uses a single-host Docker Compose + Tailscale access model.
Secrets are injected per caller through each process's environment variables.

| Deployment location               | Method                                        |
| --------------------------------- | --------------------------------------------- |
| QTG server                        | `.env` file -> Docker Compose `env_file`      |
| Dashboard (same host)             | `.env` or Docker env                          |
| Hummingbot controller (same host) | `.env` or config yaml                         |
| ops-cli (operator local)          | `~/.qtg/credentials` or environment variables |

### Secret protection rules

* `.env` files use `chmod 600` and are included in `.gitignore`
* The seed CLI output is shown only once, so save it immediately in a safe location
* Prevent secrets from remaining in CI/CD logs, shell history, or clipboard
* If later migrating to AWS Secrets Manager, Vault, and so on, keep env var names unchanged

***

## Rate Limiting

After HMAC auth passes, an in-memory fixed-window rate check runs before `call_next`.

The limiter picks one of two per-key tiers from the caller's **role class** (see
`RateLimiter.check` in `src/qtg/infrastructure/security/rate_limit.py`):

* **write tier** — keys whose role is `admin` or `operator` (the write/operate-class roles)
* **read tier** — `agent` keys (read-class)

| Setting                   | Default | Description                                                          |
| ------------------------- | ------- | -------------------------------------------------------------------- |
| `MG_RATE_LIMIT_ENABLED`   | `true`  | enables rate limiting (requires `MG_AUTH_ENABLED=true`)              |
| `MG_RATE_LIMIT_READ_RPM`  | `300`   | per-minute request limit for read-class (`agent`) keys               |
| `MG_RATE_LIMIT_WRITE_RPM` | `60`    | per-minute request limit for write-class (`admin` / `operator`) keys |

* The tier is selected by the key's role class, not by the individual route. A higher write
  limit therefore covers every route an `operator` or `admin` key can reach.
* When the limit is exceeded: `429 {"detail": "rate limit exceeded"}` + `Retry-After` header
* `/healthz` and `/dashboard/*` are exempt from rate limiting
* It does not operate when `rate_limit_enabled=True` is set while `auth_enabled=False`