path: root/dev_guide/concurrency.md

# Concurrency & durability

This guide documents the runtime architecture decisions that keep live game
state safe: the single-owner loop, single-file durable money, the account
lifecycle transactions, and the graceful-shutdown sequence.

## Single-owner game state

One goroutine — the **owner loop** (`internal/game/owner.go`) — executes all
mutation of live game state: session input handlers, connect/disconnect
transitions, every tick callback (the master per-tick system sequence and all
dynamically registered callbacks), and admin mutations. Network and admin
goroutines submit tasks; they never touch live players, sessions, Hub
membership, or game runtime maps directly.

Rules:

- `Game.RunOnOwner(fn)` (and `Game.HandleSessionOnOwner` for session input)
  block the caller until the closure has executed on the owner. Connection
  goroutines submit one task per input line, giving per-client backpressure
  without ever blocking another client.
- A FULL task queue applies backpressure: submitters block until the owner
  drains a task, and a full queue never bypasses the owner — running a task
  on the caller's goroutine while the owner keeps executing would break the
  single-owner invariant. A STOPPED owner (graceful shutdown, after
  admissions have closed) refuses the task and drops it rather than running
  it on the caller's goroutine: `RunOnOwner` returns without executing the
  closure, because running it there would mutate live state concurrently
  with the shutdown sequence and after the final-save barrier. (Synthetic
  tests that construct a Game without an owner run tasks inline on the
  caller.)
- Code already running on the owner must NOT call `Game.RunOnOwner` — it
  would deadlock waiting for itself. Owner-executed code calls functions
  directly.
- Task execution is FIFO, so tick callbacks serialize against session input
  and admin mutations in submission order. Tick callbacks still run in
  engine registration-ID order (see tick_order.md); the owner wrapper only
  moves their execution onto the owner goroutine.
- Slow clients cannot stall the world: `Session` writes are queued
  (`net.Session.StartWriter` puts a transport session into queued mode) and a
  dedicated per-session writer goroutine performs the network I/O with
  per-message write deadlines. A full queue disconnects only that client.
- Saves run on the owner and serialize a value snapshot; concurrent
  read-modify-write cycles of the same files cannot happen.
- Password work is the one deliberate exception: Argon2id verification and
  hashing (~64 MiB, tens of milliseconds) run on the CALLER's connection
  goroutine in `HandleSessionOnOwner` (`StatePassword` and
  `StateNewAccount` are intercepted before any owner task is submitted).
  A burst of logins therefore costs memory and CPU on connection
  goroutines, never a stall of the owner's command/tick queue. The owner
  task re-checks the stored hash against the one that was verified so a
  password change mid-login cannot authorize a stale attempt, and
  `player.hashLimiter` caps concurrent Argon2id computations at 4.
  All resulting state transitions (echo, session state, account saves)
  still happen on the owner.

Unit tests construct Games without an owner; such Games run submitted tasks
inline on the caller's goroutine (legacy behavior), which is fine for
synthetic tests but production Games always have the owner (started by
`game.New`).

## Admin editor isolation

All `/api/` routes run through one boundary in `internal/admin/owner.go`:

1. Authenticate, read the request body (maximum 8 MiB) on the HTTP goroutine.
2. Submit one synchronous `Game.RunAdminOnOwner` task. The **whole handler**
   runs there: validation against current files, plan construction, file
   commits, runtime migration/eviction, store reloads, and undo history.
3. Write the buffered response on the HTTP goroutine after the task finishes.

Reads use the same boundary so editor API reads cannot observe half an edit.
Neither slow uploads nor slow downloads hold the owner. File I/O does: large
bulk edits can briefly pause gameplay. This deliberately avoids a second
transaction scheduler, fine-grained locks, or optimistic plan retries. Use a
maintenance window for large world reorganizations.

`RunAdminOnOwner` rejects requests during shutdown with HTTP 503, including
requests queued before shutdown but not yet started. It never executes on the
HTTP goroutine after the owner stops. A disconnected request is skipped if it
has not started; once started, its edit completes even if the browser leaves.
A failed/lost HTTP response is not proof that the edit did not commit.

### Endpoint inventory and callback contract

| API family (all behind the same boundary) | Mutations / live dependencies |
| --- | --- |
| `/rooms`, `/rooms/{id}`, `/rooms/new-empty` | Room create/update/delete, room index and runtime room state |
| `/rooms/link`, `/rooms/insert`, `/rooms/remove`, `/rooms/move`, `/rooms/rename`, `/rooms/swap` | Multi-file room plans, course changes, room-ID migration or eviction where applicable |
| `/rooms/bulk-edit`, `/rooms/bulk-delete`, `/rooms/resolve-duplicate` | Bulk room changes, indexes and runtime cleanup |
| `/room-dirs/create`, `/room-dirs/rename` | Room file organization and indexes |
| `/items`, `/mobs`, `/drops`, `/hazards` and their ID routes | Definitions plus existing cache/index invalidation; items also rebuild crafting |
| `/objects`, `/techs`, `/modules`, `/triggers` and their ID routes | Definition files; existing manual-reload semantics remain unchanged |
| Entity `/rename`, `/move`, `/dirs` routes | Definition file organization, undo history, existing cache invalidation |
| `/courses`, `/courses/{id}` | Course files, exit reconciliation, course store reload |
| `/files`, `/files/rename` | Raw file data and history; NOT a semantic room-ID migration or automatic cache reload |
| `/colors`, `/colors/reset` | Persisted config and live color maps |
| `/undo/undo`, `/undo/redo` | File images, room-ID migration, index/cache rebuilds, history together |
| Remaining API routes | Read-only queries; serialized to get coherent snapshots |

Register new endpoints on `apiMux`, never outside this boundary. Do not wrap
individual callbacks in `RunOnOwner`: they are **already on the owner**.
`cmd/thoi/main.go` wires direct course reload, craft rebuild, casino eviction,
and `MigrateRoomIDsOnOwner` callbacks. `UndoStack`'s mutex protects history,
not gameplay: its apply path and the following rebuild must remain in the
same enclosing API task. Login password rehash saves and history clearing
also run on the owner, with expensive password hashing outside it.

This guarantees **isolation from gameplay**, not new crash-atomicity or
rollback guarantees for every editor operation. Existing file transaction
and error behavior is unchanged. Raw Files edits and external filesystem
writes still bypass semantic validation/migrations; use structured room
operations for live renumbering, and stop the game for external bulk edits.
Some definition editors still need the in-game `reload` command; serialization
does not add missing cache invalidations or migrate existing spawned entities.

## Durable money (single-file wallet)

Money lives in the character file and **nowhere else**. Every casino money
mutation is one atomic `AccountStore.SaveCharacter` (`persistence.WriteFile`:
temp file → fsync → rename → directory fsync) that carries both the balance
and the `Player.PendingWagers` ledger. There is no second durable store, so
there is no cross-file crash window to reconcile.

- **Wager** — debits chips then credits and appends a `PendingWager` (its
  wallet handle plus the exact chips/credits debited) in the SAME save. The
  entry stays until the round resolves.
- **Settlement** — `PayCredits(amount, stakeIDs...)` credits the payout and
  removes the listed stakes in the same save. A table passes the hand's stake
  IDs (its payout includes their return); a slot win passes its triggering
  stake (its payout is winnings only). Either way the credit and the stake
  consumption are atomic, so a crash can never pay the win and refund the
  stake.
- **Loss/forfeit** — `ResolveStake`/`ResolveStakes` consume the pending
  stakes in one save; the round resolved and the stake is gone.
- **Refund** — `RefundOutstanding` returns every pending stake (chips in
  kind, credits as credits) and clears the ledger in one save. Graceful
  shutdown (`Game.runShutdown` → `Casino.RefundAll`) and eviction use it.

A save failure **before replacement** reverts the in-memory balance and
ledger together. On `persistence.ErrDurabilityUncertain`, every wallet
operation (wager, payout, stake resolution, refund) instead keeps the applied
balance and ledger, because the visible file already carries both. The wallet
logs the storage fault and reports the operation as applied so callers do not
retry a payout or refund a resolved stake. This does not promise survival of
a subsequent OS/power crash; it prevents a later save from overwriting the
visible result with reverted memory.

At startup (`game.New` → `recoverPendingWagers`, `internal/game/money.go`)
any stake still listed in a character's `PendingWagers` is refunded: a listed
stake means the process died before its round resolved. That is the entire
recovery story — no ordering, checkpoints, evidence, or operator queue.

### Two-phase journal recovery is out of scope

This game deliberately does **not** implement a durable money journal or
two-phase crash recovery. That design was built and then removed: it required
a second file (`data/economy/journal.jsonl`) plus balance checkpoints, record
ordering (`Seq`), payout `Covers` linkage, `open_payouts` evidence, and a
startup recovery pass with per-record ambiguity handling. Its entire purpose
was to make an ungraceful stop mid-round produce zero credit discrepancies,
at a cost of ~1,400 lines of engine code and ~2,000 lines of tests.

With the crash-consistency bar deliberately lowered to "a rare ungraceful
stop may lose or refund one in-flight stake, and must never double-pay", the
single atomic character save is sufficient: a listed pending stake is
refunded, and a settlement whose save fails before replacement leaves the
stake pending and the win unannounced (never paid on top of the refund).
Post-replacement uncertainty follows the applied-state policy above. Do not reintroduce
a second money store, per-record ordering, or an operator-inspection queue
without an explicit decision to restore bank-grade guarantees.

Currency arithmetic is checked (`internal/casino/credits.go`): explicit
`MaxCredits` cap, bounded `DefaultMaxBet` default (the explicit unlimited
sentinel is still honored), floor conversion for payout multipliers, and
overflow rejected instead of wrapping. The cap is shared by the wallet:
bank withdrawals are rejected when they would push the on-hand balance
past it, and gameplay credit grants (shop sales, pickups, activity
payouts) clamp at the cap and log rather than wrap.

## Account lifecycle transactions

Account and character lifecycle operations (create, rename, delete, purge)
are single journal transactions (`internal/persistence/txn.go`): a journal
entry with before-images is written durably BEFORE any file changes, the
operations apply, and the journal is removed at the commit point. Deletes
sync their parent directory; moves sync both affected directories (once if
they are the same). Rollback also syncs deletions, including retries where
the file is already absent, before retiring the journal. An apply failure
attempts rollback; if rollback fails, the journal stays for recovery. A crash
mid-commit is rolled back at the next startup by `persistence.Recover`.

Characters carry a stable UUID (`player.Player.ID`) and owner field
(`player.Player.Account`), assigned at creation and backfilled at first
startup. Cached membership authorizes by the character file's owner, not by
its renameable filename, so a stale cached name can never authorize
connecting, renaming, or deleting a character reassigned to another account.

After any lifecycle mutation, `Game.publishAccountUpdate` republishes the
account state to every authenticated session of that account — including
sessions sitting in menus — resetting stale pending selections. The online
check and the file mutation run as one owner step, closing the
check-then-rename gap.

## Graceful shutdown

Signals (SIGINT/SIGTERM) and the in-game `shutdown` command converge on
`Game.RequestShutdown`, which runs the sequence exactly once:

1. **Admissions stop:** session input and disconnect forfeit/save callbacks
   become no-ops (server-initiated shutdown is not a player disconnect).
2. **Tick stop and join:** `Engine.StopAndWait` stops the ticker and waits
   for the in-flight callback to finish.
3. **Drain and persist (on the owner):** remaining queued commands execute,
   outstanding wagers are refunded (`Casino.RefundAll`), and every online
   character is saved. Save failures are logged for the operator.
4. **Owner stop:** the owner loop drains and stops.
5. **Transports close:** HTTP(S) servers drain, listeners close, and
   sessions flush their outbound queues before their sockets drop.
   `ListenAndServe` returns and main exits normally — no `os.Exit`.

## Room-ID migration

Live room-ID changes (admin rename, swapid) execute as one mutation plan:
the renamed room's file (id field, filename, self-loop references) plus every
other room whose references moved, applied as a single journal transaction,
followed within the same owner task by the full runtime migration
(`Game.MigrateRoomIDsOnOwner` for the editor): player
state, Hub room buckets, casino room keys, world room-keyed state (ground
items, seeded markers, object states), and live mob instances. Undo/redo
store the whole plan (all before/after images plus the idMap) and apply the
inverse/forward runtime migration through the same wired callback.