Chapter 8: Worker Instances and the Spine/Worker Architecture

Worker Instances and the Spine/Worker Architecture

Source: chandra-worker-diagnostics.html; chandra-instance-dashboard.html (new tool, v31c30); chdev-reload.lisp, and chandra/instances/{9101,9102,9103,9104,9195}/ (start-enterprise.lisp, run-ent.sh) from the source zip.

A Chandra deployment is physically realized as one or more worker instances — independent OS processes, each with its own AllegroCache database, each serving one physical isolation surface (Chapter 5) that the Spine Editor (Chapter 13) packed one or more logical Levels (Chapter 4) into.

One process, one database, one instance directory

Confirmed directly from the source zip rather than inferred from the UI: each worker instance lives in its own directory under chandra/instances/<port>/ — five instances were present in the reviewed snapshot, at ports 9101, 9102, 9103, 9104, and 9195. Each instance directory carries its own start-enterprise.lisp, run-ent.sh, and its own chandra-db-<port>/ AllegroCache database directory.

chdev-reload.lisp confirms the database path is configuration-driven, not hardcoded: *chandra-db-dir* defaults to "chandra-db/" but is overridable per-process via chandra.conf's :db-dir key, and start-enterprise.lisp resolves its own source root and instance root independently (CHANDRA_SOURCE_ROOT / CHANDRA_INSTANCE_ROOT environment variables, falling back to the process's own load-path and working directory) — meaning shared source code can be deployed once while each instance still gets its own mutable, isolated instance state. This is the literal mechanism behind Chapter 5's "T cancels out of S*" result: scaling out is adding more of these independent instance directories at the same size, not growing one instance's database.

What NOT to do to a running instance's database — a documented incident

start-enterprise.lisp's own comments preserve a real incident worth restating here, because it's a sharp, specific operational warning rather than generic caution: an earlier version of the boot sequence closed and reopened the AllegroCache database on every boot, on the theory that AllegroCache might have its own thread-level staleness independent of AllegroServe's worker pool. That theory may have been right, but the fix was not safe — after a close/reopen cycle, HTTP requests to certain routes began failing with an AllegroCache internal error (BACKEND-LAST-TRANS with args (NIL)), even though direct REPL reads confirmed the underlying data was intact. The corruption was written to the database's on-disk structures during the close/reopen, persisted across a full process restart, and required deleting and recreating the database to recover. The current boot sequence's explicit comment: "Do not reintroduce AllegroCache close/reopen here without real documentation or vendor guidance on this." If you are ever debugging instance staleness, this specific fix is not the answer, and the comment exists specifically so nobody has to relearn this the hard way.

Instance identity: /api/enterprise/self

Every worker instance exposes its own identity manifest at /api/enterprise/self (documented fully in Chapter 9) — instance name, immutable ID, environment, build badge, and (as of a recent addition) personality_id, the clean identifier of whichever personality (Chapter 6) seeded that instance's spine (e.g., "cmmc-defense-7level" rather than a raw filename). This one endpoint is the shared foundation for three separate tools:

• Worker Diagnostics (chandra-worker-diagnostics.html) — local, non-polling diagnostic surface; explicitly does not depend on CEE or the spine instance being reachable, so it works even when the instance it's diagnosing is unhealthy.
• Instance Dashboard (chandra-instance-dashboard.html, new as of this session, build v31c30) — a per-worker dashboard whose title literally renders as INSTANCE DASHBOARD — <instance_name>, with the name pulled live from the same self manifest. The browser tab title becomes CHANDRA · Instance Dashboard — <INSTANCE_NAME> — meaning if you have several worker dashboards open in different tabs, each tab is self-identifying without needing to check the URL.
• Marshaller (Chapter 7) — reads the exact same manifest remotely, at VERIFY time, to populate its fleet directory.

Why workers are the physical realization of isolation surfaces

Chapter 5 establishes the math (S* is population-independent; scale by adding boundaries, not by growing one). Chapter 6 establishes that the Industry Configurator never touches physical deployment at all. Chapter 13 establishes that the Spine Editor is the tool that actually computes instance count and packs Levels into instances. This chapter is where that math becomes concrete: a "worker instance" is one of Chapter 5's isolation boundaries, realized as an actual OS process with its own database, its own acache (Chapter 18), and its own identity manifest. When this manual talks about an isolation surface being "sized," it means, physically, deciding how many of these worker-instance directories to stand up and which logical Levels each one hosts.

Known incomplete feature: Andon

The Spine UI (visible live on vatican.chandrahub.net) includes an "ANDON ALL" control alongside its dashboard. Per James directly: this functionality is not fully developed yet. Don't document Andon as a working operational feature in this or any other chapter until that's confirmed otherwise — it should stay a named-but-unelaborated control until there's real behavior behind it to describe.