# Iron AOL — Architecture > **Note.** This is the architecture document from the Iron AOL project, published here > as part of a public overview. The implementation itself is not public yet, so filenames > such as `inbox_node.py` appear as plain code spans rather than links — they name real > modules in the private tree. Everything else is the document as written. This document is the map. It explains what the system is made of, why each piece exists, and how a request travels through it. Every layer here has a deeper page under `docs/architecture/`; this page is the one you read first. Jargon is defined the first time it appears. If a term is used without a definition, that is a bug in this document — please open an issue. --- ## 1. What Iron is Iron AOL (Autonomous Operating Layer) is a runtime for **teams of AI agents that are declared as files instead of written as code**. You describe an agent — what it knows, which tools it may touch, what memory it may read and write — in a folder of markdown files. Iron compiles that folder into a runnable process. The agent starts when work arrives, does the work, and exits. Everything it did is written to an append-only log that nobody can edit afterwards. Three things make this different from the usual "agent framework": **Agents are separate operating-system processes, not objects in a loop.** Iron runs on ROS 2, the robotics middleware. An agent is a *lifecycle node* — a process the system can start, configure, activate, deactivate and kill from outside. If one agent crashes, nothing else notices. If the reasoning engine crashes, the message bus and the job runners keep going. **Long work survives restarts.** A task that takes three days and needs a human to approve step 7 is not a Python function holding an open socket. It is a sequence of journaled steps on a durable log. Kill the machine mid-task; when it comes back, the task resumes at the step after the last one that finished. **Everything is written down before it happens.** Every tool call, every fact learned, every irreversible action, every routing decision is an event on an append-only log with a computed identifier. The system's memory is *derived* from that log, so memory can always be checked against it. The reference deployment is a **newsroom**: agents that watch official government and news sources, cross-check claims across independent sources, and write bulletins. See `aem/newsroom/`. --- ## 2. The rules the design obeys These are load-bearing. A change that breaks one of them is a redesign, not a patch. 1. **One interface, swappable backends.** Where something is stored never leaks into the code that reads or writes it. The event log started as a SQLite table behind `emit()`; it is now NATS JetStream; no caller changed. 2. **Capture is dumb, complete and immediate. Derivation is smart, selective and delayed.** Writing down *what happened* must never be skipped, filtered or blocked on an LLM call. Turning that into knowledge costs money and judgment, so it happens later, in the background, and is allowed to be incomplete. 3. **Declaration lives in the agent folder; mechanism lives in the engine.** `aem/` says *what* an agent is allowed to do. `src/iron_core/` implements *how*. Neither reaches into the other. 4. **Fail closed.** An unrecognised memory kind kills the build. An unknown tool verb is treated as destructive. An undeclared source is invisible. The safe default is always "no". 5. **The declaration is validated when a human is reading the error**, i.e. at build time, not at 3 a.m. inside a running agent. 6. **Two planes, never mixed.** Control messages (small, request/response, right now) go over ROS 2. Durable events and streams (large, ordered, replayable) go over NATS. See §4. --- ## 3. The layers | # | Layer | Code | One line | |---|---|---|---| | 1 | Spine | ROS 2 + `rmw_zenoh`, NATS JetStream | how processes talk and how history is kept | | 2 | Cognition | `cognition/` | receive, route, spawn an expert, reply | | 3 | AEM mesh | `aem/`, `aem/catalog_builder.py` | agents declared as folders, compiled to a catalog | | 4 | Tools | `mcp/` | what an agent can *do*, and the two gates on it | | 5 | Perception | `msp/` | what an agent can *see*, streamed in | | 6 | Memory | `memory/` | the log, and everything derived from it | | 7 | Jobs | `jobs/` | long work that survives restarts | | 8 | Workspace | `workspace/` | letting an agent change files, safely | | 9 | Console | `view/` | watching and steering it live | --- ## 4. Layer 1 — the spine Iron uses two messaging systems on purpose, because two genuinely different jobs are being done. ### The control plane — ROS 2 with Zenoh **ROS 2** is the Robot Operating System: a middleware for making many independent processes cooperate. Iron uses three of its ideas: - **Topics** — named channels a process publishes to and others subscribe to. `/iron/raw_inbox` carries incoming messages from any channel. - **Actions** — a long request with progress and a result, which the caller can cancel. Delegating a task to an expert is an action, defined in `AolTask.action`. - **Lifecycle nodes** — processes with an explicit state machine (unconfigured → inactive → active). The system starts, configures, activates, and shuts down an expert from the *outside*; the expert does not manage its own life. **Zenoh** (`rmw_zenoh_cpp`) replaces ROS 2's default DDS transport. DDS discovers peers by flooding the local network with multicast; with dozens of short-lived expert processes appearing and disappearing that becomes the bottleneck. Zenoh routes through a broker (`rmw_zenohd`, started first in the launch file) and works across networks without tunnelling tricks. See `docs/decisions/0002-zenoh-over-dds.md`. ### The durable plane — NATS JetStream **NATS** is a message broker; **JetStream** is its persistence layer: messages are written to disk, kept in order, given a sequence number, and can be re-read from any point later. Two streams are declared in code, in `stream_config.py`: | Stream | Subjects | Holds | Duplicate window | |---|---|---|---| | `MEM` | `mem.>` | the memory ledger — chat turns, tool calls, facts | 120 s | | `JOBS` | `jobs.>` | the durable-jobs ledger — one subject per job | 900 s | Both are declared `deny_delete` and `deny_purge`: **the log cannot be edited or erased through the normal API.** That is the whole point — derived memory is checked against it. `bootstrap()` creates each stream if it is missing and verifies it field by field if it exists. On any difference it **refuses to start** rather than silently reshaping a live stream. ### Why the split matters A message that must arrive *now* and a message that must survive *forever* have opposite requirements. Mixing them means either your control loop pays disk latency, or your audit trail is lossy. Iron pays for two systems to avoid choosing. --- ## 5. Layer 2 — cognition Three processes, in order of how much they think. ### The inbox — `inbox_node.py` Zero LLM calls. It subscribes to `/iron/raw_inbox`, keeps a table mapping `source:user_id` to a session, and forwards the message as an action goal. It exists so that *routing does not cost money or latency*. Deciding which conversation a Telegram message belongs to is bookkeeping, not reasoning. It also means the system still accepts and queues input when the reasoning layer is down. The inbox can **stick** a session to one expert: while a user is mid-conversation with the finance expert, their next message goes straight there without re-routing. Cognition controls that through `/iron/session_control`. ### The engine — `cognition_engine_node.py` The router. It reads `catalog.json` to see which experts exist, then uses a small, cheap model (`gemini-3.1-flash-lite` by default) to split an instruction into `(domain, cleaned_instruction)` pairs and hand each to an expert. Before routing it pulls context: recent turns from the session store, relevant facts, and a scorecard of how each expert has performed. After routing it emits `routing_decided`, and later `task_judged` — a second pass asking whether the result actually answered the question. Replying does **not** involve a model. The reply channel is declared as data in the mesh file: ```yaml delivery: telegram: server: telegram tool: send_telegram_text args: { chat_id: "{dest}", text: "{text}" } ``` Adding Slack is a block here plus a tool server. It is never a code change. ### The expert — `expert_action_node.py` One process per task, spawned by `delegation.py`, killed when idle. At startup it reads its own entry from `catalog.json` and builds itself: the tools it was granted, the memory planes it declared, the sources it may read. **Experts are disposable, and that is a design choice, not a limitation.** Because `expert_finance_t1a2b3` dies in five minutes, its memory *cannot* live inside the process. Memory is keyed by **role**, stored centrally, and mounted at spawn. That is what makes an expert portable: the folder is the agent, the process is just a lease. Guards in `delegation.py`, all of them learned from failures: - `MAX_DEPTH = 4` — an expert delegating to an expert delegating to… stops here. - `MAX_CONCURRENT_DELEGATIONS = 5` — bounded fan-out. - Every cross-process wait has a timeout, because a freshly-spawned process can drop its first response before discovery settles. A lost message must fail cleanly, never hang the engine forever. An expert can also do two things mid-task that most frameworks cannot: **interject** (an operator injects a correction into a running turn, `interjection.py`) and **ask** (the expert stops and asks the operator a question, `question.py`). --- ## 6. Layer 3 — the AEM mesh **AEM** = Agent Expert Mesh. It is how you add an agent without writing code. ### The shape ```text aem// AEM.md the mesh: defaults, tool registry, delegation, delivery skills// SKILL.md who this expert is + what memory it declares (portable) capabilities.md what THIS deployment grants it (local) flows/*.py deterministic pipelines it owns tests/*.py tests for those flows ``` A **mesh** is a team with a shared namespace. Five exist: `core` (general purpose), `dev` (software engineering), `newsroom` (the flagship), `erp` and `sim` (parked). ### Why two files per expert This split is the reason an expert can be shipped between deployments: - `SKILL.md` is **portable**. It holds identity, instructions and the *shape* of the memory the expert needs. It says "I am a verification analyst and I need read access to episodic memory." It contains nothing about your machine. - `capabilities.md` is **local**. It holds what your deployment actually grants: which tool servers, which live sources, which peers, at what clearance. It is `add`/`remove` deltas against the mesh defaults. Ship `SKILL.md`. Never ship `capabilities.md`, and never ship the facts the expert learned about your tenant — that is a data leak, and it is enforced by type, not by convention. ### Delegation is mesh-scoped ```yaml delegation: scope: all ``` Every expert in a mesh may delegate to every other expert in that mesh. No per-expert allow-lists to maintain — that is what "mesh" means. Exceptions go in one expert's `capabilities.md`. ### The compiler `catalog_builder.py` is the **only** reader of the `aem/` tree. It resolves defaults, unions the mesh's global tools with each expert's local ones, validates every memory declaration and every clearance, and writes `catalog.json`. It fails the build loudly on: an unknown memory kind, a memory kind that is in the menu but not yet implemented, an unknown clearance, a malformed server descriptor, an over-reaching effect declaration. **A typo must never ship as silent no-memory.** --- ## 7. Layer 4 — tools and the two gates A **tool** is something an agent can call: read a file, search the web, send a message. Iron speaks **MCP** (Model Context Protocol), the standard for exposing tools to language models, so most of the ecosystem's servers work unchanged. Three transports, declared per server: | Transport | Meaning | |---|---| | `local` | start an MCP server as a subprocess, on demand, in the expert's own process | | `native` | a tool that is itself a ROS node (e.g. Telegram) | | `remote` | a networked tool reached over ROS | Tools used to be spawned as bridge nodes at boot. They are now started in-process by `tool_manager.py`, on demand, per expert. The expert reads its tool roster from `tool_manifest.json` — a pre-built index — instead of listing tools live on every task. ### Gate 1: clearance — *may this agent hold a tool this powerful?* `tool_filter.py` derives what a tool **does** from the verb anywhere in its name, and compares it to the expert's clearance word. | Effect | Example verbs | Required clearance | |---|---|---| | `read` | get, list, search, fetch, describe | `read` | | `write` | create, update, send, apply, upload | `read-write` | | `destructive` | delete, drop, purge, **run, exec, shell, query** | `full` | Three deliberate decisions in that table: - **No verb recognised → destructive.** Fail closed. - **`query` is destructive, not read.** A SQL `query` can `DROP TABLE`. - **A server's own annotations may raise a tool's effect but never lower it.** An MCP server is an untrusted subprocess; it must not be able to declare its own destructive tool "read-only". A *human-written* declaration in `capabilities.md` **may** lower an effect, because git reviewed it. That is the only way to grant a confined command runner to an expert who must not have a real shell — and `catalog_builder` caps how far it can be lowered. ### Gate 2: target — *may it point that tool at THIS path?* `path_guard.py` exists to close one specific hole: **self-escalation**. A `read-write` engineer legitimately holds `edit_file`. Without this gate, the first thing it could edit is the `capabilities.md` that says what it may edit. The path guard makes governance files unwritable regardless of clearance. Two gates, two different questions. You need both. --- ## 8. Layer 5 — perception (MSP) **MSP is not MCP, and the difference is the direction of travel.** - MCP registers tools the model **calls out to**. The model decides. - MSP streams a source's **current state into** the model at wake. The model does not ask; it simply arrives already knowing. An expert declares MSP sources; at wake, `msp/client.py` reads each one's composed state and puts it in the prompt. Read-only, no held connection. This exists because "what is the current state of the world" is a terrible fit for a tool call. A tool call is a decision, and the model has to *decide* to check — which it often will not. Sources are registered in the mesh file and granted per expert: ```yaml msp: sources: acled: startup: on_demand command: "uvx acled-msp --country India --since 2026-07-01" working_memory: driver: feldera relations: [acled_events, fatalities_by_admin1] ``` Registering a source grants it to **nobody**. An expert opts in by name in its `capabilities.md`. Sources are discovered by name at use time through a NATS-KV phone book (`msp/discovery.py`), so they can be started on any free port. **Feldera** is the live working-memory engine: an incremental view maintainer. Instead of re-scanning a feed to answer "fatalities by region", it keeps that answer continuously up to date as records arrive. --- ## 9. Layer 6 — memory The largest subsystem: 22 modules. The shape is one rule applied consistently. > **Write everything, immediately and stupidly. Understand it later, selectively > and expensively.** ### The write path Anything worth remembering calls `emit()` (`memory_plane.py`). It publishes an envelope to `mem.` on the `MEM` stream: ```text { event_id, ts, tenant, namespace: [...], actor, kind, payload, result, cause } ``` `namespace` is a path — `["facts", "newsroom", "analyst"]`. Every token is validated; a token containing `.`, `*`, `>` or a space is rejected, because it would corrupt the subject hierarchy. `cause` points at the event that caused this one, so any state is traceable backwards. `emit()` is synchronous and never filtered. It is allowed to be boring. ### The read path `projector.py` is the single writer to `memory/projection.db`, a SQLite database with `event_id` as primary key. It consumes the stream and materialises queryable tables. Because the identifier is the primary key and JetStream has a duplicate window, **the same event delivered twice writes once.** Readers open that database read-only. `memory_recall.py` does full-text search over it, plus `replay(namespace_prefix)` — ordered history for one scope, which is both the audit view and the rebuild view. ### The background workers Each is its own process in the launch file; each turns raw events into something more useful: | Process | Job | |---|---| | `extractor` | pull durable facts out of raw episodes | | `consolidator` | compress raw episodes into summaries | | `knowledge_graph` | project facts as entities and relationships into Neo4j | | `semantic_vector_indexer` | embed facts into Qdrant for meaning-based search | | `procedural_vector_indexer` | the same, for learned procedures | | `document_indexer` | extract text from PDF/DOCX/XLSX in MinIO, chunk it, index it | | `procedural_reviewer` | gate learned procedures before they can be reused | | `memory_injector` | put the right memory into the right prompt at spawn | | `nats_kv` | short-lived scratchpad, TTL-expiring | ### The two axes Memory is indexed on two independent axes, not one list of stores. **By type — what job it does:** | Type | Answers | Written by | |---|---|---| | working | "what is true in this task right now?" | the agent, every step | | episodic | "what happened, and when?" | dumb capture | | semantic | "what is true in general?" | the extractor, *derived* | | procedural | "how do I do this — what worked before?" | the consolidator, *derived* | | identity | "who is this expert?" | declared in `SKILL.md` | **By tier — how fast it is to reach:** in-context (what is in the prompt now) → hot external (recent, indexed) → cold archival (full history). Semantic and procedural memory are never written directly. They are **distilled from episodic memory in the background**. That distillation is the part most systems skip. ### Scopes Scope is a **namespace prefix on one store**, never a separate store: ```text own -> facts// + facts/system mesh -> facts/ + facts/system system -> facts (everything) ``` **No declaration means no memory.** That is deliberate — it is the wall that stops every expert from drowning in every other expert's context. Formats are not stores. Which physical backend an expert's memory lands in is declared per deployment, never hard-wired. --- ## 10. Layer 7 — durable jobs Some work does not fit in a request. "Watch this feed, and when something breaks, corroborate it across sources, then ask a human, then publish" runs for days and must survive a reboot. ### Flows A **flow** is an ordinary synchronous Python function, `run(ctx, inp)`, living in an expert's `flows/` directory. It obeys five rules, the important two being: 1. Everything non-deterministic goes inside `ctx.step(...)`. 2. Code between steps computes only from `inp` and earlier step results. Then the runtime can replay the function from the top after a restart, skip every step already in the journal, and land exactly where it stopped. ### The journal Each job has a diary — one NATS subject: ```text jobs..evt.. the diary (read, never consumed) jobs..wake.. the queue (consumed and acknowledged) ``` Splitting them is what lets the runner's consumer see only wakes and never its own writes. Every event id is **computed, not random** (`events.py`), so publishing the same logical event twice produces the same id and JetStream discards the second. ### Control flow A flow leaves the runtime in exactly one of three ways (`runtime.py`): | Exit | Meaning | |---|---| | `return` | finished | | `raise Waiting` | sleeping at zero cost — a timer, an approval, or a signal | | `raise Parked` | dead-lettered — a human must `/resume` or `/cancel` | `Waiting` is the important one. A job waiting three days for approval consumes no process, no thread and no connection. It is a row. ### Step fingerprinting `ctx.step` skips a step whose name is already in the journal. Keyed on the name alone that is a silent correctness hole: edit a flow while one of its jobs is in flight, and on resume you get back a result recorded for a *different* call. So each step hashes its actual call — name, function, arguments. On a mismatch the job **parks** rather than reusing a wrong result. Journals written before this feature carry no hash and are trusted, so deploying it parked nothing already running. ### The supporting cast - **runner** — one per mesh, consumes the wake queue. Holds a 30-second KV lease so two runners can never double-run a job; it deliberately dies rather than risk that, and respawns once the lease lapses. - **clock** — turns flow timers and due-waits into `timer_due` events, with deterministic ids from `(timer, tick)`. - **expert_gateway** — the one process speaking both NATS and ROS. When a flow needs judgment it raises a durable wait; the gateway spawns the expert and signals the answer back. This keeps runners entirely ROS-free. - **keep_awake** — inhibits system sleep while steps are running. ### The confirmation gate Two event kinds guard irreversible actions: - `effect_intent` — written **before** the action fires. - `effect_confirmed` — written only after a *fresh, independent observation* shows the world actually changed. Write-ahead, then verify. A crash between the two leaves evidence that something may have happened, which is the honest state to be in. --- ## 11. Layer 8 — the workspace sandbox For agents that change files, three layers of containment. **A throwaway workspace.** `scratch.py` issues a disposable working directory; the default provider (`git_worktree.py`) makes it a linked git worktree. Nothing runs in the real tree. **OS-enforced confinement.** `sandbox.py` runs commands under **bubblewrap**: writes outside the scratch are blocked, network is denied by default, credential stores are masked, and the process gets its own PID namespace. Verified by writing the escape attempts — absolute-path write, `..` traversal, `$HOME` write — as tests. Two limits, both documented rather than hidden: - **Read isolation is partial.** `/` is mounted read-only, so a command can still read most of the filesystem. Named secret paths are masked. Tightening this breaks real toolchains. - **No CPU or memory limit yet.** A runaway build is bounded only by the timeout. A cgroup v2 limit is the next increment. On macOS and Windows, `sandbox.available()` returns false and confined execution **refuses to run**. That is the intended behaviour, not a gap. **Human approval per change.** `hunks.py` parses a diff into hunks and attributes each one; `apply.py` applies only approved hunks and proves the result. The operator approves changes one hunk at a time in the console. --- ## 12. Layer 9 — observability and the console ### Shipping output ROS 2's `output='screen'` sends a process's output to the launch terminal and nowhere else. So the launch file wraps every process in `obs/ship.py`, which tees output byte-for-byte to the terminal **and** publishes it to `iron.log.