How to design, ship, and operate an AI agent that is reliable, efficient, performant, scalable, and secure enough to serve real companies — from a 5-person startup to a 50,000-person enterprise.

This guide distills hard-won lessons from production agents (Claude Code, OpenHands, SWE-agent, GoClaw, Hermes, nanobot, PicoClaw, ZeroClaw, Multica, Paperclip) and grounds them in current engineering guidance from Anthropic and OpenAI plus the security and compliance standards you'll actually be audited against (OWASP Top 10 for Agentic Applications, NIST AI RMF, the EU AI Act, and 2025–2026 prompt-injection research). It focuses on the parts most articles skip: the enterprise tax — governance, security, compliance, integration, cost control, and the operating model — that separates a demo from a system a CISO will sign off on.

📖 How to use this guide

  • Read Parts 0–2 to decide whether and what to build. Most failed agent projects die here.
  • Read Parts 3–7 for the architecture and reliability engineering.
  • Read Parts 8–10 for the enterprise gates: security, compliance, multi-tenancy, observability, cost.
  • Read Parts 11–15 for delivery, scale & rollout: deployment topologies (SaaS/self-hosted/hybrid), how to adopt from pilot to org-wide, how to handle thousands of concurrent requests, the operating model, and a 30/60/90 plan.
  • Every part ends with an ✅ Actionable checklist. Skim those for a design review.

📋 Table of Contents

🧮 Part 0 — The Core Equation

The single most important idea in agent engineering:

Reliability  ≈  Model capability  ×  Harness quality
                    (mostly fixed)      (your job)

Enter fullscreen mode Exit fullscreen mode

The model is roughly fixed for the life of your project. The harness — system prompts, tools, sandboxes, memory, orchestration, guardrails, and observability — is where ~80% of production quality comes from. After hundreds of production sessions the pattern is consistent:

It's almost never a model problem. It's a configuration and harness problem.

For enterprise, add a second equation that most teams discover too late:

Enterprise-readiness  ≈  Harness quality  ×  Trust surface
                                              (security + governance + observability)

Enter fullscreen mode Exit fullscreen mode

A brilliant agent that can't prove what it did, can't be scoped to a tenant, and can't be audited will not ship in a regulated company. Budget for the trust surface from day one — it is not a phase 2 feature.

🧭 Part 1 — Decide Before You Build

1.1 Workflow or Agent?

Anthropic's guidance (Building Effective Agents, 2024) draws the line that matters:

Workflow Agent
Control flow Predefined code paths LLM directs its own steps
Best for Well-defined, decomposable tasks Open-ended tasks, unknown # of steps
Cost/latency Low, predictable Higher, variable
Failure mode Predictable Compounding errors

Rule: start with the simplest thing that works — a single well-prompted LLM call with retrieval often beats an agent. Add agentic autonomy only when the number of steps is genuinely unpredictable (e.g. coding, research, multi-system triage). Autonomy trades latency and cost for capability; make that trade deliberately.

The common production patterns, in rising order of complexity: augmented LLM → prompt chaining → routing → parallelization → orchestrator-workers → evaluator-optimizer → autonomous agent. Reach for the lowest rung that solves the problem.

For fixed business processes, make the control flow deterministic. Expense approvals, employee onboarding, KYC, and refund flows have known steps — encode them as an explicit state machine / durable workflow (e.g. LangGraph for the graph, Temporal for durable execution) and let the LLM be flexible only inside a bounded sub-task ("draft the summary," "classify this ticket"). This is the single most effective cure for the runaway-reasoning-loop failure in corporate settings: the agent literally cannot wander outside the defined transitions. Reserve open-ended autonomy for the genuinely unpredictable work. (Budgets, stuck detection, and circuit breakers in Part 4 and Part 7 back this up — a state machine bounds what can happen, budgets bound how long.)

1.2 Build vs Buy vs Assemble

Path When it's right Watch out for
Buy a SaaS agent Commodity use case (support deflection, meeting notes) Data residency, lock-in, no access to the harness
Assemble on a platform/SDK You want control of the harness but not the kernel Framework abstraction hiding prompts/tokens — insist you can see them
Build the harness on raw LLM APIs Differentiated workflow, strict data/compliance needs Cost of the "last mile" to production is large

Anthropic's advice holds: frameworks help you start but "reduce abstraction layers and build with basic components as you move to production." If a framework hides the prompts and token flow, you can't debug or cost-control it — a dealbreaker at scale.

1.3 Qualify the use case with 5 questions

A use case is a good agent fit when you can answer yes to most of these:

  1. Verifiable success? Can you check the outcome (tests pass, ticket resolved, invoice matched)?
  2. Feedback loop? Does the environment give ground truth each step (tool results, errors)?
  3. High enough value? Agents use ~4× the tokens of a chat; multi-agent ~15× (Anthropic). The task must be worth it.
  4. Tolerable blast radius? What's the worst a wrong action does? Scope permissions to that.
  5. Human oversight fits naturally? Support, coding, and ops all have obvious review points.

✅ Part 1 checklist

  • [ ] Chose the lowest rung (workflow before agent) that solves the problem
  • [ ] Wrote down the success metric and how it's measured automatically
  • [ ] Ran a build/buy/assemble decision with data-residency constraints included
  • [ ] Estimated cost-per-task and confirmed the task value exceeds it

🏛️ Part 2 — The Enterprise Tax

A consumer demo becomes an enterprise product when it satisfies requirements that have nothing to do with the model. Plan for these before the pilot, because retrofitting them is expensive.

Requirement What it means concretely
Identity & access SSO (SAML/OIDC), SCIM provisioning, role-based access to tools and data
Multi-tenancy Hard isolation of data, secrets, workspaces, and cost per company/team
Data governance Data residency/region pinning, retention limits, PII handling, "no-train" guarantees
Auditability Every action attributable to a user + reproducible; immutable audit log
Compliance SOC 2 Type II, ISO 27001, GDPR/CCPA, and sector rules (HIPAA, PCI-DSS, FINRA)
Security Prompt-injection defense, secrets isolation, sandboxing, least privilege
Reliability/SLA Uptime targets, graceful degradation, incident response, RTO/RPO
Cost control Per-tenant budgets, rate limits, chargeback/showback, model routing
Observability Tracing, evals, alerting — without logging sensitive conversation content
Change management Versioned prompts/tools, safe rollout, rollback, user training

The mindset shift: in traditional software a bug breaks a feature. In an agent, a minor change cascades — one bad step sends the agent down an entirely different trajectory (Anthropic, Multi-Agent Research System). The enterprise tax is what keeps those cascades observable, bounded, and reversible.

✅ Part 2 checklist

  • [ ] Named the compliance regime(s) you must satisfy and the data classes involved
  • [ ] Confirmed a "no-train / data-isolation" path with your model provider
  • [ ] Decided the tenancy boundary (company / team / user) up front
  • [ ] Made audit logging a P0, not a P2

🏗️ Part 3 — Reference Architecture

Every production agent that works is recognizably the same system: a small reliable kernel loop wrapped in a thoughtfully engineered harness, exposed through thin surface adapters. Here is the enterprise-shaped version.

┌────────────────────────────────────────────────────────────────────┐
│  SURFACES (thin adapters)                                          │
│  Web app · Slack/Teams · IDE · API · Email · Cron/Webhook          │
└───────────────┬────────────────────────────────────────────────────┘
                │  authenticated, per-tenant request
┌───────────────▼────────────────────────────────────────────────────┐
│  GATEWAY / CONTROL PLANE                                           │
│  AuthN (SSO) · AuthZ (RBAC) · rate limit · budget check · routing  │
└───────────────┬────────────────────────────────────────────────────┘
                │
┌───────────────▼────────────────────────────────────────────────────┐
│  AGENT RUNTIME (the kernel)                                        │
│  Loop: Observe → Think → Act → Observe                             │
│  Session state (append-only events) · iteration/cost budgets       │
│  Context engine (cache-stable prefix + compaction)                 │
│  Sub-agent orchestration (context firewalls)                       │
└───┬────────────────┬───────────────┬───────────────┬───────────────┘
    │                │               │               │
┌───▼─────┐    ┌─────▼─────┐   ┌─────▼──────┐   ┌────▼─────────┐
│ TOOLS   │    │  MEMORY   │   │  SANDBOX   │   │  MODEL LAYER │
│ registry│    │ L0/L1/L2  │   │ per-tenant │   │ provider     │
│ + MCP   │    │ + files   │   │ isolation  │   │ abstraction  │
└───┬─────┘    └───────────┘   └────────────┘   └──────────────┘
    │ enterprise connectors (RBAC-scoped, per-tenant secrets)
┌───▼──────────────────────────────────────────────────────────────┐
│  SYSTEMS OF RECORD: DB · CRM · ticketing · data warehouse · APIs │
└──────────────────────────────────────────────────────────────────┘

  Cross-cutting: OBSERVABILITY (tracing, metrics, evals, cost) ·
                 SECURITY (guardrails, secrets, audit) ·
                 GOVERNANCE (policy, HITL approvals)

Enter fullscreen mode Exit fullscreen mode

Design principles that make this scale (from OpenHands V1, Hermes, GoClaw):

  • One loop, many surfaces. A single agent core powers CLI, chat, API, and cron. Surfaces are thin translators, not forks of the logic.
  • Immutable models + append-only state. Agent, tools, and config are immutable; the only mutable thing is ConversationState, which you append events to, never mutate in place. This makes the system replayable, debuggable, auditable, and safe to parallelize.
  • The loop is an async generator, not a web of callbacks — you get backpressure, cancellation, and typed terminal states for free.
  • Control plane is separate from the runtime. Auth, budgets, and routing live in front of the loop so you can enforce policy without touching agent logic.

✅ Part 3 checklist

  • [ ] Kernel is one loop; surfaces are adapters
  • [ ] State is append-only events (replayable/auditable)
  • [ ] Control plane (auth/budget/routing) sits in front of the runtime
  • [ ] Provider access goes through one abstraction, never scattered SDK calls

🔄 Part 4 — The Reliable Kernel

4.1 The loop

All production agents converge on Observe → Think → Act → Observe — 4–5 phases, not a callback web. Keep the kernel small and boring; put cleverness in the harness.

Three proven shapes:

  1. Async generator (OpenHands, Claude Code) — yields each step; caller controls backpressure/cancellation.
  2. Explicit step() returning a typed union (SWE-agent, GoClaw) — a ~30-line forward_with_handling() wraps the model call with requery on format errors (max 3).
  3. Per-session FIFO steering queue (nanobot, PicoClaw) — a user can inject a correction mid-loop; queued-but-unrun tools are skipped with a synthetic "Skipped due to user message" result so the model knows what didn't run.

4.2 The tool_use/tool_result invariant — the #1 correctness bug

Every tool_use must have a paired tool_result before the next model call (API requirement). On cancellation or error, emit a synthetic result ("Cancelled: Bash(mkdir) errored"). OpenHands' runner enforces: drop orphan results, backfill missing ones, and microcompact each iteration. Get this wrong and you get random 400s and corrupted transcripts in production.

4.3 Budgets: stop on cost, not vibes

Battle-tested defaults (Hermes, Claude Code, OpenHands):

Budget Default Why
Max iterations / task 20–25 Bound runaway loops
Per-task cost cap e.g. $2–$3 Cost is the real stop signal — step count varies 5× across models
Max requeries on parse fail 3 Don't loop on malformed output
Consecutive timeouts 5 → hard abort Escape stalls
Context overflow compact at ~80%, then continue Never hit a hard 400

Anthropic's own finding: token usage alone explains ~80% of task-performance variance on hard browse tasks. Budgets aren't just cost control — they're your primary lever on both quality and spend.

✅ Part 4 checklist

  • [ ] Loop is 4–5 phases, kernel < a few hundred lines
  • [ ] Synthetic tool_result emitted on every error/cancel path
  • [ ] Stop conditions are cost-based, with iteration/timeout backstops
  • [ ] Steering queue lets a human correct mid-run

🛠️ Part 5 — Tools & Enterprise Integration

Tools are the agent's hands — and, per Anthropic, you should spend as much effort on the agent-computer interface (ACI) as on the prompt. On SWE-bench they spent more time optimizing tools than the overall prompt.

5.1 Design tools like a great docstring for a junior engineer

  • Poka-yoke (mistake-proof) the inputs. SWE-agent forced absolute file paths after seeing the model fail with relative ones once it changed directories — the fix was flawless thereafter.
  • High-signal outputs only. Return name, image_url (semantic) — not uuid, 256px_image_url (noise). Paginate and cap responses (~25K tokens) by default; steer toward many small searches over one giant dump.
  • Consolidate chained calls. schedule_event (finds availability and books in one call) beats list_users → list_events → create_event. Fewer round-trips = fewer tokens, fewer errors.
  • Instructive errors. A tool error should tell the model how to fix it, not just fail.
  • Response-format enums. Let the agent choose concise vs detailed; concise uses ~⅓ the tokens.

5.2 The registry with three safety gates (GoClaw)

  1. Global profile — read_only / coding / messaging / full.
  2. Per-tool capability metadata — read-only vs mutating, concurrency-safe or not.
  3. Per-invocation safety check on the *parsed input* — Bash("ls") is safe; Bash("rm -rf") is not. Fail closed: if you can't classify it, block or serialize it.

Tools self-register at import time (Hermes, PicoClaw) — no hand-maintained lists that drift.

5.3 MCP: the integration standard for enterprise

The Model Context Protocol (MCP) is now the de-facto open standard ("USB-C for AI") for connecting agents to tools, data, and workflows, supported across Claude, ChatGPT, VS Code, Cursor, and more. For enterprise it gives you:

  • Build once, integrate everywhere — one connector works across clients.
  • A governance seam — you can put allowlists, per-tenant credentials, and audit at the MCP boundary.

Enterprise cautions with MCP: agents encounter unfamiliar tools with wildly varying description quality (Anthropic). Curate an internal MCP catalog: vet each server, standardize descriptions, pin versions, and scope credentials per tenant. Treat a third-party MCP server as untrusted code and network egress — sandbox it. Anthropic even built a tool-testing agent that uses Claude to rewrite weak tool descriptions; the model-optimized definitions beat human-written ones on their internal Slack/Asana evals and helped reach state-of-the-art on SWE-bench Verified — so let the agent improve its own tool docs, then eval-gate the result before promoting it (Anthropic, Writing Effective Tools for AI Agents, 2025).

MCP has matured — build on the standard, don't reinvent it. Two 2025–2026 developments matter for enterprise: (1) the official MCP Registry (launched Sept 2025) is a curated server directory with provenance/ownership metadata — use it (or a private mirror) as the vetting front door to your internal catalog instead of hand-collecting servers; (2) the MCP authorization spec now aligns with OAuth 2.1 / OpenID Connect, adds Enterprise-Managed Authorization (IdP admins grant consent centrally rather than per-user prompt fatigue), and mandates issuer (iss) validation (RFC 9207) to close a "mix-up" attack class inherent to MCP's one-client/many-server shape. Require these of any server you admit.

When the tool count gets large, don't load every schema. A vetted catalog can still be hundreds of tools; putting all their schemas in the prompt bloats context and hurts selection accuracy. Use tool search / progressive tool disclosure — the agent discovers and loads only the relevant definitions per request (Anthropic reports large should-call-rate gains from this), which also keeps the cache-stable prefix intact (Part 6.1) because schemas are appended, not swapped.

5.4 Skills: the compounding asset

Skills are SKILL.md files (YAML frontmatter + markdown procedure) loaded by progressive disclosure: a one-line description in the system prompt, full content on demand, referenced files only when invoked. Agents can write new skills after solving a hard problem (Hermes, Multica). Skills — not prompts — are the durable, reusable, portable asset; every run gets cheaper as the library grows.

✅ Part 5 checklist

  • [ ] Tools are mistake-proofed and return semantic, paginated output
  • [ ] Frequently-chained ops are consolidated into single tools
  • [ ] Registry enforces profile + capability + per-invocation checks (fail closed)
  • [ ] Enterprise systems integrated via a vetted, per-tenant-scoped MCP catalog
  • [ ] A skills library exists and grows from solved problems

🧠 Part 6 — Context & Memory

On agentic workloads, input tokens are ~90% of the bill (roughly a 100:1 input:output ratio). Context engineering is cost engineering.

6.1 Cache stability — the single biggest cost lever

  • Assemble the system prompt once at session start and freeze it. No mid-conversation mutations.
  • Byte-stable prefix + volatile tail. Prefix (system prompt + tool schemas + frozen transcript) is cacheable; the volatile tail (clock, file listings, plan state) is rebuilt each turn and kept out of the cached region.
  • Result in practice (Hermes/Claude Code): ~99.9% of the prefix served from cache at ~0.1× base price. The costliest mistake is breaking the cache by editing the prompt mid-session.

6.2 Compaction — before overflow, not at it

Trigger at ~80% of the input budget. Summarize the oldest ~70% into a typed checkpoint (durable memory, execution summary, preserved requirements, skill refs) and keep the ~4–12 most recent messages verbatim. Offload bulky tool outputs to workspace files — keep head + tail + a path preview and reload on demand. Naive full-transcript replay is O(k²); managed compaction makes it O(k).

6.3 Memory tiers

Tier Contents Loaded via
L0 working current session events in-context (append-only)
L1 episodic session summaries + embeddings, ~90-day retention memory_search tool
L2 semantic knowledge-graph entities/relations, temporal validity memory_expand tool

Start file-based (MEMORY.md, USER.md, history.jsonl) — don't reach for a vector DB until you exceed ~1M tokens of durable knowledge. Read memory at session start, inject it immutably, and let updates take effect next session (frozen-snapshot pattern — Hermes). For long-horizon runs, persist the plan to memory before context truncates, and spawn fresh sub-agents with clean contexts via careful handoffs (Anthropic).

6.4 Retrieval strategy — match the index to the data shape

Most enterprise knowledge lives in systems of record, not the prompt — so retrieval (RAG) quality drives answer quality.

  • Default to hybrid retrieval (vector + keyword/BM25 + reranking). It's cheap, well-understood, and good enough for the majority of document/FAQ/knowledge-base use cases.
  • Use a knowledge graph (Graph-RAG) where the data is genuinely graph-shaped — org charts, entitlements, project→owner→dependency relationships — and the questions are relational ("who owns the services that depend on X?"). Graph-RAG can meaningfully reduce relationship errors there, but it is not a hallucination silver bullet: it's more expensive to build and maintain, and adds no value over hybrid retrieval on flat document corpora. Reach for it deliberately, not by default.
  • Ground the answer either way — cite sources, prefer primary systems of record, and have the agent verify claims against retrieved evidence rather than trusting recall.

Enterprise note: memory is a data-governance and attack surface. Memory poisoning — an attacker getting malicious content written into memory that the agent later reads back as trusted — is a distinct, persistent threat (it's a top-ranked risk in the OWASP Agentic Top 10; unlike a one-shot prompt injection, a poisoned memory keeps misdirecting every future session that loads it). Scope memory per tenant, apply retention limits, attribute every write to an actor, keep an immutable version history so you can audit and redact, and scan memory for injection/exfiltration patterns before injecting it back into the prompt.

✅ Part 6 checklist

  • [ ] Prompt is frozen; prefix is byte-stable and cached
  • [ ] Compaction triggers at ~80%, keeps a live tail, uses a cheaper helper model
  • [ ] Bulky outputs offloaded to files; loaded on demand
  • [ ] Memory is per-tenant, retention-bounded, and injection-scanned

🛟 Part 7 — Reliability Engineering

"In agentic systems, minor issues that would be trivial for traditional software can derail agents entirely." — Anthropic

7.1 Classify failures before you retry

Failure type Response
Rate limit / transient Retry with exponential backoff + jitter
Malformed stream Discard mid-stream cleanly, requery (max 3)
Stall / no progress Timeout; break the pattern
Provider outage Failover to backup provider/model
Permanent (auth, bad request) Surface to human; do not loop

Never silent-retry. Log every retry with its reason. Circuit-break when the model repeats an identical failing call 3× — back off instead of burning budget.

7.2 Stuck detection

Detect repeated identical actions, oscillation between two states, or zero net change over K steps → break the loop. Have the agent track progress in a TODO.md/NOTES.md; an observer watches for no forward motion. Hard stops: max iterations, wall-clock timeout, cost ceiling.

7.3 Durable execution — resume, don't restart

Agents are stateful and errors compound; a restart from scratch is expensive and infuriating. Anthropic combines "the adaptability of the model with deterministic safeguards like retry logic and regular checkpoints," and resumes from where the error occurred. Concretely:

  • Checkpoint state to durable storage (DB + git worktree), not just in-context.
  • Resume tokens / session continuation — --continue reloads history, recaps, and picks up.
  • Autosubmit on failure — capture partial work (git diff) and ship the partial result rather than losing everything.

7.4 Deploy without breaking in-flight agents

Agents are long-running, so a normal deploy can catch them mid-trajectory. Use rainbow deployments: run old and new versions simultaneously and shift traffic gradually, never cutting a running agent over mid-task (Anthropic).

7.5 Provider resilience

One provider abstraction, multiple backends (Anthropic native, OpenAI-compatible, Bedrock/Vertex, CLI subprocess). Layer retry → cooldown → failover chain → cache. Normalize all provider stream formats to one internal shape so vendor JSON differences never leak into your loop. This also protects you from single-vendor outages and price changes — a real enterprise procurement requirement.

✅ Part 7 checklist

  • [ ] Failures are classified; retries are logged, backed off, and circuit-broken
  • [ ] Stuck detection with hard stops
  • [ ] State checkpointed durably; runs resume, not restart
  • [ ] Rainbow (or blue/green) deploys protect in-flight agents
  • [ ] Multi-provider failover behind one abstraction

🔐 Part 8 — Security, Compliance & Governance

This is the part that gets an enterprise deal signed or killed.

8.1 Defense-in-depth (ZeroClaw, GoClaw)

Layer Control
1. Channel allowlist users/chats/IPs before the loop sees input
2. Autonomy coarse mode (read_only/supervised/full) + per-tool overrides
3. Workspace workspace_only=true, forbidden-paths, resolve symlinks before enforcing
4. Shell command allowlist/blocklist + dangerous-flag/pipe pattern matching
5. Sandbox OS isolation (Landlock/Bubblewrap, Seatbelt, Docker/microVM) per tenant
6. Audit tamper-evident tool receipts (HMAC of session + name + args + result + ts)

Every mutating action passes through a per-invocation check on the parsed input, and the sandbox is the trust boundary — a sandboxed backend can auto-approve because it can't escape.

8.2 The Lethal Trifecta — prompt injection

Simon Willison's rule (2025): untrusted input + access to private data + a way to exfiltrate = disaster. Break at least one leg.

Detection is not containment — this is the single most important security lesson of the last year. In The Attacker Moves Second (2025; researchers from Anthropic, OpenAI, and Google DeepMind), adaptive attackers bypassed 12 published prompt-injection/jailbreak defenses with >90% success, and human red-teamers reached ~100% — against defenses that had originally reported near-zero vulnerability. The takeaway: a guardrail/classifier model is a useful layer but never the load-bearing one. Safety must come from architecturally breaking a leg of the trifecta (remove the private data, the tool reach, or the egress), not from detecting the injection. Operationalize it with the Agents "Rule of Two" (Meta, 2025): in a single un-supervised run, allow at most two of {processes untrusted input · can access private data/systems · can change state or communicate externally}. The moment a flow would have all three, insert a human approval or split the flow.

  • Treat all tool output and retrieved content as untrusted. Never feed it straight into exec/subprocess.
  • Guardrail model in parallel — one instance screens input while another does the work; separating the two beats one model doing both (Anthropic). Treat this as one layer of defense-in-depth, not the primary control (see the adaptive-attack finding above).
  • Egress control — restrict where the agent can send data; block arbitrary outbound network from the sandbox.
  • Scrub credentials from output (regex + dynamically registered secret values) before display or logging.
  • Outbound-payload redaction middleware — when a prompt (with retrieved context or tool output) is about to leave your trust boundary for an external LLM, run it through a middleware that detects and masks/redacts PII, secrets, and card/PHI data first (e.g. a Presidio-style scrubber, NeMo Guardrails, or Llama Guard as a screen). Two cautions so this doesn't backfire: (1) redaction is itself a correctness risk — masking an ID the task actually needs breaks the task, so redact by class and keep reversible tokens where the agent needs referential integrity; (2) the middleware adds latency and is another injection surface, so run the screening model in parallel (per the bullet above) rather than inline in the critical path. The cleanest way to avoid the problem entirely is to route the sensitive task to a self-hosted model (see §8.3) so the payload never leaves.
  • Human-in-the-loop for high-impact actions — payments, deletes, external sends, prod changes require an approval gate (Once / Session / Permanent scopes), delivered where people already work (Slack/Teams). Scope the approval so it doesn't become approval fatigue: high-risk → always ask, medium → session-scoped trust, low → auto in a sandbox.

8.3 Secrets & identity

  • Never put secrets in the prompt. Inject at tool-execution time from a vault; the model sees a handle, not the value.
  • Per-tenant credential isolation — one company's API keys are never reachable from another's session.
  • Act-as / delegated identity — the agent should act with the calling user's permissions, not a god-mode service account. Enforce RBAC at the tool boundary, not just the UI.
  • Route by data sensitivity, not just cost. Keep the model layer provider-agnostic and add a routing rule: sensitive/regulated payloads go to a self-hosted, in-VPC model (e.g. an open-weights model served via vLLM), while non-sensitive reasoning can use a commercial frontier API. This satisfies data-residency and no-egress requirements and optimizes cost — but keep the routing decision itself deterministic and auditable (classify by data label, not by the model's discretion). This complements the complexity-based routing in Part 10.

8.4 Compliance you'll be asked for

Control What to have ready
SOC 2 Type II / ISO 27001 Audited controls over the agent platform
GDPR / CCPA Data residency/region pinning, DSAR support, retention limits, DPA with provider
No-train guarantee Contractual assurance customer data isn't used to train models
Sector rules HIPAA (BAA), PCI-DSS (never let the agent touch raw card data), FINRA/SEC record-keeping
AI governance Model/prompt versioning + an AI risk register, aligned to the frameworks you'll be audited against: NIST AI RMF, the EU AI Act (GPAI transparency obligations and provider penalties become enforceable 2 Aug 2026 — a hard date, not a someday), and the OWASP Top 10 for Agentic Applications (2026) as the concrete threat checklist for the agent itself
Immutable audit log Every action → which user, which tenant, which tool, which inputs, what result, when

✅ Part 8 checklist

  • [ ] Six-layer defense-in-depth implemented, fail-closed
  • [ ] At least one leg of the lethal trifecta is broken for every risky flow
  • [ ] High-impact actions gated by human approval
  • [ ] Secrets in a vault, injected at execution, per-tenant isolated
  • [ ] Agent acts with the user's RBAC scope, not a superuser
  • [ ] Compliance artifacts (SOC 2, DPA, no-train, audit log) in place

🧱 Part 9 — Multi-Tenancy & Isolation

Design for multi-tenancy from day one — retrofitting it is a rewrite.

  • Session model: per-session serial, cross-session concurrent. Lock per session_key (all work in a session is strictly serial → no history races); run different sessions in parallel. This is the simplest correct model for multi-tenant chat/agent workloads (nanobot, PicoClaw, GoClaw).
  • Tenant as the first dimension of every session key, DB row, workspace path, and cost record.
  • Data isolation at the database, not the app. Every query carries tenant_id in the WHERE clause (or Postgres RLS) — never rely on app-level ACLs alone.
  • Workspace isolation via git worktrees / per-tenant sandboxes — sibling worktrees give true parallelism with no checkout collisions and crash-safe discard.
  • Secrets and API keys encrypted per tenant.
  • Cost and rate limits per tenant — one noisy tenant can't starve or bankrupt the others.

Sub-agents / multi-agent where warranted: an orchestrator delegates to workers with separate context windows as context firewalls — each returns a distilled ~1–2K-token summary, and large artifacts are written to a filesystem and passed by reference to avoid the "game of telephone" (Anthropic). Cap nesting depth (≤3) and concurrent children (≈5, semaphore-guarded). The tradeoff is real in both directions: multi-agent burns ~15× the tokens of a chat, but Anthropic's orchestrator-worker research system also outperformed a single agent by ~90% on their internal research eval — so reserve it for high-value, parallelizable work (research, breadth-first triage) where that quality lift pays for the tokens, not routine coding.

✅ Part 9 checklist

  • [ ] Tenant is the first dimension everywhere (sessions, rows, paths, cost)
  • [ ] DB-level tenant isolation (WHERE/RLS), not app-level only
  • [ ] Per-tenant secrets, budgets, and rate limits
  • [ ] Per-session serial / cross-session concurrent locking
  • [ ] Multi-agent reserved for high-value parallel work, with firewalls + caps

📊 Part 10 — Observability, Evals & Cost Governance

You cannot operate what you cannot see — and agents are non-deterministic between runs even with identical prompts.

10.1 Tracing on an append-only event log

Every Action, Observation, and Thought is a typed event with timestamp + source. The event stream is the single source of truth: replayable, debuggable, audit-friendly. Add per-turn spans: tokens in/out, tool calls, latency, cost, model used. Anthropic monitors decision patterns and interaction structure without reading conversation content — critical for privacy/compliance. Full production tracing is what let them diagnose "agent can't find obvious info" failures systematically.

10.2 Evals — treat the agent as a flaky dependency

  • Start immediately with ~20 real queries. Early changes have huge effect sizes; you don't need hundreds of cases to see signal (Anthropic).
  • LLM-as-judge with a rubric (accuracy, completeness, tool efficiency) — a single judge call outputting 0.0–1.0 + pass/fail is the most consistent.
  • End-state evaluation for state-mutating agents — grade the final state, not each step, since valid paths differ.
  • Keep humans in the loop — testers catch hallucinations, source bias, and edge cases evals miss.
  • Prevent spec-gaming — the reward must be hard to fake (real tests pass, build green, no lint errors). Have the agent verify before claiming done.

10.3 Cost governance

  • Meter input + output tokens per turn, attribute cost to the requesting task chain, and answer "who is expensive and why."
  • Hard per-task/per-tenant ceilings → stop-reason cost_exhausted, not a surprise bill.
  • Model routing (two axes): by complexity/cost — a cheap/fast model for easy/common requests, escalate hard cases to a frontier model, fall back down a chain on failure (Anthropic's pattern — a Haiku-class model for the easy tier, a Sonnet/Opus-class model for the hard tier; map to whatever the current generation is); and by data sensitivity — sensitive payloads to a self-hosted in-VPC model, non-sensitive to a commercial API (see §8.3). Keep both routing decisions deterministic and auditable.
  • The proven wins: holding task and model constant and improving only orchestration cut cost ~41%, latency ~44%, tokens ~38% — with task success actually holding steady (78%→81%), so it wasn't a quality-for-cost trade (Writer, The Harness Effect, 2026). Efficiency is a harness property, unconditional of model.
  • Showback/chargeback per team so budgets have an owner.

✅ Part 10 checklist

  • [ ] Append-only event log; per-turn cost/latency/token spans
  • [ ] Observability captures structure, not sensitive content
  • [ ] Eval set (start ~20 cases) + LLM-judge + end-state checks + human review
  • [ ] Per-task/tenant cost ceilings + model routing + showback

🚀 Part 11 — Deployment & Delivery Models

The same agent serves a 5-person startup and a 50,000-person regulated enterprise only if you can deliver it in different topologies without forking the codebase. Because the runtime is stateless with externalized state (Part 3) and every concern sits behind an interface, the same build can ship in four shapes — you pick per customer based on their data-residency, compliance, and ops appetite.

11.1 The four topologies

Model Who it's for What runs where Trade-offs
Multi-tenant SaaS SMB → mid-market; fast self-serve You host everything; tenants are logical slices (RLS, per-tenant secrets/budgets) Lowest cost & fastest onboarding; customer must accept your cloud + a DPA/no-train guarantee
Single-tenant SaaS (dedicated) Regulated mid-market; noisy-neighbor-averse You host, but one isolated stack per customer (own DB, own sandbox pool) Stronger isolation & per-tenant SLAs; higher unit cost & ops overhead
Self-hosted / BYOC (in customer VPC) Large & regulated enterprise Customer runs the platform in their cloud/on-prem; their keys, their egress Meets data-residency & no-egress mandates; you lose direct observability — ship a support/telemetry bridge they control
Hybrid (split-plane) Enterprises wanting managed control + private data Control plane (auth, routing, billing, eval/skill/MCP catalogs, audit sink) hosted by you; data plane (runtime, sandbox, memory, model calls) in the customer VPC Best of both — you operate the fleet, sensitive payloads never leave their boundary; most complex to build & version

11.2 The rule that makes all four possible: a control-plane / data-plane split

Keep a hard control-plane / data-plane split from day one (Part 3). The control plane is auth, RBAC, routing policy, budgets, the eval + skill + MCP catalogs, and the audit sink. The data plane is the kernel loop, sandboxes, memory, and provider calls. If those two never bleed into each other, "move the data plane into the customer's VPC" becomes a deployment flag, not a rewrite. Version the control-plane↔data-plane contract explicitly so a hosted control plane can talk to a slightly older data plane during rollout.

Model routing is a deployment lever too. The two-axis router (Part 8.3, Part 10.3) lets a single hybrid deployment send regulated payloads to a self-hosted in-VPC model (e.g. an open-weights model on vLLM) while non-sensitive reasoning uses a frontier API — so a customer gets frontier quality and no-egress compliance in the same agent, decided by data label.

Package for portability. Ship as a versioned OCI image set + Helm chart (or Terraform module) so self-hosted/BYOC customers deploy a known-good, signed artifact, and rainbow deploys (Part 7.4) apply equally in their cluster.

11.3 Customizing per organization — config + connectors, not forks

Onboarding a new org is configuration and connectors, not a code fork. Everything an org needs to differ is data the platform reads at runtime, in rising order of effort:

  1. Tenant config — a DB row + vault entries: SSO/OIDC identity, RBAC role→scope map, budgets, rate limits, region pinning, retention. (Minutes.)
  2. Agent definition — agents are configurations, not code (GoClaw): markdown bootstrap files (SOUL.md, IDENTITY.md, AGENTS.md, TOOLS.md) + a toolset profile (read_only/coding/messaging/full) + autonomy level. (An afternoon.)
  3. Skills library — seed the org's SKILL.md procedures (their conventions, runbooks); the agent grows more via the eval-gated skill loop (Part 5.4). (Ongoing, compounding.)
  4. Surface adapters — turn on the channels they use: Slack/Teams, a web widget, the REST API, email, cron. Same kernel, new adapter config. (Hours per standard surface.)
  5. Connectors (the integration seam) — attach their systems of record through the vetted per-tenant MCP catalog (Part 5.3): Jira, Salesforce, ServiceNow, data warehouse, internal APIs. Each is RBAC-scoped and credentialed per tenant. (An afternoon for a standard SaaS with an MCP server; a real project for a proprietary legacy system with custom auth.)

The honest boundary on "easily." Customization effort scales with how standard the org's systems are — a connector to a system with a maintained MCP server or clean REST API is an afternoon; a proprietary internal system with undocumented auth, no API, and a VPN requirement is a genuine integration project. The platform gives you the right seam (an RBAC-scoped MCP connector) and the sandbox/audit to run it safely — but you never fork the kernel. And customization never bypasses the trust surface: per-org skills, tools, and connectors are still production config — versioned, eval-gated, sandboxed, and subject to the lethal-trifecta rules (Part 8).

✅ Part 11 checklist

  • [ ] The same build runs in all four topologies via config (no per-customer fork)
  • [ ] Control plane and data plane are separately deployable with a versioned contract
  • [ ] A customer can choose "data plane in my VPC" without a code change
  • [ ] Shipped artifact is a signed image + Helm chart / Terraform module
  • [ ] New orgs onboard via tenant config + agent definition + skills + surfaces + per-tenant MCP connectors

📈 Part 12 — The Scaling Path

Enterprises don't buy agents — they adopt them in stages. Match your engineering to the stage; don't build stage-4 infrastructure for a stage-1 pilot.

Stage Scope What matters most What to build
0. Prototype 1 team, 1 use case Prove value fast Raw API, minimal harness, manual eval, 20-case eval set
1. Pilot 1 dept, real users Reliability + safety basics Budgets, sandbox, audit log, HITL on risky actions, tracing
2. Production 1 org, SLA-backed Multi-tenancy, cost, deploys Control plane, per-tenant isolation, rainbow deploys, cost ceilings, evals in CI
3. Platform Many teams/use cases Reuse + governance Shared agent platform, MCP catalog, skills library, self-serve, policy engine
4. Enterprise-wide Whole company / external Compliance + scale SOC2/ISO, region pinning, multi-provider failover, AgentOps team, chargeback

Rules for climbing:

  • Don't skip stage 1's audit log and HITL — you'll need them the day something goes wrong.
  • Introduce the control plane at stage 2, the moment a second team wants in.
  • At stage 3, standardize the harness, not the model — teams should reuse tools, skills, guardrails, and observability; model choice can stay pluggable.
  • Horizontal scaling falls out naturally from per-session serial / cross-session concurrent + stateless runtime + externalized state. Scale the runtime like any stateless service behind a queue.

✅ Part 12 checklist

  • [ ] Know which stage you're in and built for that stage
  • [ ] Audit log + HITL exist before real users (stage 1)
  • [ ] Control plane introduced at first multi-team demand
  • [ ] Harness (not model) standardized as a platform for reuse

🚄 Part 13 — Performance & Horizontal Scale

Part 12 is about adoption (how an org grows into the agent). This part is the orthogonal, purely technical axis: serving thousands of simultaneous, long-running, token-heavy agent sessions efficiently — whether you run multi-tenant SaaS or a single-tenant/self-hosted stack for one large org. A stage-2 single-org deployment can still need 5,000 concurrent sessions, so treat throughput as its own concern.

The good news: the architecture in Part 3 was built for this. A stateless runtime with externalized state scales like any 12-factor service. The hard parts are the three things that aren't stateless web requests — long-running jobs, sandbox pools, and the provider's own rate limits.

13.1 The unit of scale: a queue + stateless worker pool

An agent run is a job, not a request. It holds a "connection" for minutes, does dozens of model round-trips, and must survive a deploy. Never dedicate a synchronous request thread to a run.

[surfaces] → [gateway/control plane] → [durable queue] → [stateless worker pool] → [sandbox pool]
                (auth, budget, admit)     (per-session          (pull one session,      (per-tenant
                                           FIFO key)             run the loop)            isolation)
         session state + memory + event log live in Postgres/object store, never in the worker

Enter fullscreen mode Exit fullscreen mode

  • Sessions land on a durable queue (SQS/NATS/Redis Streams/Temporal). Workers pull, run the loop to a terminal state, checkpoint, and release.
  • Workers are stateless and disposable — all state is externalized (Part 3/7.3), so you scale them like any queue consumer and a killed worker loses nothing (the job re-queues from its last checkpoint).
  • Autoscale on queue depth and oldest-message age, not CPU. Agent workers are I/O-bound (waiting on the model); CPU is a misleading signal. Target a p95 queue wait, scale out when it's exceeded.
  • Async + streaming/polling to the surface, never a blocked HTTP thread. The surface subscribes to the event stream (SSE/WebSocket) or polls job status.

13.2 Concurrency model — why you can shard horizontally

The per-session serial / cross-session concurrent rule (Part 9) is exactly what makes throughput scaling safe:

  • Route by session key. Hash the session_key to a partition so all work for one session is strictly serial (no history races) while different sessions run fully in parallel across the pool. This is consistent-hashing/sharding, and it's the whole trick.
  • No cross-session shared mutable state in the worker — so adding workers is linear, with no coordination cost.
  • Cap concurrency per tenant (a semaphore or per-tenant partition quota) so one tenant's burst can't consume the whole pool