Anatomy of an Agentic AI System for the Workspace

A practical guide to building — or evaluating — an enterprise agentic workspace assistant in the age of agentic AI.

TL;DR

• Build the boring parts first: permission-aware retrieval and identity are the product. A workspace assistant lives or dies on whether it enforces the source system’s ACLs at query time. The operating principle is simple: permissions are checked before any information ever reaches the LLM; only this pre-filtered, “safe” information is passed through; the LLM cannot leak information it never receives. Everything else — models, retrieval depth, memory — is a differentiator layered on top of that non-negotiable foundation.
• Make opinionated picks and defer the exotic ones. Default stack: Qdrant (vectors) + Postgres/pgvector for metadata and small-scale vectors; Qwen3-Embedding or Gemini Embedding for embeddings; Cohere Rerank / Jina v3 for reranking; recursive character chunking as the default with per-type escalation; LiteLLM as the gateway; LangGraph + Langfuse, dropping LangSmith unless you’re all-in on LangChain and want its eval maturity. Escalate to multi-hop / specialized retrieval only when your logs show hybrid vector failing.
• The intent classifier is the spine of the whole system. Route simple → direct LLM, medium/hard → plan+retrieve (hybrid vector, deepen retrieval for relationship questions), bad → guardrails. Use an embedding router (sub-100ms, ~65x cheaper than an LLM call) with a cheap-LLM fallback, not a frontier model on every turn.
• Cost-per-answer is a first-class design constraint, not a finance afterthought. A 3-step agent loop with accumulating context genuinely costs 50–70x a single cheap-tier call — the math is in Section 14. Tiered routing, budget guards, early exits, and permission-scoped caching are architecture, not optimization.

Key Findings

1. Permission-aware retrieval is architectural, not a filter. Enforce ACLs at retrieval time by mirroring source-system permissions into your index and trimming candidates to the user’s access before the LLM sees anything. Snapshotting permissions at index time or filtering after generation are both broken.
2. Recursive character chunking is the right default — the peer-reviewed NAACL 2025 Findings paper by Qu, Tu & Bao, “Is Semantic Chunking Worth the Computational Cost?”, concludes verbatim that “the computational costs associated with semantic chunking are not justified by consistent performance gains” — but structure-aware chunking wins for code (AST), slides, spreadsheets, transcripts, and tickets.
3. Most “bad retrieval” is a badly understood question. Chunking, embeddings, and rerankers are commoditized; query understanding — conversational rewriting, filter extraction, decomposition — is the last unfixed layer of RAG and where the remaining gains live.
4. Hybrid (dense + sparse) retrieval fused with RRF is the production default; add a reranker; add specialized multi-hop retrieval only when relationship queries demonstrably fail on vectors. Extra index cost and latency are real — earn them.
5. Of six candidate memory layers, two earn their latency budget: working memory (last N turns) and a compressed scratchpad. Episodic recall should be a tool the agent calls, not an always-on injection; exotic temporal/relational memory layers are architecture theatre for most workloads.
6. Agent-generated code must never run in-process. The minimum acceptable isolation for untrusted agent code is a Firecracker/Kata microVM or gVisor; standard Docker/runc shares the host kernel and is insufficient.
7. Agents need their own identity. Use OAuth 2.0 Token Exchange (RFC 8693) delegation semantics and workload identity (SPIFFE) so agents act on behalf of users with scoped, short-lived tokens — not the user’s raw credentials or a static API key.
8. Evals must be bootstrapped from real traffic, and you can build a 200-question golden set with zero human labels — mine and cluster query logs, generate questions from chunks for retrieval labels, use pairwise judging for generation. LLM-as-judge is systematically optimistic; calibrate against a human-labeled sample (target 75–90% agreement).

The Whole System on One Page

Before the section-by-section anatomy, here is the complete system — every service, datastore, queue, and plane, and how a request flows through them.

Service inventory — what each piece is, whether it holds state, and how it scales:

Details

1. Auth & Authorization: identity is the hardest part, do it first

There are four distinct authorization surfaces in a workspace agent, and conflating them is the most common architectural mistake:

1. User identity & session — who is the human.
2. Agent/service identity — who is the software acting.
3. LLM/tool access tokens — what models and tools this request may touch.
4. Data ACLs — what documents this user may see, enforced at retrieval.

User identity and delegated (“on-behalf-of”) auth

Federate to the enterprise IdP (Okta, Entra ID, Google Workspace) via OIDC/SAML. The interesting problem is what happens when an agent calls a downstream tool as the user. Two production-proven mechanisms:

• OAuth 2.0 Token Exchange (RFC 8693) — an IETF Proposed Standard (Jan 2020) defining an STS-style token exchange. The grant type is urn:ietf:params:oauth:grant-type:token-exchange. It carries a subject_token (the user, “on behalf of whom the token is requested”) and an optional actor_token (the acting party — your agent). Critically, RFC 8693 distinguishes impersonation (subject_token only; downstream sees only the user, agent is “indistinguishable from” the user) from delegation (both tokens; “principal A still has its own identity separate from B… A is an agent for B”). For agents, prefer delegation — the issued token carries an act claim that names the actor and can nest to express a delegation chain (auditable lineage), and a may_act claim authorizes who may act for whom.
• Microsoft Entra ID On-Behalf-Of (OBO) flow — a middle-tier API exchanges the user’s access token for a downstream-scoped token by setting requested_token_use=on_behalf_of. Note OBO works only for user principals, and Microsoft is candid that it is “a Microsoft flavor for a standard,” not the standard itself. Entra Agent ID (2025–2026) extends this to a two-exchange agent OBO flow.

Recommendation: Use RFC 8693 token exchange with delegation semantics as your cross-service standard, and downscope on every hop (request only the scope/audience/resource the next step needs). Be aware of the real-world caveat that “most implementations quietly allow scope to persist unchanged” — enforce attenuation in policy, don’t assume the AS does it.

Service identity for agents

An agent is a machine workload and deserves a workload identity distinct from any user. Use SPIFFE/SPIRE: the agent gets a SPIFFE ID (e.g., spiffe://acme.com/agent/research) and a short-lived, auto-rotating SVID (X.509 for mTLS, or JWT-SVID for token exchange). OpenAI now supports exchanging a SPIFFE JWT-SVID for a short-lived OpenAI access token, avoiding long-lived API keys; HashiCorp Vault Enterprise added native SPIFFE auth for non-human identities. Palo Alto’s guidance is blunt: “Autonomous AI agents are machine workloads and require dedicated workload identities, not human-centric credentials. Relying solely on OAuth or static API keys for AI agents creates security blind spots.”

For the agent↔tool boundary, follow the MCP authorization spec: as of the 2025-06-18 revision, MCP servers are OAuth 2.1 Resource Servers that must implement Protected Resource Metadata (RFC 9728), and clients must implement Resource Indicators (RFC 8707) to stop token reuse against the wrong server (the “confused deputy” problem). The Nov 2025 revision added incremental scope consent and Enterprise-Managed Authorization (Okta’s Cross-App Access / ID-JAG).

Token lifetime & attenuation: issue short-lived (minutes-to-an-hour) scoped tokens per agent/action. For fan-out to sub-agents, capability tokens that support offline attenuation are attractive — Biscuit tokens (Ed25519 public-key signatures + embedded Datalog policy) let a parent agent hand a strictly-narrower token to a child without a round-trip to the AS; macaroons do similar with HMAC caveats. There’s an emerging IETF draft, “Attenuating Authorization Tokens for Agentic Delegation Chains.” Flag: this is genuinely contested — Authress argues offline attenuation has real drawbacks — so treat it as advanced, not table-stakes.

Permission-aware retrieval (the crown jewel)

This is what makes the system deployable in a regulated enterprise. The production pattern to copy:

1. Connector ingestion pulls content plus the source ACL for every object.
2. Identity resolution maps identities across systems (so “Anthony in Slack” = “Anthony in Salesforce”).
3. Permission mirroring syncs allow/deny users and groups into an identity-and-permissions store.
4. At query time, retrieve candidates, filter through the user’s permissions, then pass only the safe set to the LLM.

The principle: permissions are checked before any information ever reaches the LLM. Only this pre-filtered, “safe” information is passed through. The LLM never even sees the restricted data — it cannot leak what it never receives. Permission checks happen externally, so security is enforced by the architecture, not left to the model. Enforce ACLs at query time, on every retrieval, in real time — not as a stale index-time snapshot. Store per-chunk ACL metadata (allowed users, allowed groups, denied users, denied groups) alongside the vector, and push the filter into the vector DB query so trimming happens before ranking.

2. LLM & tool budgeting per team (ACLs extended to cost)

Model access is an entitlement, and tokens are a metered resource. Treat both like any other ACL:

• Model-tier entitlements: which teams can call frontier models (Opus-class) vs. cheap models (Haiku/Flash-class). Encode as scopes on the team’s virtual key.
• Per-team token budgets & rate limits: enforce at the gateway with virtual keys. LiteLLM’s virtual keys give per-team budgets and rate limits natively; this is a primary reason to run a gateway.
• Cost allocation / chargeback: tag every LLM call with team/user/project and aggregate. Langfuse gives per-trace/session cost; roll it up to a chargeback dashboard.

Recommendation: LiteLLM virtual keys per team with hard budget ceilings + soft alert thresholds; model-tier scopes on each key; Langfuse for cost attribution. The point that surprises execs: a router that sends every request to a frontier model when a cheap one suffices “costs 30x more” — model-tier entitlements are a cost-control lever, not just governance. Section 14 turns this into per-query unit economics.

3. Data ingestion

3a. Chunking strategies per data type

The position that recursive character chunking beats semantic chunking for most cases is correct, and the 2025 evidence backs it. The peer-reviewed NAACL 2025 Findings paper (Qu, Tu & Bao, “Is Semantic Chunking Worth the Computational Cost?”, pp. 2155–2177) states plainly: “the computational costs associated with semantic chunking are not justified by consistent performance gains,” with fixed ~200-word chunks matching or beating semantic chunking across retrieval and answer generation. Merola & Singh (2025) reach the same conclusion; a Feb 2026 Vecta benchmark of 7 strategies placed recursive 512-token splitting first. Semantic chunking’s win (when it exists) is a few points of recall at 10–40x the embedding cost.

But “recursive by default” does not mean “recursive for everything.” Structure is signal — use it when the data has it:

Two upgrades worth the cost regardless of splitter: Anthropic’s Contextual Retrieval (prepend a doc-level context blurb to each chunk before embedding) and parent-document retrieval (embed small, return large).

3b. Data relevance: signal over noise

Not all sources are equal. Official docs should outrank a Slack rant. Implement: - Source authority weighting — a per-source-type prior (published docs > wiki > tickets > chat). - Freshness decay — exponential time-decay on a recency score; a 2024 runbook should lose to the 2026 one. - Deduplication — near-dup detection (MinHash/embedding cosine) so five copies of the same PDF don’t flood results. - Popularity/click signals — feed thumbs-up, click-through, and dwell back into ranking. Mature enterprise search systems do exactly this with metadata, owners, and status fields.

These are ranking features, not just retrieval — bake them into the reranking stage (Section 8).

3c. Data storage

Vector DB — the choice matters most. My decision:

Pick Qdrant for a dedicated store (permission filtering is exactly its strength, and permission-aware retrieval demands fast filtered search), or pgvector if you’re small or already Postgres-centric — its filtering and 5–8ms HNSW latency mean “the database query is not the bottleneck” until real scale. Migrate to a dedicated store around 50–100M vectors or when cloud costs cross a few hundred dollars/month.

Everything else: object store (S3/GCS) for raw docs and originals; Postgres for metadata, lineage, and permission mirror; Redis for cache (embeddings, hot retrievals, session state) and rate-limit counters.

3d. Data lineage & embedding models

Embedding model choice (verify current MTEB standings, which shifted hard in early 2026):

Recommendation: If you want managed and multimodal, Gemini Embedding. If you want self-hosted, data-resident, no per-token cost, Qwen3-Embedding-8B (drop to 4B/0.6B for latency). Note the FinMTEB finding: the best MTEB model can drop ~8.5 points on a domain corpus — benchmark on your own data with MRR/NDCG before committing.

The often-missed pieces — these separate a demo from a product:

• Incremental sync / CDC: don’t full-crawl. Use webhooks/change-logs to capture deltas every 1–5 minutes (mature enterprise connectors do this; eSapiens reports near-real-time via webhook/change-log triggers). Push delta changes, not re-crawls.
• Permissions sync: ACL mirroring is a continuous job, not a one-time import. Re-sync group membership and per-object ACLs on a schedule and on change events; stale ACLs are a security incident waiting to happen.
• PII detection/handling: run PII detection/classification at ingestion; mask or tag sensitive fields; support incognito/no-retention paths for sensitive conversations.
• Deletion propagation (GDPR right-to-be-forgotten): a delete in the source must propagate to the index and the embeddings and any derived memory. Track lineage (object → chunks → vectors) in Postgres so deletion is a deterministic cascade, not a hope.
• Index versioning & re-embedding: when you change embedding models you must re-embed. Version your index; dual-write and shadow-read the new index; cut over behind an eval gate. Immutable versioning (v1→v2 on re-chunk/re-index) gives rollback.

Ingestion pipeline diagram:

4. UI: buy/adopt-OSS vs build

Decision framework: For an internal tool where the assistant is the product, build a hand-rolled agentic app (React + assistant-ui or Chainlit) so you control the agentic affordances — but adopt OSS for the shell and spend your engineering on retrieval/orchestration, not re-implementing a chat textbox. Off-the-shelf chatbots (Open WebUI/LibreChat) are the right call for a fast internal pilot; graduate to hand-rolled when you need streaming plans, DAG visualization, and approval UX they don’t model well.

Where the UI needs true agentic affordances (non-negotiable): streaming the plan (not just tokens), tool-call visibility (what it’s calling and why), inline citations to permission-checked sources, human-in-the-loop approval gates for write actions, and partial-failure rendering (see Error Handling).

5. Orchestration: an intent classifier is the spine

Every message is classified into simple / medium / hard / bad, and the route follows:

• Simple (greeting, definition, chit-chat) → direct LLM answer, no retrieval.
• Medium → single retrieval pass (hybrid vector) → answer. Usually no multi-step plan.
• Hard (multi-hop, cross-source, analytical) → plan: build a tool DAG, fetch user memory, retrieve (deepen retrieval for relationship questions), execute, synthesize.
• Bad (adversarial, off-policy, prompt injection) → guardrails: refuse/deflect gracefully, log, and flag.

Classifier implementation tradeoffs:

Recommendation: an embedding router as the default, with a cheap-LLM fallback when the router’s top-class confidence is below threshold. IBM’s ModernBERT semantic router showed a 47.1% latency reduction; NVIDIA’s blueprint uses Qwen-1.7B intent routing. Routing overhead is negligible (10–50µs for the compare) against 500–2000ms inference. Implement the whole flow in LangGraph, where the classifier is the conditional edge.

Intent-classifier routing flowchart:

6. Agent library with ACLs

A registry of agents/capabilities is what turns a monolith into a platform: - Capability manifests (“capability sheets”) per agent: name, purpose, input/output schema, tools it may call, data scopes, cost tier, owner. Anthropic’s finding is directly relevant: “poor tool descriptions [can] send agents down completely wrong paths” — treat the agent-tool interface like a human-computer interface. - Versioning: immutable versions with rollback; capability manifests are versioned artifacts. - Invocation ACLs: which teams/users/agents may invoke which agents — same entitlement model as model tiers. - Approval workflows: new or modified agents go through review before they’re callable in prod (mirrors LangGraph interrupt-based approval).

7. Agent execution environment

The single most important rule: agent-generated code must never run in-process. It shares your interpreter, your memory, your secrets, and your network. The minimum acceptable isolation for untrusted agent code is a Firecracker/Kata microVM (hardware-level, separate kernel) or gVisor (user-space kernel, lighter); standard Docker/runc “shares the host kernel and is explicitly insufficient.” This matches E2B, Modal, and AWS Lambda’s public architectures.

Recommendation: E2B (or self-hosted Firecracker at volume) for untrusted code; Modal if you need GPUs in the sandbox. Beyond isolation: egress allowlists (no arbitrary outbound; whitelist approved endpoints), secrets injection at runtime (never bake into images; short-lived tokens from Section 1), resource & wall-clock limits (OWASP LLM10:2025 = unbounded resource consumption), and ephemeral sandboxes torn down per task.

8. Retrieval

8a. The retrieval problem was never retrieval: query understanding

Here’s the uncomfortable truth after two years of RAG industrialization: chunking, embeddings, and rerankers are commoditized. Most residual “bad retrieval” is a badly understood question. Embedding a bad query well still retrieves the wrong documents, precisely. The query-understanding layer is the last unfixed layer of RAG, and it’s where the remaining gains live.

Workspace queries fail for predictable reasons: they’re elliptical (“and for Q3?” — meaningless without the last three turns), jargon-dense (internal codenames, team acronyms no embedding model has seen), underspecified (“the board deck” — which of forty?), or compound (two questions wearing one trench coat). Fix them in this order of ROI:

Implement rewrite + filter extraction as one structured-output call to a cheap self-hosted model (a 4–8B Qwen handles it), cache rewrites in Redis keyed on (conversation-tail hash, query), and run it only for medium/hard intents — the classifier already told you simple queries don’t need it. Then give retrieval a retry ladder instead of a single shot: retrieve → if reranker confidence is low → try a rewrite variant → broaden filters → HyDE → admit you couldn’t find it. An honest “I couldn’t find this in Confluence or Drive” preserves more trust than a confident hallucination, and it feeds the data-source-coverage-gap metric on the management dashboard.

8b. Hybrid + RRF + rerank

The production default is hybrid retrieval: dense (embeddings) + sparse (BM25, or SPLADE for vocabulary-mismatch corpora) fused with Reciprocal Rank Fusion (RRF). RRF sums 1/(k + rank) across lists with k=60, the smoothing constant from the original Cormack, Clarke & Büttcher paper (“Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods,” ACM SIGIR 2009), which showed RRF “consistently yields better results than any individual system, and better results than the standard method Condorcet Fuse.” Use RRF over score-based fusion because it operates on ranks, not scores, so it’s immune to the normalization pathologies that break weighted averaging (a single outlier BM25 doc compresses all other scores toward zero after min-max). On WANDS, tuned hybrid reaches 0.7497 NDCG vs. ~0.698 for either method alone (~7.4% lift).

SPLADE vs BM25: BM25 for exact-match-heavy corpora (SKUs, error codes) and smaller corpora where IDF is meaningful; SPLADE for vocabulary-mismatch knowledge bases — but pre-compute SPLADE doc vectors at index time (Qdrant stores sparse vectors natively) since query-time SPLADE adds 100–300ms.

Reranking (the highest-ROI upgrade after adding BM25): take top-50–200 fused candidates and rerank with a cross-encoder.

Recommendation: Cohere Rerank 3.5 if you want managed zero-ops; Jina Reranker v3 if you self-host and need a strict sub-200ms budget. The latency/quality tradeoff: reranking adds 150–600ms but delivers 15–40% higher precision than embeddings alone — worth it for medium/hard queries, skip it for simple ones. One sobering benchmark truth: “the retriever sets the ceiling” — no reranker pushed Hit@10 above 88% because the missing 12% never entered the candidate pool. Invest in retrieval recall (and Section 8a) first.

8c. When hybrid vector isn’t enough (driven by the intent classifier)

Specialized multi-hop stacks cost more to index and add query latency; they can also lag on questions that need fresh knowledge when entity representations go stale. Do not architect for hypothetical multi-hop queries — find the real ones in your logs first, then deepen retrieval (decompose, iterate, or add a specialized index). Prefer the cheapest escalation that fixes the failure mode you actually measured.

Full pipeline: classify → understand/rewrite → (hybrid dense+sparse retrieve, ACL-filtered) → RRF fuse → rerank → [optional multi-step / decomposed retrieve for multi-hop] → assemble context → LLM → cite.

9. Context manager

Context is a scarce, actively-managed resource — this is Anthropic’s “context engineering”: find “the smallest possible set of high-signal tokens.” Techniques, and when to use each:

• Token budgeting per LLM call including tool outputs. Tool results are the biggest hidden context hog — cap and summarize them.
• Compaction/summarization — summarize old turns when the thread is long and continuity matters (Anthropic: “compaction maintains conversational flow for tasks requiring extensive back-and-forth”). Claude’s memory tool ships server-side context compaction for exactly this.
• Structured note-taking / scratchpad — persist salient facts outside the context window (“excels for iterative development with clear milestones”). This is the CLAUDE.md / progress-file pattern.
• Sliding window — keep the last N turns verbatim (working memory).
• Trimming — drop irrelevant retrieved chunks that the reranker scored low.
• Sub-agent isolation — for hard tasks, spin sub-agents with their own context windows and return only condensed summaries to the lead. Per Anthropic’s “How we built our multi-agent research system” (Jun 2025), “a multi-agent system with Claude Opus 4 as the lead agent and Claude Sonnet 4 subagents outperformed single-agent Claude Opus 4 by 90.2% on our internal research eval,” with token usage alone explaining ~80% of performance variance — at roughly 15x the tokens.

Summarize vs. re-retrieve: re-retrieve when the user pivots topic (cheaper and fresher than dragging stale context); summarize when the same thread grows long and history matters. Rule of thumb: if answering needs facts not in context, re-retrieve; if it needs the gist of a long conversation, summarize.

Six layers of memory, two that mattered

Every memory architecture diagram in 2025 showed the same six boxes. Having watched teams build all six, here is the honest verdict on which layers earn their latency-and-token budget and which are architecture theatre:

The pattern behind the verdicts: layers that condense earn their keep; layers that recall speculatively don’t. Working memory and the scratchpad are dense-by-construction — every token in them was recently relevant or explicitly distilled. Episodic recall and exotic temporal layers inject on a guess about relevance, and the guess is usually wrong, so you pay latency and context pollution for occasional wins — which is exactly why the fix is making recall deliberate (a tool call) rather than ambient (always-on injection). The LIGHT framework’s three-store result (episodic + working + scratchpad, below) is consistent with this: even in the paper, the scratchpad does disproportionate work.

10. User memory

Anthropic shipped Claude memory in 2025: it “periodically summarizes your conversations and carries forward… the most relevant context,” auto-generates memories (“when Claude detects something worth remembering… it creates a memory entry automatically”), is account/project-scoped, user-viewable/editable/deletable, opt-in (with an Incognito no-retention mode), and is moving toward file-based “Memory Files.” The developer memory tool is client-side — your app executes storage, so you control where memories live. This is the model to emulate: a background “auto-dream” process that mines past chats for durable preferences and context.

Storage choice — .md files vs vector vs hybrid:

Recommendation: hybrid. A markdown vault for canonical, user-visible preferences (portable, editable, the source of truth the user controls) plus a vector layer (Mem0-style) for transient session memory — the pattern “most mid-market deployments end up” with. Add a specialized temporal memory layer only if your domain has entities that change ownership/state over time and filters + vectors demonstrably fail. Memory system landscape: Mem0 (personalization, token-efficient), Zep (temporal, SOC2/HIPAA/GDPR), Letta/MemGPT (OS-style self-managed tiers).

(On the “LIGHT” framework: there is a genuine 2026 paper — “Beyond a Million Tokens,” arXiv:2510.27246, ICLR 2026 — presenting a memory framework named LIGHT that equips an LLM with three complementary systems: long-term episodic memory (FAISS-indexed), short-term working memory (recent turns), and a scratchpad of accumulated salient facts, improving memory-QA by 3.5–12.7% over strong baselines. Its three-store design maps directly onto layers 2–4 of the table above — validate against its BEAM benchmark if memory is central to your product. Note the name collides with Facebook’s unrelated LIGHT text-adventure environment.)

Memory governance (non-negotiable): - Write policy: only write high-confidence, durable facts (preferences, roles, recurring context) — not transcripts. Anthropic stores “preferences… not conversation transcripts.” - Retrieval-time injection: inject only memories relevant to the current query (the scratchpad-filtering step). - Decay/expiry: TTL on memories; refresh on re-observation. - User control: view/edit/delete all memories; per-project scoping; incognito mode. - Privacy boundaries: memories are account-scoped, never cross users, never leak across permission boundaries.

Memory lifecycle diagram:

11. Evaluation is the product

Public benchmarks tell you nothing about your system. Stronger claim: your eval set is the executable spec of your product. Every prompt change, model swap, chunker tweak, and reranker upgrade is gated on it — which means whoever owns the eval set owns product quality. Treat it like a test suite: versioned in git, reviewed in PRs, owned by a named team.

Building a 200-question golden set with zero labels

The classic failure mode: “a founding engineer hand-writes 80 cases in a Notion doc… six months later the gate is green every build [and] production is on fire.” Hand-imagined cases don’t match the real query distribution. Here’s how to build 200 real questions without paying for a single label up front:

1. Mine the distribution. Take 2–4 weeks of query logs, embed them, cluster (HDBSCAN or k-means), and sample proportionally per cluster. Your 200 questions now mirror what users actually ask — including the ugly, elliptical, jargon-filled ones your team would never have invented.
2. Get retrieval labels for free by inverting the problem. Sample chunks from the corpus, have an LLM generate questions each chunk answers, and apply a round-trip consistency filter (does your retriever find the source chunk for its own question at generous k? if not even then, the question is ambiguous — drop it). Every surviving (question, source-chunk) pair is a retrieval label — recall@k and MRR are now measurable with zero humans.
3. Score generation without reference answers. Absolute LLM-judge scores are noisy and systematically optimistic; pairwise judging vs. a pinned baseline (does version B beat version A on this question?) is far more reliable. Add reference-free groundedness (RAGAS-style faithfulness: is every claim in the answer supported by the retrieved context?) — which also needs no labels because the retrieved context is the reference.
4. Spend human effort where the machine disagrees with itself. Have one domain expert label the 30–50 cases where judges disagree or confidence is low, binary pass/fail with a one-line critique. This calibrates the judge (target 75–90% agreement — MT-Bench found GPT-4 agrees with humans ~80%, about human-human agreement) and seeds the four-bucket structure: production sample, adversarial, edge cases, and replays of failures that already shipped.

LLM-as-judge, with eyes open

It’s systematically optimistic. Its top failure: factual verification without reference context — a judge asked “is this accurate?” with no source defaults to scoring plausibility, and “confident, fluent hallucinations often receive high scores.” Always give the judge the retrieved context. Run deterministic checks first — millisecond code evals (schema parses? contains a phone number it shouldn’t? one of N labels?) filter obvious breakage before you spend judge tokens. And self-host the judge: a 30B-class model judging pairwise with context is cheap enough to run nightly.

The regression harness: catching drift before users do

The harness is layered by cost, so the cheap layers run constantly:

Online: sample 5–10% of production traffic through the groundedness judge; A/B new versions behind the gateway; route low-confidence answers to human review; promote failures into the regression set. Canary before deploy: no model/prompt version touches traffic until it passes the golden + adversarial sets.

Tooling: LangSmith has the more mature eval system (configurable judges, few-shot correction where human corrections feed back as few-shot examples, dataset tooling, failure clustering) and zero-config tracing if you’re on LangChain/LangGraph. Langfuse added Score Analytics (Nov 2025: evaluator precision/recall/F1) and baseline comparison, runs LLM-as-judge + code evals in-platform, and gates CI/CD. Ragas for RAG-specific metrics (context precision/recall, faithfulness). Arize Phoenix’s Evals library ships pre-benchmarked templates (70–90% precision targets).

12. Monitoring & drift

Observability stack (opinionated): - Langfuse for LLM observability — traces, sessions, per-trace/session cost, prompt management. v3 is OpenTelemetry-native, so traces slot into an existing OTel backend (Jaeger/Tempo/Honeycomb). Self-hostable (data residency). - ELK/OpenSearch for application logs. - Prometheus + Grafana for infra metrics. - Distributed tracing across agent hops — one trace ID spanning classifier → planner → each tool call → synthesis, so you can see where a multi-agent request spent its time and tokens.

LangSmith vs Langfuse — take a position: they overlap heavily (tracing + evals). Running both is redundant waste. My recommendation: run Langfuse for production observability + cost (OSS, self-hostable, framework-agnostic, OTel-native), and only add LangSmith if you are all-in on LangChain/LangGraph and want its stronger eval maturity — in which case you can arguably run LangSmith alone. Do not pay for and operate both. If forced to one: Langfuse for most teams; LangSmith for deep-LangChain shops that lean on evals.

Drift monitoring — two distinct problems:

Technical drift (is the machine healthy?): - Embedding/data drift — track distribution shift and embedding-centroid distance across time windows (Arize Phoenix does embedding drift + retrieval relevance; Evidently for statistical drift; note WhyLabs was acquired by Apple in early 2025 — plan around it). Alert on PSI/KS/Wasserstein shifts. The sentinel-Jaccard check in Section 11 is the cheapest version of this. - Model drift — silent upstream provider updates change behavior with zero code change; catch via scheduled golden-set re-runs vs. a baseline. - Infra signals — error rates, p95/p99 latency, tool failure rates, per-tool timeouts.

Non-technical drift (are users being failed, even when nothing errors?) — this is the signal most dashboards miss (“the tool wasn’t lying. It was measuring the wrong four signals”): - Hallucination tracking — sample traffic through a groundedness judge; rule-based groundedness checks (regex on known product names, prices, dates) catch a lot cheaply. - Thumbs-down rate, reformulation/retry rate (user rephrases → we failed), session abandonment, escalation-to-human rate, and sentiment of follow-up messages (frustration in the next turn is a wrong-answer signal even without a thumbs-down). - Cadence that works: daily on rubric/embedding-distance hot signals; weekly on judge-calibration drift (rebuild the human-labeled set, re-score, alarm on agreement drop) and retrieval-corpus drift.

13. Infrastructure: the substrate everything runs on

This is where “we built an agent” becomes “we run a platform.” Kubernetes is the substrate; the interesting decisions are node pools, serving economics, the async backbone, caching, and stateful-service operations.

Kubernetes topology

Four node pools, because the workloads have nothing in common:

Namespaces per plane (edge, orchestration, retrieval, exec, ingest, data, obs) with default-deny NetworkPolicies and explicit allows matching the architecture diagram — the diagram is your network policy spec. All egress through an Envoy egress gateway with per-namespace allowlists: the orchestration plane may reach LLM providers and MCP servers; the sandbox pool reaches only its task-scoped allowlist; ingestion reaches only registered connector endpoints. If a prompt-injected agent tries to exfiltrate to an arbitrary domain, the egress layer — not the model’s good behavior — is what stops it.

Secrets: External Secrets Operator + Vault (or cloud secret manager); nothing in env vars at build time; SPIRE issues workload SVIDs; skip a full service mesh at small scale (mTLS at the critical hops — gateway↔providers, orchestrator↔tools — via SPIFFE-aware sidecars) and adopt Linkerd before Istio if you later want mesh-wide mTLS with minimal ops tax.

Autoscaling that actually works here: HPA on RPS for stateless services; KEDA on RabbitMQ queue depth for Celery workers (ingestion bursts when someone connects a 10-year-old Drive); GPU pool scales on vLLM queue-wait with a warm floor of one replica per model — model load is 1–5 minutes, so scale-from-zero means a five-minute p99 for the first user. Accept the warm-floor cost or route cold-start overflow to API.

Model serving economics: what to self-host and why

The right mental model is two tiers with opposite economics:

• Utility tier (self-host on vLLM): classification fallback, query rewriting, scratchpad compression, summarization, LLM-as-judge, memory auto-dream. High-volume, low-stakes, latency-tolerant of a 4–14B model. A single L40S-class node (~$1.0–1.5/hr ≈ ₹65–95k/month) running a Qwen3-8B at FP8 with continuous batching serves thousands of tokens/sec aggregate. The same call volume through a mid-tier API at ~$3/MTok input crosses that monthly cost around ~2B input tokens/month — which a busy 1,000-person deployment hits easily once rewriting, judging, and compression run on every medium/hard query. Self-hosting the utility tier is usually the single biggest cost lever after tiered routing, and it keeps rewrites and judgments data-resident.
• Frontier tier (API via the gateway): final synthesis on hard queries, complex planning, anything where quality is the product. Spiky, low-volume relative to utility calls, and you cannot match frontier quality self-hosted. Don’t try.

vLLM configuration that matters: automatic prefix caching on (the static system prompt + tool schemas are 2–8k tokens re-sent on every call; prefix cache turns them into near-free TTFT), FP8/AWQ quantization (halves VRAM, minor quality cost at this tier), tensor parallelism only when a model doesn’t fit one card. TEI (Hugging Face text-embeddings-inference) serves both Qwen3-Embedding and the reranker; embeddings tolerate CPU at low QPS, but the reranker sits on the interactive path — give it GPU.

The gateway makes tiers operational: LiteLLM model groups (utility → vLLM, frontier → Anthropic with OpenAI fallback), retries and provider failover at the gateway (not in app code), canary weights for new models (5% traffic → compare in Langfuse → promote), and per-team virtual keys from Section 2. One deliberate choice: the orchestrator never knows provider URLs. Every LLM call, self-hosted or API, goes through the gateway — that’s what makes routing, budgeting, caching, and canarying single-point-of-control instead of scattered through app code.

The async backbone: Celery/RabbitMQ vs Temporal

Everything that isn’t the interactive request path runs async: ingestion, permission re-sync, memory auto-dream, nightly evals, long agent jobs.

Recommendation: Celery + RabbitMQ for the batch plane — priority queues (interactive-adjacent jobs never starve behind a corpus re-index), acks_late + idempotent tasks, dead-letter exchange for the DLQ, Celery Beat for schedules (permission re-sync every 15 min, auto-dream nightly, eval harness nightly). LangGraph’s PostgresSaver checkpointer already gives durable resumable execution for the agent path, which covers most of what teams reach for Temporal for. Adopt Temporal only when agent workflows genuinely span hours-to-days with human approval gates and the checkpointer’s node-granularity resume isn’t enough.

Caching: four tiers, one hard rule

The hard rule: the semantic cache key must include the user’s permission hash. A cached answer synthesized from documents user A can see, served to user B who cannot, is a data leak that bypassed your entire permission-aware retrieval architecture through the side door. Scope cache entries per-user or per-ACL-group-signature, and flush affected entries on permission-sync changes. This is the most common way otherwise-correct systems leak.

Stateful services: HA, sizing, DR

• Postgres: managed (RDS/Cloud SQL) or CloudNativePG; PITR backups. It carries the permission mirror, lineage, LangGraph checkpoints, agent registry, and gateway spend — consider splitting checkpoint churn (high write volume, low value-per-row) from the permission/lineage OLTP instance.
• Qdrant: 3-node cluster, replication factor 2, snapshots to S3 nightly. Sizing: RAM ≈ vectors × dims × 4 bytes (fp32) × ~1.5 HNSW overhead; scalar quantization cuts it ~4x, binary ~32x. 10M chunks × 1024-dim ≈ 41GB fp32 → ~12–15GB with int8 SQ — one 64GB node holds it; the cluster is for availability, not capacity, at this scale.
• Redis: Sentinel or managed; logically separate cache (evictable) from rate-limit counters and session state (not evictable).
• ClickHouse: only if self-hosting Langfuse v3 at volume; traces are columnar-friendly and cheap there.
• DR targets: RPO minutes (PITR + snapshots), RTO under an hour. The one thing you cannot quickly rebuild is the vector index — a full re-embed of a large corpus takes days and real money, so treat Qdrant snapshots as tier-1 backups, not nice-to-haves.

Environments and rollout

Dev/staging/prod with GitOps (ArgoCD + Helm, infra in Terraform). Three rollout patterns specific to this system: shadow indexes for embedding-model changes (dual-write new index, shadow-read and compare recall via the Section 11 harness, cut over behind the eval gate); gateway-weight canaries for model/prompt changes (5% → Langfuse comparison → promote); connector staging (a new connector runs against staging with a scoped service account and passes permission-sync verification before touching prod). Rough footprint for a 1,000-person org (~10k queries/day, 2–5 QPS peak): 6–10 general CPU nodes, 1 GPU node, 1 sandbox node, managed Postgres/Redis, 3-node Qdrant — on the order of $3–5k/month infra plus API spend, which Section 14 puts under control.

14. Cost per answer: rupees-per-query as a design constraint

Treat cost-per-answer the way you treat p95 latency: a budgeted, monitored, per-request property of the system — not a monthly invoice surprise. The unit economics decide whether this platform scales to the whole company or gets quietly throttled by finance.

The cost model

cost(answer) = classify + rewrite + embed(query) + retrieve + rerank
             + sum over agent steps ( input_tokens_i x rate_in + output_tokens_i x rate_out )
             + tool-call costs

The killer term is the sum: in an agent loop, context accumulates, so each step re-sends everything the previous steps produced. Input tokens grow roughly linearly per step, which makes total input cost grow quadratically with step count. This is why a “quick 3-step agent” costs an order of magnitude more than intuition says.

Worked example (illustrative rates: cheap tier $1/$5 per MTok, mid tier $3/$15, ₹85/$)

₹16 vs ₹0.26 — the 3-step agent loop really is ~60x a single call, and a 5-step loop with fatter tool outputs clears 100x. Blended at a realistic 60/30/10 simple/medium/hard mix: ~₹2.4/query. Untamed — every query to a frontier model with an eager agent loop — the same mix runs 10–20x that.

The five levers, in order of leverage

1. Tiered routing (the classifier again). 60–70% of workspace traffic is simple/medium and never needs the frontier tier or a loop. This is the same intent classifier from Section 5 wearing a finance hat — one component, two jobs.
2. Caching. A 25% semantic-cache hit rate cuts blended cost 25% outright (→ ~₹1.8/query in the example). Prompt caching takes another large bite out of the hard path: the 3–8k static prefix (system + tools) re-sent on every step of every loop drops to ~0.1x — the naive ₹16 hard query lands closer to ₹8–10 with prefix caching plus tool-output compression between steps.
3. Early exits. If the reranker’s top score is high and intent is medium, answer directly — skip the planner. If retrieval comes back empty after the retry ladder, say so in one cheap call instead of letting an agent flail through five steps trying to conjure sources. Confidence thresholds are exit ramps.
4. Budget guards. Carry a cumulative token/₹ counter in LangGraph state with a per-intent-class ceiling (say, ₹20/hard query). On breach: stop, summarize what’s done, return a partial answer with “want me to continue?” — degrade gracefully, never silently burn ₹200 on a runaway loop. Per-team monthly ceilings enforce at the LiteLLM layer (Section 2); per-query ceilings enforce in the orchestrator. You need both.
5. Context pruning between steps. Don’t re-send raw tool outputs; compress them to what the next step needs (Section 9). This directly attacks the quadratic term.

Dashboard the unit economics: blended ₹/query, ₹/query by intent class, cache hit rates, budget-guard trip rate, and ₹ per resolved query (cost divided by answers that weren’t reformulated or thumbs-downed) — the last one is the honest number, because a cheap wrong answer that triggers three retries is more expensive than one good ₹16 answer.

Error handling & reliability (cross-cutting, non-negotiable)

Agentic systems fail partially and constantly; design for it: - Idempotency keys on every side-effecting tool call — critical because LangGraph re-executes a node from its start on resume after an interrupt, so any pre-interrupt API charge or DB write must be idempotent or it double-fires. - Exponential backoff with jitter on transient failures (429s, 5xx) to LLMs and tools. - Circuit breakers per tool/provider — trip open after a failure threshold, fail fast, half-open to probe recovery. - Timeouts per tool — every tool has a wall-clock budget; a slow tool must not hang a DAG. - Dead-letter queues for ingestion and async tool jobs that exhaust retries — inspect and replay, don’t silently drop. - Partial-failure UX — when one node in a DAG fails, show the user what did succeed with citations, mark the failed branch explicitly (“couldn’t reach Jira; here’s what I found in Confluence”), and offer retry — never fail the whole answer because one of five tools timed out. - Saga / compensation for multi-step actions — if the agent performs a sequence of writes (create ticket → assign → notify) and step 3 fails, run compensating actions to undo steps 1–2, or checkpoint so a human can resume. Use LangGraph checkpointers (state snapshot per super-step, keyed by thread_id) + interrupts for human approval of high-risk steps. Gate irreversible actions (delete, payment, prod change) behind an interrupt() approval — payments under a threshold auto-approve, above it route to a human queue.

Management dashboard: what execs and platform owners see

Closing: the build order

Phase 1 — Startup scale (prove value, weeks-to-months): - IdP federation (OIDC) + permission-aware retrieval from day one (this is not optional even at MVP — it’s the trust foundation). - 2–3 connectors (Drive, Slack, Confluence) with CDC sync and ACL mirroring. - pgvector + recursive chunking + hybrid (dense+BM25) + RRF; Qwen3 or Gemini embeddings. - Conversational query rewriting from the first chat release — it’s the highest-ROI retrieval fix and it’s one cheap LLM call. - LiteLLM gateway with per-team virtual keys; LangGraph orchestration; Langfuse for traces + cost. - Embedding intent router (simple/medium/hard/bad) with LLM fallback — doing double duty as the cost-tiering router. - Off-the-shelf UI (Open WebUI or LibreChat) to move fast. - A 200-case golden set bootstrapped with the zero-label pipeline (log clustering + question-from-chunk synthesis); deterministic + judge evals in CI; nightly retrieval-recall run.

Phase 2 — Mid-size (harden & differentiate): - Migrate to Qdrant (filtered search at scale); add a reranker (Cohere 3.5 or Jina v3 on TEI). - Self-host the utility model tier on vLLM (rewrite/classify/judge/compress) — the biggest cost lever after routing; add the four-tier cache with permission-scoped semantic caching. - E2B/Firecracker sandbox for any code execution; egress allowlists; runtime secrets. - Agent registry with capability manifests + invocation ACLs + approval workflow. - User memory: working memory + compressed scratchpad + profile vault (the layers that matter); episodic recall as an on-demand tool. - Full error-handling suite: idempotency, backoff+jitter, circuit breakers, DLQs, saga/compensation, HITL approvals via LangGraph interrupts; per-query budget guards with graceful early exits. - Hand-rolled agentic UI with streaming plans, tool-call visibility, citations, partial-failure UX. - Drift monitoring (embedding sentinels, Arize Phoenix/Evidently); non-technical signals (thumbs-down, reformulation, abandonment); management dashboard with unit economics.

Phase 3 — Enterprise-grade (scale, compliance, trust): - RFC 8693 token exchange delegation + SPIFFE agent workload identity + MCP OAuth 2.1 for tool servers; Biscuit/macaroon attenuation for sub-agent chains. - Specialized multi-hop / temporal retrieval — added because your logs proved relationship or time-scoped queries fail on hybrid vectors, not speculatively. Same bar for exotic temporal memory. - Milvus if you cross billion-vector scale; Temporal if agent workflows span days with approval gates. - GDPR deletion propagation across index/embeddings/memory; PII detection; index versioning + shadow re-embedding on model changes. - SOC2/ISO-grade audit trails (every query, response, and document-access path logged); tenant isolation; incognito/no-retention paths. - Chargeback, model-tier entitlements, and cost governance enforced at the gateway; canary + online evals + A/B as the standard deploy path.

The one-sentence thesis: build the permission-aware retrieval and identity spine first, make the boring reliability parts (idempotency, sandboxing, evals-from-real-traffic, per-query cost budgets) non-negotiable, and add the exotic parts — specialized multi-hop retrieval, temporal memory, multi-agent, attenuated delegation — only when your own production logs prove you need them.

Caveats

• Fast-moving landscape. MTEB standings, reranker leaderboards, and gateway features shifted materially in 2025–2026; several cited data points come from vendor blogs and aggregators — re-verify against primary leaderboards and benchmark on your own corpus before committing. The FinMTEB result (best general model drops ~8.5 points on-domain) is the reason.
• All ₹/$ figures in Sections 13–14 are illustrative, built on round list-price assumptions to make the ratios (the 60x agent-loop multiplier, the utility-tier breakeven) legible. Re-run the math with your providers’ current rates and your actual token profiles; the ratios are durable, the absolute numbers are not.
• Vendor sources. Several comparisons (gateway, sandbox, memory, reranker rankings) are from vendors with a stake (Requesty, Agentset, ZeroEntropy, Northflank, etc.); claims are cross-checked where possible, but treat specific benchmark numbers as directional.
• Some cited features are recent or release-candidate (MCP 2026 revisions, Entra Agent ID, Auth0 for AI Agents GA, LangSmith Engine) — confirm GA status and API stability before you depend on them.
• Benchmark disputes are real — memory vendors (Mem0 vs Zep) publicly dispute each other’s LongMemEval/LOCOMO numbers; run evals on your own workload rather than trusting any single reported score.
• This is an architecture guide, not a security audit. The auth patterns (especially offline token attenuation) are genuinely contested; involve your security team and threat-model your specific deployment.