When you type a prompt in Claude Code, a loop starts. Nine conditions can keep it running automatically, without asking you for input. Most of them have nothing to do with whether your task is finished.

The loop lives in query.ts, Claude Code’s core agent logic file, exposed via an npm source map in March 2026 and since analyzed by several engineers in the community. One while(true) spanning over 1,400 lines of production TypeScript.

This post opens that loop.

A note on sources. The implementation details here are based on the Claude Code v2.1.88 source, exposed via an npm source map in March 2026. query.ts is not publicly documented. Everything in this post is community analysis, cross-referenced against the official Claude Code docs. Anthropic ships frequently, so the architecture is stable but specific line numbers are not.

The Blueprint:

1. Start here: the naive version, and what breaks it

2. The architecture: why an async generator, why single-threaded, and what the choice costs

3. Before you type anything: what loads at session start and what it spends

4. The nine conditions: each failure mode that grew the loop past 1,400 lines

Start here: the naive version

Every agent tutorial gives you this:

while True:
    response = call_llm(messages)
    if response.tool_calls:
        results = execute_tools(response.tool_calls)
        messages.append(results)
    else:
        break 

While that 'naive' approach works fine in a controlled demo - where the API is stable and the task is trivial, it hits a wall the moment it encounters the messy reality of production - where network jitter is a given and long-running scripts have a habit of hanging indefinitely.

Then production happens. The API times out mid-response. The context window fills after 20 tool calls. A bash script hangs with no exit condition and freezes the terminal. A governance hook needs to review the output before the session ends. The laptop sleeps. The session is gone.

Each of those is a line in query.ts. The production version handles all of them. The rest of this post explains how.

The loop is not the smart part

Worth stating before anything else.

query.ts is a state machine. It calls an API, reads the response, executes whatever tools the model requested, feeds the results back, and calls the API again. The intelligence is entirely in the model. The loop keeps things correct when production introduces conditions the model cannot handle on its own.

That framing matters for every design decision that follows.

The architecture

Claude Code is built on a JavaScript runtime (Bun, per the v2.1.88 source analysis) with TypeScript in strict mode. The UI is Ink, a React implementation that renders to terminal cells. The agent logic lives in query.ts.

The loop is declared as an async generator:

export async function* query(
  params: QueryParams,
): AsyncGenerator<
  | StreamEvent
  | RequestStartEvent
  | Message
  | TombstoneMessage
  | ToolUseSummaryMessage,
  Terminal
>

Two decisions are embedded in this signature.

Typed events, not buffered output. The loop yields events as they happen: text tokens, tool calls, tool results, compaction markers. Each one renders to the terminal immediately. The result is that the user sees output character by character, with no buffering between the API and the screen.

A typed exit reason. The loop returns Terminal, not void. When it exits, it tells the caller exactly why: task complete, context limit hit, user interrupted, budget exhausted. Session recovery, Remote Control reconnection, and resume behavior all depend on this.

The loop is single-threaded: one instance, one thread, one loop. There is no shared mutable state between concurrent operations, no locks, and no race conditions within a session.

This is a deliberate bet. Parallelism adds throughput, but it also adds shared state between concurrent tool executions. Shared state introduces a class of bugs that single-threaded designs cannot have.

Ultimately, Anthropic favored a design that prioritizes deterministic, single-threaded correctness over the throughput gains you might get from aggressive parallel execution.

The cost of that bet is specific. A bash tool that runs a tight loop, a synchronous operation that blocks for minutes, a script with no exit condition: none of these can be interrupted from within the loop. The event loop freezes until the operation completes or the user hits Ctrl+C. That is the production risk this architecture accepts.

↓ INTERNALS: Architectural Tradeoffs 1

Early versions of Claude Code used recursion. The query function called itself. In long sessions with hundreds of tool calls, the call stack grew until it exploded. The current design carries mutable state in a State object between iterations. Each continue is a state transition, and a transition field records the reason, so error recovery and compaction logic know what happened in the previous turn.

↓ INTERNALS: Architectural Tradeoffs 2

The async model is cooperative multitasking, not preemptive. Tasks yield control voluntarily at await and yield points. While the loop waits on an API response, the same thread handles terminal rendering, keypress events, and JSONL session writes. Nothing runs simultaneously; everything is interleaved. This keeps the terminal responsive even during long tool executions.

Before you type anything

Before you ever see a prompt cursor, Claude Code performs a complex ten-step initialization.

It loads settings in a specific priority hierarchy with managed policies acting as the immutable overrides, followed by a recursive walk up your directory tree to hunt for project-level rules, memory files, and MCP configurations.

Only after aggregating all that state does it finally assemble the system prompt.

The cost before you type a single character: approximately 8,000 to 12,000 tokens. On a 200K context window that is 4 to 6%. On a 32K window it is 25 to 37%, which constrains what tasks are feasible before the first token of actual work.

One detail worth noting: CLAUDE.md injects as a user message, not part of the system prompt. The model treats system prompt content as configuration and user message content as context. That distinction affects how strictly it follows CLAUDE.md instructions.

↓ INTERNALS

The system prompt is not a single string. Claude Code assembles it from conditional fragments based on your environment, settings, and active context. The base system role definition is approximately 2,900 tokens. Tool definitions add roughly 3,000 more for 18+ tools. CLAUDE.md content adds between 500 and 2,000. The assembled prompt is a dense technical document: behavioral rules, tool usage philosophy, coding style guidelines. The line “three similar lines of code is better than a premature abstraction” is stated in the system prompt. This is the model’s operating manual.

Two models, one pipeline

Every reasoning turn runs on an Opus-class model (claude-opus-4-6 per the v2.1.88 analysis, though this may shift across versions). Opus sees the full system prompt, the complete tool catalog, and the conversation history. It decides what to do next.

A smaller, faster model handles exactly two background tasks.

A warmup request fires at session start: one intentionally truncated API call with stop_reason: max_tokens. The response content is irrelevant. This is a health check, verifying the API is reachable and the quota is valid.

File path extraction runs after bash commands produce output, identifying which file paths appear in the results. The prompt is concise and single-purpose (179 words in v2.1.88).

Every reasoning turn goes through Opus. The smaller model handles only those two tasks.

↓ INTERNALS

Claude Code uses aggressive prompt caching. Content blocks carry cache_control breakpoints. The first pass writes to a server-side cache at 1.25x input cost. Subsequent requests with matching prefixes pay approximately 10% of normal cost. In observed session traces, 90% or more of tokens per turn are served from cache. Without caching, the economics of long sessions would be brutal. Prompt caching is what makes the model feel like it “remembers” the session despite being stateless on every API call.

The nine conditions

The loop terminates when a terminal condition is met. Nine conditions cause it to run another iteration automatically, without asking the user. Each started as a bug report.

① Tool result received. The model calls a tool, which executes and injects its result back into context. The loop continues so the model can decide what to do with it. This is the core of the agentic pattern.

② API error or network failure. When the request fails, the loop retries with exponential backoff, up to 10 times. The user sees nothing.

③ Max output tokens mid-response. The model hit its output limit before finishing. The API signals this with stop_reason: max_tokens, and the loop continues to let it complete.

④ Token budget warning injected. As the session approaches the context limit, the loop injects a warning telling the model to start wrapping up, then continues so the model can actually read it.

⑤ Proactive compaction triggered. When context is full, one of four compaction strategies runs and the loop continues with the compressed result. The pipeline is covered in the next section.

⑥ Reactive compact after 413. The API rejected the request: context too large. The proactive strategies ran and were not enough. Reactive compact runs a full context collapse. The full mechanics are in the next section.

⑦ Sub-agent Task tool completes. When a child agent spawned via the Task tool finishes, the parent loop continues. Sub-agents are implemented as tools, so the parent loop cannot distinguish between waiting on a child agent and waiting on a file read. The tradeoff: the parent receives only a summary of the child’s work (estimated at 1,000 to 2,000 tokens in the source analysis), so intermediate steps and decisions inside the sub-agent are invisible to it.

⑧ Stop hook signals continue. When the model signals it is done, a user-defined stop hook can override that and keep the loop running. Stop hooks are a governance layer the model cannot override.

⑨ Pre-tool hook blocks execution. A hook intercepted a tool call before execution and denied it. The denial injects as a tool result. The loop continues so the model can reason about the denial.

The compaction pipeline

When the context window fills, the loop does not crash. It compacts.

Four proactive strategies run in order, cheapest first. Tool Result Budget truncates oversized tool outputs with no model call needed. Snip Compaction removes older tool results from history, also without generation. Microcompaction summarizes middle conversation turns one at a time using a scoped model call. Autocompact fires at a usage threshold, first attempting Session Memory Compaction (extracting key facts into a compact object) and falling back to full conversation compaction if that is not enough.

The ordering is not arbitrary. You exhaust the cheap options before paying for the expensive ones.

Reactive compact is separate. It fires only after the API returns 413, which means the proactive strategies have already run and were not enough. The loop collapses the full context and continues. The circuit breaker hasAttemptedReactiveCompact ensures this fires once only; a second 413 is a terminal condition.

Think of it as a two-stage safety system: proactive compaction keeps you under the limits during normal operation, while the reactive cycle acts as your emergency circuit breaker when things inevitably go sideways.

↓ INTERNALS

Sub-agents are context-efficient for a specific reason. A sub-agent runs its own full loop internally, potentially consuming 100K or more tokens on a complex task, then returns only a compact summary to the parent. The parent pays for the summary, not the full work. This is why the Task tool is the right primitive for parallelizing work in a single-threaded system: each sub-agent has its own isolated context window.

Session persistence

Every message, tool call, and result writes to a JSONL file under ~/.claude/projects/ in real time, as each event happens rather than at session end.

Three things depend on this.

claude --resume reconstructs exact conversation state at the point of interruption. --fork-session copies history into a new session at any chosen point, leaving the original unchanged. Remote Control reconnects automatically if the laptop sleeps and wakes, because session state is on disk rather than in memory.

The TombstoneMessage type in the generator signature connects to this log. When compaction removes messages, their tombstone stays in the JSONL so the replay log stays consistent after compression.

How other loops handle the same problems

The naive loop has none of the nine conditions. Context overflow crashes it. Network failures terminate it. It is the right starting point, not a production architecture.

Hermes Agent (Nous Research, MIT license) takes a different position on parallelism. When the model requests multiple tools, Hermes executes them through a thread pool rather than sequentially. The throughput gain is real. So is the exposure: two tools executing in parallel can write to the same file at the same time. Race conditions in tool output are a category of bugs that Claude Code’s single-threaded model cannot produce.

LangGraph is not an agent loop. It is a framework for constructing them. Human-in-the-loop pauses are explicit: interrupt() stops execution, Command(resume=value) continues it. Claude Code’s permission system is implicit, handled internally with no graph definition required. The explicit approach is more debuggable; the implicit one requires less setup. Hermes is model-agnostic across providers; Claude Code is tied to Anthropic’s API.

The loop is not the smart part of Claude Code. The model is.

query.ts exists to keep the model correct when production introduces conditions the model cannot handle on its own: lost connections, context limits, governance hooks, failed tool calls, slow networks.

Every line beyond that naive version exists because something failed in production.

If your agent loop is shorter, you have not hit those failures yet.

INTERNALS.md is a technical series on how production systems actually work. No tutorials. No framework evangelism. Just the layer beneath.

If this was useful, the best thing you can do is share it with one engineer who would care.

Share INTERNALS.md

Sources

• Claude Code official docs: how-claude-code-works, agent-sdk/agent-loop, remote-control, channels

• Harrison Guo: Claude Code Deep Dive Part 2 — The 1,421-Line While Loop

• Jonas Kim: Claude Code Architecture Analysis (v2.1.88 source analysis)

• Zain Hasan: Inside Claude Code — An Architecture Deep Dive

• Alex (AgenticLoops): Disassembling AI Agents Part 2 — Claude Code

• VILA-Lab: Dive into Claude Code

• Arize AI: How Hermes Implements an Open-Source Agent Harness Architecture

• Ken Huang: Chapter 1 — The Harness Paradigm

Here’s a bug that doesn’t show up in your logs.

You ship a RAG system. Users ask questions about internal data: support tickets, product docs, sales notes. Cosine similarity scores come back at 0.74, 0.81, 0.78. The LLM generates a confident, fluent answer.

While the resulting answers aren’t always obviously broken, they fail in a specific, repeatable pattern.

• Someone asks about “pipeline” and gets sales documents when they meant data infrastructure.

• Someone asks about “incident” and gets both engineering postmortems and customer support tickets, randomly.

• Someone asks about “ARR attainment” and gets a document about spreadsheet formulas.

You tune the prompts. You adjust chunk sizes. The results are still wrong.

Prompt engineering and chunking strategies fail here because the root cause lies in how the foundational vector space is created.

A note before we start. This post assumes you’ve shipped or worked on a RAG system and have firsthand experience with it underperforming on domain-specific queries. If you need a foundation first, this primer is a solid 10 minutes. You’ll get more from this post with that context.

The Blueprint:

• The Illusion: Why your embedding model is a map of the internet, not an understanding machine.

• The Geometry: How high-dimensional space pathology compresses your similarity scores.

• The Failure Modes: How to diagnose and fix the 4 silent bugs killing domain retrieval (including Hubness and Concept Collision).

• The Playbook: A 3-step engineering roadmap to evaluate, adapt, and fine-tune your space.

What you think is happening

Most engineers picture an embedding model as a kind of understanding machine.

You feed it text. It reads the text, grasps the meaning, and produces a number that represents that meaning.

Two pieces of text with similar meanings get similar numbers. You compare numbers. You find meaning.

This mental model feels right. It explains why “cat” and “feline” end up close together. It explains why the system works at all.

But this mental model is wrong, and that’s exactly why several production RAG pipelines fail.

Maps, not minds

An embedding model doesn’t understand anything. They possess zero semantic comprehension. They strictly operate as coordinate systems - a map of language drawn by learning how words and phrases appeared in the internet.

Every piece of text you feed it gets assigned a location on that map. Texts that appeared together constantly, in the same articles, answering the same kinds of questions, end up near each other. The model never gets to redraw the map when it sees your internal data. It just projects everything onto the existing one, using whatever surface patterns and statistical echoes it recognizes.

The crucial part: the map was drawn by reading the internet.

Billions of web pages, Wikipedia articles, Reddit threads, news posts, academic papers. It’s a dense, detailed map of how language is used on the open web. This is why it works well for general questions. “Cat” and “feline” appeared near each other constantly. “Paris” and “capital of France” showed up together in thousands of articles.

But your company’s specific use of “pipeline”, “incident”, “P0”, or “ARR attainment”? Those meanings were never on the original map. The model does the only thing it can: it finds the nearest coordinates it does have. It always returns something. There is no “I don’t know”.

Here is the part that makes this dangerous: the model never warns you. It returns a confident-looking coordinate and a plausible similarity score regardless of whether your data falls within the model’s training distribution or unmapped domain. A 0.79 similarity score looks identical for both a perfectly relevant retrieval and a catastrophic silent failure.

The cosine score only tells you distance on the map. It doesn’t tell you whether the map covers your territory.

The map has a geometry problem

Even when you’re asking about something the model did learn from the internet, the similarity scores can mislead you. And the reason has nothing to do with meaning.

When you compare two embeddings, you’re computing the angle between them. Small angle = similar. Large angle = different. This is called cosine similarity.

In 2D, this works well. Arrows can point in many directions. Two random arrows have a wide variety of angles, so you can clearly tell similar from different.

Now go to 768 dimensions. Something counterintuitive happens.

In raw high-dimensional spaces, the expected angle between random vectors concentrates near 90°. Modern contrastive objectives (the training method used by OpenAI, Cohere, BGE, and similar models) push useful variance into the tails and partially fix this. But when your data distribution differs from what the model was trained on, the collapse returns in practice: your domain queries end up in a poorly-calibrated corner of the space, and the scores stop being informative.

There’s a second problem on top of this.

Research from 2021 (Timkey & van Schijndel) measured exactly which dimensions drive cosine similarity scores in pre-contrastive BERT-style models. Their finding: just 1 to 3 dimensions out of 768 dominate the entire similarity calculation. The other 765 barely matter.

These are dimensions with unusually high variance that spread wide across the corpus. Because cosine similarity is an angle calculation, whatever dimensions spread widest have the most influence. The good news: simple post-processing (subtracting the mean, standardizing) largely mitigates rogue dimensions, and modern contrastive models have improved significantly. The caveat: when you’re running a general model on out-of-distribution domain data, these geometric pathologies re-emerge in subtler forms. You won’t know unless you measure.

Your data is a different country

Now put both problems together.

The map was drawn from the internet. The map’s geometry loses resolution in high dimensions. And your users are querying for domain-specific contexts that never existed in the base model’s training corpus.

In practice: your model confidently navigates general topics. Common business language, standard industry terms, broad concepts. The map is detailed there. But when your users ask about your company’s specific vocabulary, the model has no coordinates for those meanings. It falls back to surface patterns: which words the document contains, what those words meant on the internet.

A 2024 study tested seven state-of-the-art embedding models on financial domain text: SEC filings, earnings calls, analyst reports. Every model performed significantly worse on domain text than on the general benchmark. More striking: a model’s general benchmark score had almost no correlation with its domain performance. The rankings reshuffled completely.

The general benchmark tells you how well the map covers the internet. It says nothing about your territory.

Four ways the wrong map shows up

These failure patterns look different on the surface. Underneath, they’re all the same problem.

1. All your similarity scores look the same

You query for something very specific. You query for something very general. The scores come back in the same narrow range regardless (say, 0.62 to 0.79) and don’t separate relevant from irrelevant.

What’s happening: your embedding space is compressed. All vectors crowd into the same corner. The metric can’t discriminate between meaningfully similar and meaningfully different.

Diagnostic: compute cosine similarity on 1,000 random document pairs. Standard deviation below 0.1 = compressed space.

Fix: switch to a model trained with contrastive objectives. This is a geometry problem. A configuration change won’t fix it.

2. The same word retrieves documents from the wrong category

“Pipeline” returns sales content when the user asked about data infrastructure. “Incident” mixes engineering postmortems/CoE’s with support tickets. Domain terms with one specific meaning in your organization keep returning documents from a different context.

What’s happening: concept collision. The model placed these terms at their web-scale coordinates, where they overlap multiple meanings. Your organization uses them narrowly; the training data used them broadly.

Diagnostic: for your five most domain-specific terms, manually inspect the top 10 retrieved documents. If a single-meaning term consistently returns two or more semantic categories, you have concept collision.

Fix: fine-tuning with pairs from your corpus. The training process will encounter these collision cases naturally and learn to separate them.

3. The same few documents appear in every result set

Across many different queries, five or ten documents keep showing up in your results. Not always obviously relevant. I once watched a model return the same three runbooks for every “incident” query. Scores looked great, all above 0.75. Turned out those runbooks were long, covered every keyword, and sat geometrically near the center of the cluster. Classic hub. One line of code fixed half the pain.

What’s happening: hubness. In high-dimensional compressed spaces, some vectors land near the geometric center of the distribution. These documents become close neighbors to almost everything. Not because they’re relevant, but because of where they sit geometrically.

Diagnostic: log which document IDs appear in top-10 results across 100 diverse queries. A small number appearing more than 20 times is a hub problem.

Quick fix:

embeddings -= embeddings.mean(axis=0) 

before indexing. Pushes hubs away from the center. Often produces immediate improvement.

Proper fix: switch to or fine-tune a model with better geometry.

4. Scores look fine but answers are wrong

Cosine scores are in a reasonable range. The LLM generates coherent text. But the answers are consistently, subtly wrong in ways no automated metric catches.

This is the hardest one to diagnose because everything in your pipeline looks healthy.

What’s happening: the model navigated confidently to the nearest coordinates it knows, which aren’t where you needed to go. The score is high because these really are the nearest neighbors on the map. They’re just neighbors on the wrong map.

Diagnostic: build 50 to 100 (query, relevant document, irrelevant document) triplets, manually curated. Measure NDCG@10. Below 0.5 on domain queries is a clear signal.

Fix: domain adaptation. Fine-tuning, or switching to a purpose-built model for your vertical.

Drawing a better map

Three steps. In order. Every time.

Step one: your own labeled eval set is the only reliable compass.

Build 100 to 200 labeled triplets from your corpus: real queries, real relevant documents, real irrelevant documents, curated by someone who understands your domain. Run your current model. Measure NDCG@10. (Several python library already provides out of the box NDCG calculation).

This number is your baseline. Not MTEB. Not cosine scores. This number, on your data. If NDCG@10 is above 0.6, your embedding model probably isn’t the bottleneck. Check chunking strategy, query preprocessing, and reranking first.

Step two: check if a domain-specific model exists.

For finance, biomedical, and legal: purpose-built models exist and have been empirically validated. FinBERT, PubMedBERT, legal-BERT variants. Start there and measure on your data.

For everything else: look at the MTEB retrieval sub-score specifically, not the overall average. The overall average blends in clustering and classification tasks that don’t predict RAG performance. A model that scores 55 on retrieval and 70 overall beats one scoring 45 on retrieval and 75 overall, every time.

Step three: when to fine-tune.

Fine-tune if: NDCG@10 is below 0.5, no domain model exists, and you have at least 1,000 to 5,000 (query, relevant document) pairs available.

No labeled pairs? Generate them synthetically: prompt an LLM to produce 3 to 5 realistic queries per document in your corpus, then mine hard negatives using your current embedding model. Teams using this approach on specialized domains consistently see 10 to 30% NDCG improvement.

Don’t fine-tune if: you have fewer than 500 training pairs, or your measurement says the problem is elsewhere.

One strong opinion: if your domain has heavy internal jargon, proprietary ontology, or vocabulary that means something different inside your company than it does on the web, start with fine-tuning or a domain model.

Attempting to bridge this gap purely with prompt engineering and general-purpose embeddings introduces tech-debt that scales linearly with your data complexity.

A few gotchas before you go.

High-cardinality repetitive data (think: millions of similar support tickets) makes hubness worse even after fine-tuning. The “same few documents everywhere” problem is structural when your corpus has too many near-duplicates.

Multilingual or code-heavy domains are harder than they look. A fine-tune on English financial text won’t help your Japanese runbooks or Python error traces.

And if your domain evolves fast, a startup changing its product every quarter for example, fine-tuned embeddings drift quickly. Re-training cadence and embedding versioning become a day-2 problem many teams underestimate. Weigh that cost before committing.

↓ Internals: how fine-tuning reshapes the map

Fine-tuning uses contrastive training: you give the model (query, relevant document, hard negative) triplets and train it to pull relevant pairs closer while pushing hard negatives apart.

The objective function is called InfoNCE. It frames learning as: given a query and a batch of documents, identify the correct one. The temperature parameter (τ, typically 0.05) controls how sharply the model focuses on near-misses, the documents that look like the right answer but aren’t. Low temperature = sharp focus on the hardest confusions in your corpus.

Hard negatives from your corpus are what make fine-tuning work. They teach the model the exact distinctions your users encounter, not generic web-scale distinctions. This is why in-domain hard negatives produce dramatically better results than random negatives.

The geometry side effect: contrastive training spreads vectors more uniformly across the space, reducing anisotropy. A good fine-tune with in-domain hard negatives fixes three of the four failure modes simultaneously: score compression, concept collision, and hubness.

What survives

• Embedding models are coordinate systems, not reasoning engines. The map was drawn from the training corpus. It won’t redraw itself for your data.

• Your cosine score measures distance on that map. A confident-looking score only tells you those two points are close together. It says nothing about whether the map covers the territory your users are asking about.

• Your own labeled eval set, built from your data, is the only reliable compass. Every other decision follows from that measurement.

INTERNALS.md is a technical series on how production AI systems actually work. No tutorials. No framework evangelism. Just the layer beneath.

If this was useful, the best thing you can do is share it with one engineer who would care.

Share INTERNALS.md

Sources

• All Bark and No Bite: Rogue Dimensions in Transformer Language Models, Timkey & van Schijndel, 2021

• SimCSE: Simple Contrastive Learning of Sentence Embeddings, Gao, Yao & Chen, EMNLP 2021

• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere, Wang & Isola, 2020

• Contrastive Representation Learning, Lilian Weng

• Do We Need Domain-Specific Embedding Models?, Tang et al. (FinMTEB), 2024

• Hubness Reduction Improves Sentence-BERT Semantic Spaces, 2023

• Why, When and How to Fine-Tune a Custom Embedding Model, Weaviate

• Hard Negative Mining, sentence-transformers documentation

  ---

A skill is a small program.

It has three execution stages: 1\ what loads every turn, 2\ what loads on invocation, and 3\ what loads on demand. Because a skill is a program, it suffers from typical software rot—environment drift, version sensitivity, and silent, non-reproducible failures.

You’ll see these failures in specific shapes. A skill that cost 20% of your context window, silently, before the agent did any work. A skill that worked perfectly until you shared it with a teammate, and ran the build in the wrong directory. A skill tuned carefully on one model, producing worse output the moment you upgraded to a better one.

These aren’t separate bugs. They’re four faces of the same misunderstanding: treating a loader specification like a prompt.

This post is about what skills actually are underneath, and why understanding the runtime changes everything you do at the surface.

A note on scope. Skills aren’t a Claude-only thing anymore. Anthropic published the SKILL.md format as an open standard in December 2025, and the same files now work across Claude Code, Kiro, Cursor, Codex CLI, and others. The mental model in this post applies to all of them. I’ll use Claude as shorthand for the agent harness reading the skill. Swap in your runtime of choice.

What skills are not

The first time I wrote a Skill, I thought I was writing a long prompt the agent would consult.

I wrote one big SKILL.md. Maybe 1,200 lines. Workflow at the top, a map of every module in our codebase, example code, message contracts between services, framework-specific patterns, and at the bottom a list of every gotcha I knew. It worked. It also consumed about 20% of the context window before the agent did any actual work.

I rewrote it. Same instructions, same output, different architecture: a 180-line SKILL.md that pointed at three reference files and one helper script. The new version cost 7%.

The instructions didn’t change. The architecture did. That’s where the 3× difference lived, and it was the first sign that I was not, in fact, writing a long prompt.

A prompt is static text. You write it, you ship it, the model reads all of it on every turn. Skills don’t work like that. Skills are a loader specification. You’re describing what should be in context, when, and at what cost. The text matters, but the structure decides what survives the trip into the model’s working memory.

That reframe is the whole post. Everything else falls out of it.

A real skill restructure. Same task, same model, same output. The 3× difference came from where the instructions lived, not what they said.

The runtime

Skills run on a principle Anthropic calls progressive disclosure. The official documentation defines it plainly:

Skills can contain three types of content, each loaded at different times.

This is why two skills with identical instructions can behave completely differently. One loads 180 lines on demand; the other dumps 1,200 lines every turn.

Anthropic built these levels to protect your context window. If a skill front-loads everything, it crowds out the conversation history and tool outputs. By using progressive disclosure, you stop paying for “just in case” instructions and only pay for “just in time” execution.

Each level loads at a different time, with a different cost. Most authors put everything at Level 2.

Level 1: Metadata. The name and description from YAML frontmatter. Always loaded, every turn. The official docs put this at roughly 100 tokens per skill installed. The agent uses the description to decide whether the skill is relevant. It’s a routing decision, not a usage decision. This is the most important level to get right. If the description is wrong, nothing else matters.

Level 2: SKILL.md body. The procedural instructions. Loaded only when the agent decides the skill applies, by reading the file via bash. Anthropic’s best practices documentation puts the recommended ceiling at 500 lines. This is where most people pile on content they shouldn’t.

Level 3: References and scripts. Bundled files referenced from SKILL.md. References are markdown the agent reads only when the body points to them. Scripts are executable code the agent runs: output enters context, the source code does not. Effectively unlimited.

The Anthropic engineering team (Barry Zhang, Keith Lazuka, and Mahesh Murag) described it in their October 2025 announcement as: “Like a well-organized manual that starts with a table of contents, then specific chapters, and finally a detailed appendix, skills let Claude load information only as needed.”

Get the architecture right and your skill costs almost nothing until it earns its place. Get it wrong and you pay every turn.

Mental model

Picture a kitchen during dinner service.

The chef’s attention is the scarce resource. Same as the agent’s context window.

There’s a pinboard on the wall with recipe titles and one-line summaries. Pasta Carbonara: Italian classic, use when guest wants creamy pasta with bacon. The chef glances at it constantly. It’s small enough to hold in peripheral vision. That’s frontmatter.

When a guest orders, the chef picks the matching card and pulls down the full recipe. Ingredients, steps, technique notes. The recipe is not on the wall. It would be too cluttered, too distracting, too much to scan during service. It comes down only when needed. That’s SKILL.md.

The recipe sometimes says for the sauce, see Sauce Reference, page 47. The chef walks to the binder, opens to page 47, reads only that page. Doesn’t read the whole binder. That’s references/.

In the corner, a stand mixer. The recipe says use the mixer for three minutes. The chef does not read the mixer’s circuit diagram. The chef hands it ingredients, presses a button, gets output. That’s scripts/.

The metaphor holds under pressure, which is the only test of a metaphor. Every failure mode I hit in my own skills traces back to violating the kitchen.

The Antipattern Ledger

When I first started migrating my workflows to the SKILL.md, I treated the runtime like a smart intern who could “just figure it out.”

I was wrong. Because the skills runtime is a deterministic loader, minor architectural choices—like where you put a single line of YAML—can silently break the agent’s reasoning. These aren’t just bugs; they are antipatterns. Each one below represents a moment where I violated the “Kitchen” logic and paid for it in context drift, high latency, or hallucinated outputs.

Frontmatter on reference files

The first thing I got wrong, before I understood progressive disclosure existed.

I added YAML frontmatter to my reference files because SKILL.md had it, and the references felt important enough to deserve metadata. I didn’t realize what frontmatter actually does.

Frontmatter is what gets loaded into the system prompt at startup. Every file with frontmatter contributes its name and description to the always-loaded set. The pinboard. Adding frontmatter to a reference file pins it to the wall as if it were a top-level skill. It isn’t. Now the pinboard shows fifty entries instead of five, most of them sub-pages that were never meant to be visible at routing time.

In practice: the agent would occasionally trigger a reference file directly instead of the parent skill. Instructions out of context, without the skill body that gave them meaning. The output was subtly wrong and I couldn’t figure out why, because the reference file looked fine in isolation. I didn’t realize it had been promoted to skill-level visibility.

The fix was one line per file: delete the frontmatter from references. They’re not skills. They’re chapters that other skills point to.

One monolithic skill

This is the 20%-to-7% story.

When I built a skill to capture context across multiple modules and message systems, I put everything in one SKILL.md. It seemed cleaner. One file, one source of truth. Easy to read, easy to edit.

It also meant that every time the skill triggered, the agent loaded the entire 1,200-line file. Module map, contracts, patterns, and gotchas. Even when the task only needed two of those four.

Splitting it into a 180-line spine with three reference files dropped context consumption from 20% to 7%. Same task, same output, same model.

This compounds. A skill that costs 7% instead of 20% means you can install three of them in the same context budget, run longer sessions before compaction, hit fewer cliffs on long-horizon tasks. The savings aren’t local. They show up everywhere downstream.

Hardcoded workspace paths

I shared a skill with a teammate and it ran the build command in the wrong directory.

My instructions said something like navigate to modules/web and run the build. That worked in my repo. My teammate’s repo had four modules. modules/web didn’t exist; they had packages/frontend/web. The skill silently picked the wrong directory and produced output in the wrong place. No error. Just wrong output.

The fix was to write instructions that ask the agent to discover the right path rather than declare it. Search for the build configuration. Identify the module by its package.json. Read the workspace structure before assuming. The skill became more abstract, but it became portable.

This is the failure mode that doesn’t appear until you share. If you only ever run a skill on your own machine, you can hardcode anything and it will work. The moment another engineer runs it, every implicit assumption surfaces as a bug.

Missing gotchas

My monorepo uses Turborepo. The build command has to run from the repo root for the configuration to resolve correctly. If you run build from inside a module directory, the build still runs. But the cache misses, the dependency graph gets misread, and the output is subtly wrong.

The agent’s default was reasonable: I’m working in the web module, so I’ll run the build from the web module. That’s correct in 90% of repos. It was wrong in this one.

No amount of “explain the why” in the instructions would have prevented it. The wrongness wasn’t conceptual; it was environmental. The agent’s prior was correct on average. My environment wasn’t average.

The fix was a single line in a Gotchas section: Always run turbo build from the repository root, never from inside a module. One line. The next time the agent reached for the build command, it consulted the gotcha and ran correctly.

This is what Gotchas are for. The agent has reasonable defaults. Your environment isn’t average. That gap is the whole job of the Gotchas section, and it’s why mature skills treat it as the most important section to maintain over time.

Not knowing why the skill worked at all

The deepest mistake. I didn’t write evals.

I built a writing skill for my personal Claude desktop. It was based on Scott Adams’ writing principles: short sentences, active voice, front-loaded points, one idea per paragraph. I tuned it on Sonnet 4.6. It worked exactly the way I wanted: drafts came out clean, direct, in my voice.

Then I upgraded to Opus. Better model, I assumed. Better output.

The output was worse. Every sentence ran 5 to 7 words. Technically short. But choppy. No rhythm, no flow, nothing that read like me. The writing felt like bullet points dressed as prose.

What happened is subtle. Sonnet read “write short sentences” and applied judgment: short where brevity sharpened the point, longer where the rhythm needed it. It understood the spirit. Opus read the same instruction and followed it literally. Every sentence, hard constraint, no exceptions.

The more capable model has stronger priors about what “good writing” looks like. Its version of clear prose is the statistical center of good writing on the internet. My voice isn’t the statistical center. Opus pulled hard toward its own aesthetic, and away from mine.

A skill tuned on one model is calibrated to that model’s compliance characteristics, not just its capabilities.

A more capable model isn’t automatically a better fit. Sometimes it’s worse, because it interprets your instructions instead of following them.

I had no evals. No way to know how much had drifted, which instructions were being over-applied, or what a passing output even looked like quantitatively. I’d never defined what “sounds like me” meant in terms a test could check.

Anthropic’s skill-creator, the tool the team uses to build their own skills internally, has an explicit eval methodology. The core move is paired runs: for every test prompt, run the agent twice. Once with the skill, once without. You’re not measuring whether the output is good. You’re measuring whether it’s better than baseline, and by how much.

For a writing skill, not all assertions are scriptable. But some are: output length, sentence count, average sentence length, readability score. The rest is structured human review, with the previous output alongside the new one and a notes field. That’s what Anthropic’s eval-viewer in skill-creator produces.

I now keep a small 'Golden Set' per skill—a practice we’ll dissect in an upcoming post on automated skill validation—to ensure my voice doesn't drift when the underlying model changes. Three or four realistic prompts. Rerun the suite on every model bump, every skill edit. Check the deltas.

It worked when I tested it is not evidence. It’s the absence of measurement.

What survives the post

Four things should stick.

Skills are loader specifications, not prompts. Frontmatter is a routing mechanism. SKILL.md is a triggered payload. References and scripts are deferred chapters. Once you see the architecture, every authoring decision becomes a question of which level does this content belong at?

Architecture decides cost. The same instructions, in the wrong shape, can consume 3× the context window. That penalty compounds across every skill installed and every turn taken. The fix is structural, not prose-level.

The agent has reasonable priors. Your environment doesn’t. Gotchas exist because the model’s defaults are correct on average and your situation isn’t average. Workspace paths, build systems, team conventions: none of it lives in the model’s training. It has to live in the skill.

A model upgrade is not free. A skill tuned on one model is calibrated to that model’s compliance characteristics. A more capable model interprets your instructions instead of following them, and for skills that encode personal or organizational voice, that interpretation is the failure. The only way to know if an upgrade helped or hurt is to measure it.

INTERNALS.md is a technical series on how production AI and Data systems actually work. No tutorials. No framework evangelism. Just the layer beneath.

If this was useful, the best thing you can do is share it with one engineer who’d care.

Share

Sources

• Agent Skills overview, Claude API documentation

• Agent Skills best practices, Claude API documentation

• Equipping agents for the real world with Agent Skills, Barry Zhang, Keith Lazuka, Mahesh Murag, Anthropic Engineering, October 2025

• skill-creator/SKILL.md, Anthropic skills repository

• Agent Skills open standard, December 2025

• The Day You Became a Better Writer, Lakshmanan Meiyappan

• Scott Adams’ original post, via Internet Archive

Your agent worked yesterday. Today it’s returning wrong answers. No stack trace. No failed tool call. No exception. Just quietly incorrect output, shipping to production.

Here’s what probably happened. You added a node that writes to the same state key as an existing one, and LangGraph’s execution engine ran both in parallel. One write clobbered the other. The state is corrupt. Your agent looks fine because technically, it is running. It’s just running on a lie.

This post is about the engine that makes that possible and the one decision that prevents it.

Why Graphs Won the Agent Runtime

DAGs (Directed Acyclic Graphs) are elegant when data flows in one direction. The moment an agent needs to retry, re-plan, or loop through tool calls until some condition holds, they fall apart. A ReAct loop is not a DAG. It’s a cycle.

And cycles are what production agents actually do.

LangGraph’s answer is to model agents as cyclic graphs with typed state. Nodes compute. Edges - including conditional ones decide where execution goes next. State carries everything across steps. This isn’t a convenience; it’s the minimum structure that lets an agent loop without losing what it learned on the last pass.

The tradeoff is explicitness. You declare the schema up front. You declare the edges. The graph is compiled before it runs. In return, you get something most frameworks can’t give you: a runtime that pauses, resumes, replays, and parallelizes cleanly because the engine knows, precisely, what the graph is.

The Real Primitives: Actors and Channels

The public API shows you StateGraph, nodes, and edges. The engine underneath doesn’t work that way.

Before we go technical, here’s the mental model that makes everything else click.

💡 Mental Model: The Autonomous Pancake House

Stop thinking of a graph as a Boss shouting orders at Employees (function calling). Instead, imagine a kitchen that runs entirely on mailboxes (message passing).

• The Channels (Mailboxes): There is a “Batter” mailbox and a “Plates” mailbox. They aren’t just boxes, they have rules written on the lid.

• The Actors (Specialized Cooks): The Fryer cook doesn’t wait for a command. She sits by the “Batter” mailbox. The moment batter appears, she wakes up, cooks it, and drops the result into the “Plates” mailbox.

• The Reducer (The Lid Rule): What if two cooks drop a pancake onto the same plate at once?

  • No reducer: The plate shatters; it only expects one item. (InvalidUpdateError)

  • With reducer (operator.add): The rule on the lid says “Stack them.” Both pancakes land, and breakfast is saved.

In this kitchen, no one talks to each other. They only talk to the mailboxes. This is why LangGraph can pause, resume, or run ten cooks at once; the state is in the mailboxes, not in the cooks’ heads.

That's the mental model. Here's what it maps to in the engine.

Under the hood, LangGraph runs on a model borrowed from Google’s Pregel paper. Its real primitives are actors and channels. Actors called PregelNode internally subscribe to channels, read from them, and write to them. Channels hold values. Reducers decide how those values update when multiple writes arrive in the same step.

Here’s the reframing that matters: a state key in your StateGraph is a channel. A reducer is that channel’s update function. When you annotate a field with operator.add, you’re configuring the channel to append on update. When you leave it unannotated, the channel overwrites - and if two actors write to it in the same step, the engine throws.

So “state” is not a dictionary that nodes mutate. State is a set of channels, each with its own update semantics. Nodes don’t call each other; they publish to channels. Other nodes subscribe. This is message passing, not function calling.

Supersteps: The Execution Model

LangGraph executes in discrete steps called supersteps, and each one has three phases:

1. Plan: the engine inspects channel state and selects which actors to run

2. Execute: selected actors run in parallel

3. Update: writes are merged into channels through the reducers

The crucial property is that a superstep is transactional. If any actor in the step raises an exception, the entire step’s writes are discarded. None of the parallel results land. This isn’t a bug - it’s the guarantee that makes checkpointing meaningful. You never observe a half-applied superstep.

Selection is deterministic, too. An actor fires only when a channel it subscribes to has new data. The engine loops until no actor has pending work, or a step limit is hit. This is Bulk Synchronous Parallel - the same model that powers Apache Spark’s graph processing layer.

Two consequences fall out of this design, and both matter at 2am in production.

First, parallelism is free when nodes are independent. If two nodes subscribe to the same input channel and write to different channels, they run in the same superstep with no extra configuration. The engine figures it out.

Second, concurrent writes to the same channel need a reducer. Without one, the engine has no way to know which write wins - so it refuses to guess.

That second consequence is where most production bugs live.

The Silent Corruption Problem

Here’s the error the runtime throws when you get it wrong:

InvalidUpdateError: At key 'todos':
Can receive only one value per step.
Use an Annotated key to handle multiple values.

This error is loud. It fires immediately. You see it, you fix it.

The silent version is worse.

If your graph has a single path today and no concurrent writes, the overwrite default works fine. You never see the error. Then weeks later - you add a parallel branch. A fan-out pattern, a subagent, a retry. Suddenly two nodes land writes in the same step. If the bug fires only intermittently, you get something uglier than an exception: wrong answers in production, with no trace of why.

The fix is one line:

from typing import Annotated
import operator

class AgentState(TypedDict):
    messages: Annotated[list, operator.add]  # append, do not overwrite
    todos: Annotated[list[Todo], operator.add]

This isn’t theoretical. It’s still happening in production frameworks. As of November 2025, running the official research example in deepagentsjs throws this exact error - the todos state key has no reducer, so the underlying channel falls back to LastValue, which refuses concurrent updates. The fix is the same in every case: annotate the channel with a reducer like operator.add so concurrent writes append instead of collide. CopilotKit shipped the identical patch for their LangGraph integration in August 2025 (PR #2276).

⚡ If you remember one thing from this post: every state key that could receive concurrent writes needs a reducer. The question isn’t whether you have parallel execution today. It’s whether you might add it next sprint.

Compilation: The Step Most Writers Skip

graph.compile() is not a convenience call - it’s where LangGraph turns your declarative graph into an executable plan.

During compilation, the engine does four things:

1. Validates that every conditional edge routes to a declared node

2. Builds the channel topology from your state schema and reducers

3. Constructs the actual PregelNode objects from your node functions

4. Freezes the graph - the compiled object is immutable

What compile() returns is a Pregel instance. That’s what you invoke, stream from, and checkpoint. The StateGraph you built was a blueprint. The Pregel is the machine.

This matters because compilation errors are caught before a single token of inference runs. Return a value from a conditional edge that isn’t in the edge map, and compilation throws immediately — something like:

ValueError: Expected one of ['tools', 'end'], got 'tool'
Check your conditional edge return values.

Typo in an edge name. Zero LLM calls wasted. That’s the contract compile() offers - and most agent frameworks can’t give you anything close to it.

Checkpointing Is Write-Ahead Logging

Every database engineer already knows how checkpointing works, even if the docs don’t put it that way. LangGraph’s checkpointer is write-ahead logging applied to agent state.

After every superstep, the full channel state is serialized and persisted. On resume, the engine reads the latest checkpoint and continues from the superstep boundary. Simple idea, enormous consequences.

The persisted object is a Checkpoint - a versioned snapshot containing channel_values, channel_versions, and any pending writes from nodes that succeeded while a sibling was failing. The thread_id is the namespace key: one agent session, one thread. Different threads can’t see each other’s state.

One primitive, four use cases - really the same capability wearing different clothes:

• Durable execution - crash mid-run, resume from the last checkpoint, lose nothing

• Human-in-the-loop - interrupt before a node, serialize, wait for approval, resume

• Time travel - load any historical checkpoint, replay from there, fork alternate paths

• Partial failure recovery - if one node fails mid-step, the completed parallel writes get stored as pending; on resume, only the failed node re-runs

The Postgres checkpointer shows the mechanics clearly. It maintains four tables. checkpoints holds state snapshots as JSONB. checkpoint_blobs holds large values as binary. checkpoint_writes logs pending writes from mid-step failures. checkpoint_migrations tracks schema versions. Each superstep is an insert; resume is a read of the latest row for a given thread_id.

The trap is write amplification at scale. A long-running agent with a growing message list writes the full state every superstep. If messages accumulate unboundedly and checkpoints fire every step, checkpoint size grows linearly with conversation length.

One developer documented four-second reads on long threads in their production chatbot - the pickled message history sitting in checkpoint_blobs had grown large enough that loading a conversation meant a database query, a binary download, and a full pickle deserialization before the UI could render.

The fix is to evict. Summarize old messages, offload large tool results, keep the hot state small. DeepAgents handles this with a SummarizationMiddleware that compacts conversation history once token usage crosses a threshold. That’s the pattern in one sentence: checkpointing is cheap only if the state stays bounded.

Subgraphs and the State Boundary Problem

Every production agent eventually outgrows a single graph. A planner calls an executor. A router dispatches to specialists. A retrieval pipeline feeds into a synthesis step. LangGraph handles this natively - a compiled graph is itself callable as a node inside another graph.

The mental model: a subgraph is a reusable block of graph. Think of it like a function - compile once, invoke from anywhere. Same runtime, same checkpointer, same supersteps. When the parent reaches the subgraph node, execution descends into the child, runs to completion, and returns to the parent’s next superstep.

The boundary is where it gets interesting

Parent and child have separate state schemas. They have to - otherwise every subgraph would inherit every key from every possible parent, and the schemas would explode.

When execution crosses the boundary, state must be mapped. LangGraph does this one of two ways:

• If the schemas share keys, state flows through those keys automatically. The parent’s messages channel wires to the child’s messages channel. Updates propagate.

• If the schemas don’t share keys, you pass state explicitly at invocation and transform the child’s output back into the parent’s shape. This is the boundary that silently breaks.

The failure mode: you write a subgraph with state key documents. Your parent has state key retrieved_docs. The subgraph runs, writes to documents, returns. The parent’s retrieved_docs is still empty. No error. No stack trace. Just a synthesis step running on zero documents, producing a confident-sounding but ungrounded answer.

The fix is explicit mapping:

def call_retrieval(state: ParentState) -> dict:
    result = retrieval_subgraph.invoke({"query": state["user_question"]})
    return {"retrieved_docs": result["documents"]}  # explicit key mapping

Treat subgraph boundaries like API contracts. Declare them explicitly. Validate the output shape.

⚠️ Subgraphs ≠ Subagents

Don’t confuse structural organization with behavioral autonomy.

• Subgraphs are about code hygiene. Nested graphs that keep your main graph from becoming a spaghetti monster. They share the same execution engine and flow state through explicit channel mappings.

• Subagents are about context isolation. An autonomous loop with its own context window. It doesn’t just share state - it filters it, preventing context pollution where the messy reasoning of one specialist confuses the planner.

You use a subgraph when you want to repeat a pattern. You use a subagent when you want to delegate a problem.

DeepAgents: The Harness Around the Graph

LangGraph gives you the runtime. DeepAgents gives you the architecture.

A vanilla ReAct loop fails in four predictable ways: shallow planning, context overflow, context pollution, state bloat. DeepAgents is a direct answer to each.

It’s an open-source harness from LangChain - a CompiledStateGraph wrapped with four middleware layers, each solving one failure mode:

• TodoListMiddleware - write_todos forces the agent to decompose the task and write the plan into context before acting

• FilesystemMiddleware - ls, read_file, write_file, and friends let the agent offload large content to a virtual filesystem. The prompt holds pointers, not pages.

• SubAgentMiddleware - task spawns an isolated subagent with its own context window. Only the final result returns. The messy middle stays hidden.

• SummarizationMiddleware - watches token usage, compacts history when it crosses a threshold. Keeps checkpoints cheap.

Notice what DeepAgents doesn’t invent. Every capability sits on top of LangGraph primitives - channels, reducers, checkpointing, subgraphs. DeepAgents isn’t a different framework. It’s a set of opinionated patterns for using LangGraph well at scale.

That distinction matters more than it looks. If you understand LangGraph’s internals, DeepAgents reads as a library of workable defaults. If you don’t, it reads as magic. And magic becomes impossible to debug the first time something breaks.

Five Failure Modes You Will See in Production

Every one of these is documented - in GitHub issues, official docs, or production writeups.

1. Concurrent write without a reducer. InvalidUpdateError: At key 'X': Can receive only one value per step. Two actors wrote to the same channel in one superstep without an annotated reducer. The engine refuses to guess which write wins. Fix: Annotated[list, operator.add] or a custom reducer. Sources: Official docs · deepagentsjs #65

This is the first bug every multi-agent system ships.

2. Empty update from a node. InvalidUpdateError: Must write to at least one of [...] A node returned nothing. Happens most often with conditional routing nodes that forget to return a payload on certain branches. Fix: Always return an explicit dict - even {"messages": []} satisfies the engine. Sources: langgraph #740 · langgraph #2644

The engine is strict by design — it would rather throw than guess.

3. State bloat at checkpoint time. Checkpoint reads in seconds, not milliseconds. Unbounded message lists, large tool results stored inline. Every superstep serializes the full channel state. Messages that grow without a cap grow the checkpoint linearly. Fix: Summarize or offload. Keep the hot state small. Source: lordpatil, July 2025 - four-second reads on a production chatbot, traced to checkpoint_blobs.

Checkpointing is only free if you treat state as precious. Most people don’t, until this.

4 & 5. Subgraph state silently not flowing / Serialization failures.

These appear less as sudden crashes and more as slow-burn confusion. Schema mismatch at a subgraph boundary (failure 4) produces no error - just a parent state that never updates, and an agent that silently runs on stale data. Fix: explicit key mapping at the invocation site. (Official subgraph docs)

Serialization failures (failure 5) fire during put_writes or resume: TypeError: Object of type X is not JSON serializable. The JsonPlusSerializer uses ormsgpack; anything it can’t encode breaks the checkpoint. Fix: keep only serializable objects in state, or use JsonPlusSerializer(pickle_fallback=True) for DataFrames and custom types. (langgraph #3441 · langgraph #5769)

Each of these is the kind of bug you ship without noticing. Each has a one-line fix once you see it. The difference between a senior engineer and a principal one on agent code is knowing which to check first.

What to Take Away

Three things that should survive this post.

LangGraph is a message-passing runtime with transactional supersteps - not a graph of function calls. Once you see channels and reducers as the real primitives, everything else follows: parallelism, checkpointing, subgraphs, even DeepAgents.

Every state key that might receive concurrent writes needs a reducer. The overwrite default is safe until it isn’t, and the failure mode is silent corruption.

DeepAgents is a library of patterns for managing the context budget. Planning, filesystem offloading, subagent isolation, summarization - four answers to the same underlying question: how do you keep the hot state small while the task stays long?

Next issue: Embeddings Internals. Why cosine similarity gets weird in high dimensions. What contrastive training actually learns. Why a general-web embedding model will silently fail on your domain data - and how to know when to fine-tune versus pick a better base model.

INTERNALS.md is a technical series on how production AI systems actually work. No tutorials. No framework evangelism. Just the layer beneath.

If this was useful, the best thing you can do is share it with one engineer who'd care.