Building with LLMs, oriented

A base model is trained to continue text: given everything so far, produce a plausible next piece. That objective rewards fluency, and nothing in it rewards saying "I don't know." A remembered fact and an invented one are produced by exactly the same machinery, so confidence in the output tells you nothing about whether it is true.

The assistant you actually call is not that base model. It has been through further training — humans rank candidate responses, and reinforcement learning pushes the model toward the preferred ones — to make it follow instructions and behave like a helpful assistant. That layer is why prompting works at all, and it is also why "the model is just autocomplete" is a half-truth that will mislead you.

The beginner mistake is treating the weights as a knowledge base you can query. They are frozen at a training date, they hold no record of your last request, and they cannot look anything up on their own.

Concretely: the glossary's own definition — pretraining teaches the model to predict the next word given everything before it, and a token is about 3.5 English characters. So a 1,000-word English brief is roughly 1,570 tokens of prediction, and a citation the model produces for it is one more prediction, formatted like a citation.

Rule: if correctness matters and the fact is not in the request, do not expect the model to supply it.

Text is split into tokens — chunks roughly a few characters long — before the model sees it. The context window is the total space a request can occupy, and it is easy to underestimate what lands in it: the system prompt, every message in the history, every tool definition, every attached document or image, and the output being generated all draw on the same budget.

Two things surprise people. First, quality degrades before capacity does. As the token count climbs, recall and accuracy get worse, so curating what goes into context matters more than how much room is left. Filling the window because you can is a reliable way to make results worse.

Second, caching changes what you pay, not what you occupy. Cached tokens still consume the window exactly like uncached ones.

Concretely: current flagship models carry a 1M-token window and can emit up to 128k output tokens in one request; others carry 200k. Overrun the window during generation on a current model and the response stops with stop_reason: "model_context_window_exceeded"; hit your own max_tokens and it stops with max_tokens. Both look like a finished answer to code that never reads the field.

Rule: treat context as a curated budget, and always read the stop reason before trusting a response.

You send an ordered list of messages, each tagged as user or assistant, plus the model name and a cap on output tokens. The instructions that set the assistant's role and rules do not go in that list; they belong in the top-level system field. Putting them in a user turn is the most common structural error, and it works badly rather than failing loudly.

Nothing is remembered between calls. A multi-turn conversation exists only because you resend the whole transcript, which is also why long conversations get slower and costlier with every turn.

Sampling is the other half. At each step the model produces a distribution over next tokens and one is drawn. That is why identical inputs can return different text, and why a test asserting an exact output string will eventually fail. Assert on properties that survive resampling: the shape parsed, a field present, a classification in the right bucket.

The knobs have moved. The API reference now marks temperature, top_p and top_k deprecated: models released after Claude Opus 4.6 do not support setting them. A temperature of 1.0 is accepted for backwards compatibility, and any other value is rejected with a 400. Even at 0.0, results were never fully deterministic.

Concretely: the reference's own prefill. Send "What's the Greek name for Sun? (A) Sol (B) Helios (C) Sun" followed by an assistant turn reading "The best answer is (", and the response continues "B)", because the final assistant message is treated as the start of the answer. Two user turns in a row are combined into one. Send temperature: 0.2 to a current model and you get a 400 instead of a calmer answer.

Rule: instructions go in the system field, the transcript is your responsibility, and nothing you can set makes the same prompt return the same characters twice.

You mark a point in the request, and everything up to that point can be reused by later calls. The trap is subtle: caching is a prefix match. Anything that changes before your marker produces a different prefix and a total miss. A timestamp, a request identifier, or a per-user greeting placed near the top quietly defeats the whole mechanism. Invalidation also cascades in a fixed order — changing tool definitions invalidates more than changing a later message does — so a tool tweak can silently reset caching you thought was stable.

The fix is to order the request by volatility: stable content first, the marker after it, per-request content last. Then read the cache-read and cache-write counts the response gives you.

Concretely: on Claude Opus 5 the minimum cacheable prefix is 512 tokens (1,024 on Sonnet 5); a 5-minute cache write costs 1.25× base input and a read 0.1×, so a reused 3,000-token system prompt pays for its write on the second read — provided the response's cache_read_input_tokens is not zero. The hierarchy is tools → system → messages, and editing one tool's description invalidates the entire cache beneath it. The entry lives five minutes, refreshed on every hit.

Rule: put static content first, and verify caching from the usage numbers rather than assuming it.

The techniques are learnable in an afternoon: be explicit about the task, show examples, structure the input so sections are unambiguous, give the model room to reason, and split several tasks into several calls.

The step before that matters more. If you cannot state success criteria and check a change against them, you cannot tell an improvement from a coincidence, and single-example testing will happily convince you a useless change helped. Know, too, when prompting is the wrong tool: latency and cost often yield faster to a different model, and a fact the model cannot access needs retrieval, not persuasion.

Three principles do most of the work. Build cases from your real task distribution, including awkward inputs (ambiguity, typos, sarcasm, irrelevant material), because clean cases are not where systems fail. Automate grading where the task allows: exact match, a similarity measure, or a model judging against a rubric. And favor more cases over more polish per case, since confidence comes from volume and hand-grading throttles iteration.

The two commonest failures are vague criteria and a single dimension. "Works well" cannot be measured; a specific threshold on a specific metric can. Accuracy alone hides regressions in safety, latency and cost.

Concretely: the documentation's own criterion is "less than 0.1% of outputs out of 10,000 trials flagged for toxicity by the content filter", a number a rerun can move. Its cheapest grader is output == golden_answer; its model-graded rubric reads "The answer should always mention 'Acme Inc.' in the first sentence. If it does not, the answer is automatically graded as 'incorrect.'", with the grader told to reason first and the reasoning then discarded.

Rule: write the test before the prompt, check that the failure is a prompting failure at all, and remember that a test you cannot re-run is an anecdote, not an evaluation.

You describe each tool with a name, a description and an argument schema. When one applies, the model returns a tool request naming it and the arguments it chose. Your code runs the operation and sends the result back in a follow-up request, and the model continues.

Two consequences follow. Your code executes, so the model's request is untrusted input: validate the arguments as you would anything from outside. And the model selects on the description, so a vague one is why a tool never gets called, or gets called at the wrong moment. Anticipate a missing argument, too: the model may ask, or it may pick a plausible value and proceed, as with a weather lookup for a city nobody named.

Concretely: the reference's round trip. You define get_weather with one required location; the user asks about San Francisco; the response comes back with stop_reason: "tool_use" and a block naming the tool and its argument; your code sends "15 degrees Celsius, partly cloudy" back in a tool_result. Offering tools at all costs context: on Claude Opus 5 the tool-use system prompt alone is 286 tokens, 406 when you force a call.

MCP standardizes that boundary. A server exposes tools and data through a common protocol any client can use, so each side implements one standard rather than every app integrating every source. It is plumbing, not a new model capability: the model still emits a tool request and receives a result.

The security risk carries over, and a one-line install hides it. Connecting a server grants real capability, and what it returns is untrusted third-party input (the safety chapter), more so because you did not write the source.

Rule: treat tool arguments as untrusted input, write the description for the model that must choose, and evaluate an MCP server like the dependency with permissions that it is.

A single request answers a question. An agent pursues a goal: it decides what to do next, calls a tool, sees the outcome, and repeats until the task is finished. Structurally it is the tool round trip from the previous chapter, executed in a loop rather than once.

That loop is where the engineering lives. Each turn appends to the transcript, so context grows and the run gets slower and costlier as it proceeds — long tasks need a deliberate strategy for what to carry forward and what to drop. And an agent that can act needs permission boundaries, because "call a tool" now means "change something."

The mistake is assuming autonomy is the goal. More freedom is not automatically better; an agent given a vague objective and unlimited tool access produces confident, expensive, unpredictable work. Agents get good results when the task is well-scoped, the tools are few and sharply described, and there is a clear condition for being done.

Concretely: the Agent SDK's definition is "an application that completes a task by planning its own steps and calling tools that read files, run commands, or edit code" — and its feature table names the two things this chapter says are hard: permissions ("control which tools run automatically, which need approval") and context management.

Rule: design the stopping condition and the permission boundary before adding capability.

Take the user's question, search a knowledge base you control, and include what comes back alongside the question. The model then answers from material it was never trained on: information newer than the model, private to your organization, or too large to have memorized.

Despite the name, the model retrieves nothing. Retrieval is an ordinary search problem in your own system, and the model sees only what your search returned. So quality is bounded by retrieval, not the model: if search surfaces the wrong passage, a better model produces a more fluent wrong answer.

Concretely: the API has a content block for exactly this — "type": "search_result", carrying a source (any stable string, a URL or an identifier like kb://article-1234), a title, the text, and a citations switch. Pass three such blocks with what your search returned and, with citations enabled, the answer cites them automatically; pass thirty and you have paid for twenty-seven distractions.

An embedding model converts text into a list of numbers placed so that texts with similar meaning land near each other. Comparing two vectors gives a similarity score, so you search by meaning, not keyword: a query about a scheduled earnings call can find the right document with no words in common. Embedding is not part of the Claude API at all, so choosing a provider is a separate decision with its own cost, latency and domain-fit tradeoffs.

Two practical traps. Query and document must be embedded consistently; mixing models, or ignoring a provider's query-versus-document distinction, quietly degrades results. And nearest is not relevant: similarity always returns your top matches, however poor, so a result set is never evidence that a good answer existed.

Rule: debug retrieval before the prompt, prefer fewer and better passages, and inspect what retrieval returns before trusting it, because nearest neighbors are not relevance judgments.

Direct attacks are the familiar case: someone crafts input to talk your application out of its rules. Defenses are layered — screen input before it reaches the main call, state boundaries explicitly in the system prompt, and act on users who keep probing.

Indirect prompt injection is the harder problem. Your user is trustworthy, but the model is reading a web page, an email, a document, or a tool result — and whoever authored that content can embed instructions aimed at the model. The moment an agent reads untrusted content and can also act, the content becomes an attack surface.

The structural defense is to make untrusted material recognizable as data. Deliver third-party content inside tool results rather than system or user text, say what it is and where it came from, encode it so it cannot break out of its container, and state in the system prompt that retrieved content is information to report, never commands to obey.

Concretely: the documentation's agent that reads inbound email. The message body goes in a tool_result, never in the system prompt or a user text block; it is JSON-encoded so a quote in the email cannot close the container; the system prompt says tool content is untrusted data; the raw body is first passed through a small classifier call with Claude Haiku 4.5 and only returned if no injection is found; and your own follow-up instruction is sent in a user turn after the tool_result, because instructions placed inside one may be ignored.

Rule: anything the model reads that you did not write is data, and least privilege limits the blast radius when that fails.

Capacity is metered along several axes at once: requests per minute, input tokens per minute, and output tokens per minute, each tracked per model. Exceeding any of them returns a rate-limit error with a header telling you how long to wait. Respecting that header, with backoff and jitter on retries, prevents the retry storm that turns a brief limit into a sustained outage.

Since generation is variable, plan for partial output: a response can stop at the token ceiling mid-structure, and code that assumes well-formed output will break on the day it matters.

Concretely: 60 requests per minute may be enforced as one per second — the bucket refills continuously rather than resetting on the minute — so a burst can fail while the per-minute average looks fine. Two 429s look alike and are not: a rate limit carries a retry-after header and waiting works; a spend limit carries none, its error code reads enforced_spend_limit_reached, and every retry fails until the limit resets. And on most models cache reads do not count toward the input-token limit, so a cached prefix raises throughput as well as cutting cost.

Rule: obey the retry delay the API gives you, and treat truncation as an expected outcome rather than an exception.