AI & LLMs curriculum

There is no plan held anywhere before generation starts — the sentence exists only as it is produced, each step re-reading everything written so far. That explains why a wrong early word is committed to rather than corrected, and why asking for the answer before the reasoning gets a worse answer than asking for the reasoning first.

Every position can look at every earlier position at once, rather than sweeping through in sequence, so "put the important part last so it is freshest" describes a mechanism that does not exist. Position still matters, but through training distribution and context length, not through anything resembling short-term memory.

Nothing is retained between calls: the illusion of a conversation is entirely your client resending the transcript each time. Two consequences follow immediately — the cost of a long chat grows with its history rather than with the latest message, and a transcript your code edits or truncates is the only past the model has.

There is no lookup step that either succeeds or fails, so a well-attested fact and a confabulation are produced identically and arrive with identical fluency. That is why confidence in the wording carries no information about accuracy, and why a citation the model generated must be checked rather than trusted.

The training objective rewards a likely continuation, and "I don't know" is rarely the likely continuation of a confident question. Nothing in that loop penalizes inventing a plausible detail more than it penalizes abstaining, so the fix is structural — supply the facts, or explicitly permit and reward abstention in the prompt.

Anything after the freeze exists for the model only if you put it in the prompt, and the failure is quiet: rather than reporting ignorance, it answers from the world as it was. Worse, the model's own belief about the cutoff date is itself a trained-in guess, so asking it when its knowledge ends is unreliable.

The pretrained model continues text; following an instruction, holding a role and stopping at a sensible point are all added afterwards by a separate stage. That separation explains why the assistant behavior is a layer that can be argued with, and why a jailbreak is best understood as pulling the model back toward plain continuation.

Common words are usually one piece, rare ones are split into several, and the split depends on the exact spelling and even the leading space. So token counts cannot be derived from either word or character counts, and two strings that look equally long to a human can differ several-fold in what they cost.

Structure is expensive. Punctuation-dense JSON, long identifiers, base64, URLs and repeated indentation all fragment into many pieces per character, so a payload that looks compact can cost several times what the same information costs as prose. Measuring the real inputs is the only way to size a context budget.

Scripts under-represented in the tokenizer's vocabulary fragment into far more pieces, so the same message costs more and fits into less window in some languages than others. It is a real fairness and cost issue rather than a curiosity: identical functionality can be several times more expensive to serve in one language.

The model sees pieces, not letters, so anything requiring character-level manipulation — counting occurrences, reversing a string, strict syllable or rhyme constraints — is being asked through a representation that discards the detail. Do it in code and hand back the result rather than prompting harder.

They share one budget, so a prompt sized right up to the limit leaves nothing to answer with and the reply is cut off — or refused before it starts. Always reserve headroom for the longest acceptable output, and remember that in a loop the transcript grows every turn while the ceiling does not.

It is a ceiling only: setting it high does not produce a longer answer, and hitting it truncates mid-sentence rather than wrapping up gracefully. The tell is a response that stops abruptly with a stop reason naming the cap — at which point structured output is usually syntactically invalid and must be repaired or regenerated.

A counting endpoint prices and sizes a request without generating anything, which makes it the cheap way to enforce a budget, to decide whether a document fits, and to choose between models by window. Estimating with a characters-per-token rule of thumb is exactly where the inflation cases above bite.

Retrieval accuracy is not flat across a long context: material at the beginning and end is found reliably while the middle is measurably weaker. So position is a design decision — put the instruction and the critical passages at the edges, and treat "it is in there somewhere" as a weaker guarantee than it sounds.

It does not add randomness on top of a fixed answer; it changes how sharply the existing probabilities are peaked before one is drawn. Low values concentrate mass on the likeliest continuation, high values spread it toward the tail — which is why high settings produce not just variety but occasional incoherence.

Greedy decoding becomes overwhelmingly likely, not certain: floating-point non-associativity, batching and infrastructure changes can all move a near-tie, and no provider promises byte-identical output. Build on that assumption — cache on the request, verify the shape of what returns, and never make two runs matching a correctness requirement.

Instead of a fixed cutoff, the candidate set is grown until the accumulated probability reaches a threshold, so it is wide where the model is uncertain and narrow where it is confident. That adaptivity is the point: a fixed top-k truncates a genuinely open choice and an already-decided one identically.

Both operate on the same distribution, so moving them together makes the result impossible to attribute — a change could have come from either, and their interaction is not additive. Pick one as the variable, leave the other at its default, and change it in visible steps against a fixed set of inputs.

Anything with one right answer — extraction, classification, routing, code that must run — wants the distribution sharpened, because variety there is just error. Generation that is judged on range rather than correctness wants the opposite. Using one setting across a pipeline means both halves are mis-served.

Generation halts on a literal match, and the matched text is not included in what comes back, so the sequence has to be something that cannot appear inside legitimate output. The stop reason distinguishes this from a natural ending — a distinction your code needs, because the content is complete in one case and deliberately cut in the other.

Roles are structural rather than cosmetic: the model was trained on the alternation, so concatenating everything into one long user turn discards information the format carries. It also means the transcript is a data structure you can edit, truncate and replay, which is what makes caching and compaction possible.

Putting durable rules in the dedicated field keeps them separate from the conversation, so they are not compacted away, not confusable with user content, and not re-stated every turn. Smuggling them into a fake opening user message makes them look like something the user said — and therefore something a later user turn can plausibly override.

Because content is an array of typed blocks, one turn can carry text alongside an image, a document, a tool request or a tool result — and a response can contain several blocks that must each be handled by type. Code that assumes a string works until the first image or tool call and then fails on a field that does not exist.

Continuity is something your code constructs by appending each reply before the next call, which means you also own what is forgotten. That is the lever for compaction and for cost control, and it is the hazard: dropping the turn where a constraint was agreed removes the constraint, silently and permanently.

Each value demands a different branch — finished normally, truncated at the cap, halted on a stop sequence, or waiting for a tool result — and treating them alike is how a truncated answer gets parsed as complete. In a tool loop it is also the loop condition, so ignoring it is how a run either stops early or never stops.

The response reports what you were actually billed, including any cached and cache-write portions separately, which is the only trustworthy figure — an estimate made before sending cannot know how the prefix was cached. Logging it per call is what lets a cost regression later be attributed to a specific stage.

An alias moves when the provider moves it, so a pipeline built on one is subject to behavior changing without a deploy, and an eval run last month is not comparable to one run today. Pin the exact id, upgrade deliberately, and re-run the suite as part of that upgrade rather than discovering the difference in production.

The header pins the response shape, so new fields and changed defaults arrive when you choose rather than when they ship. That makes an upgrade a dated, reviewable act with a diff behind it — and it is why a client that omits the header can start failing on a day you changed nothing.

What arrives is a sequence of events your client assembles, not a partial copy of the final object, so the accumulation logic is yours to get right. The practical consequence is that state must be rebuilt as you go — content blocks open, receive deltas and close — and the final message exists only once you have assembled it.

Message-level events carry metadata and usage, block-level events open and close a region, and deltas carry the fragments. Handling them as one undifferentiated feed of text drops exactly the information you need: which block a fragment belongs to, and whether the stream ended properly.

The first token arrives sooner and the last one does not, so streaming improves how the wait feels without shortening the work. Anything measured from the completed response — a parse, a tool call, a downstream request — sees no improvement at all, which is why streaming is a UX lever rather than a throughput one.

A structured value arrives in fragments and is invalid at every intermediate point, so parsing before the block closes fails or, worse, succeeds on a truncated prefix. Accumulate the fragments and parse once at the close — incremental parsers exist, but they are for display, not for anything you act on.

A 200 and a first event only mean the request started. An overload, a timeout or a content stop can arrive after tokens have already been shown, so the client needs a path that handles failure after partial output — including deciding whether to keep what was displayed or retract it.

Past a size threshold a non-streaming request is refused or times out at the transport layer, because the connection would sit open with nothing on it for minutes. That makes streaming a requirement for long output rather than an optimization, and it is worth discovering in development rather than at the first long response.

Submitting asynchronously costs substantially less per token in exchange for results arriving within a window rather than immediately. That suits evaluation runs, backfills and bulk classification — anything with no user waiting — and the saving is large enough that a pipeline doing offline work at interactive prices is usually paying for nothing.

Each request succeeds or fails on its own, so a batch is not an atomic unit and completion does not mean everything worked. Results are reconciled by the custom id you supplied, which means that id must be unique and meaningful to you — and that ordering cannot be assumed from the submission order.

The match is byte-exact and anchored at the very beginning, so a changed timestamp, a reordered tool list or one extra space near the top is a miss. There is no partial credit and no warning: the request simply costs full price, which is why cache performance has to be read from the usage figures rather than assumed.

Ordering is the whole technique. Anything that varies per request — the user's message, retrieved passages, the current time — has to come after everything you want cached, because the prefix ends at the first difference. A system prompt that interpolates a timestamp invalidates every call that follows it.

Markers are explicit rather than inferred, and everything up to a marker forms the cacheable unit — so placement decides what is reusable. Several breakpoints let a long stable region and a slower-moving one be cached separately, which is what keeps a small edit from discarding the large prefix in front of it.

Entries expire after an idle period and each hit extends the life of the entry, so a steadily used prefix stays warm while an occasional one does not. That makes traffic shape part of the economics: the same prompt is cheap under continuous load and pays the write premium repeatedly under sporadic use.

The first call pays a premium to store the prefix and later calls pay a fraction of the normal input price, so the break-even is a small number of hits. Below it caching is a net loss, which is exactly the case for a prompt that varies per user or a prefix that is invalidated before it is reused.

Because the match runs from the start, changing a byte near the top discards everything after it regardless of how much was identical. So a small routine edit to a system prompt has a cost far out of proportion to its size, and it is worth batching such edits rather than shipping them one at a time.

The model has your words and nothing else — not the ticket, not the conversation you just had, not the obvious-to-you constraint. Most disappointing output traces to a requirement that was never written down, so naming the goal, the reader, the format and what to do in the ambiguous case is the first thing to try.

Adjectives like concise or formal are interpreted against the training distribution rather than against your house style, while two or three worked examples pin the format exactly. It is also the cheapest way to convey edge-case handling — show the awkward input and the desired response rather than describing the rule.

Whatever your examples share gets learned along with the intended pattern: all short, all about one domain, all answering yes. The model has no way to tell which properties were the lesson, so vary everything that is incidental, and include at least one example of the case you are worried about getting wrong.

Working through the steps raises accuracy on multi-step problems, and it raises latency and price too, because every reasoning token is generated and billed. So it pays where the task genuinely has steps and is pure overhead on lookup or classification — which is why it belongs to the task, not to the house style.

Keeping the working in its own tagged section means the caller can strip it cleanly, log it for debugging, and still get parseable output. Mixed together, the two become a parsing problem and the reasoning leaks into whatever is shown to a user — and asking for no reasoning at all just removes the benefit.

A prompt carrying a dozen competing rules degrades on all of them, and the failure is silent: you cannot tell which instruction was dropped. Splitting into focused calls makes each one testable, cacheable and individually fixable, and the extra request is usually cheaper than the retries a compound prompt produces.

A prohibition still puts the forbidden thing into context, and it leaves the alternative unspecified — so the model has to infer what you wanted instead. Replacing "do not apologise" with a description of the opening you do want removes both problems, and it is testable in a way that an absence is not.

Hand-picked examples are tidier than production: shorter, better formed, and free of the cases you did not think of. A prompt tuned on them looks excellent and degrades immediately on live traffic, so pull a sample of genuine inputs — including the malformed ones — before concluding anything about a prompt.

Tagged regions do two jobs at once: they tell the model unambiguously where the instructions stop and the document starts, and they give your code a reliable boundary to slice on. That is more robust than a prose heading or a row of dashes, both of which can appear inside the content itself.

Where a provider still accepts a trailing assistant turn, seeding it with the first characters of the shape you want constrains the reply without a word of instruction — the continuation simply has to follow what is already there. It is a property of continuation rather than obedience, which is why it is more reliable than asking.

This was the standard cure for a response opening with a sentence of throat-clearing before the payload. It is now largely superseded: schema-constrained decoding gives the same guarantee structurally, and current Claude models reject a trailing assistant turn outright, so code relying on the technique fails rather than degrading.

When the decoder is restricted to tokens the schema permits, validity stops being something the prompt has to ask for and becomes a property of generation. That removes an entire class of retry logic, and it shifts the remaining risk: the output always parses, so a wrong value inside a valid shape is what you now test for.

Hitting the output ceiling produces a cut-off string rather than a short but valid document, so any parse must be wrapped and any failure must distinguish truncation from a genuine format error. Checking the stop reason answers that immediately, and the fix for truncation is a bigger cap or less output, never a better prompt.

Forcing output to be nothing but a data structure removes the space in which multi-step work happens, so accuracy on anything requiring thought drops even though every response parses. Give the reasoning its own field inside the structure, or do it in a separate call — do not expect both from one constrained response.

What comes back is a request: a name and a set of arguments. Your code decides whether to run it, runs it, and owns every consequence — which means authorization, validation and rate limiting live on your side and cannot be delegated to the prompt. A tool the model should not be able to trigger must not be offered.

The model stops and hands back a tool request; you execute, append a result carrying the matching id, and call again with the extended transcript. Nothing happens automatically between those steps, so a loop that forgets to send the result, or sends one with the wrong id, stalls rather than erroring clearly.

The name and description are the only things the model has when deciding whether this tool fits, so they are prompt text and deserve the same care — state what it does, when to use it, and when not to. Two tools with overlapping descriptions produce the wrong choice far more often than a badly typed parameter does.

Enums, required fields and format constraints remove whole categories of invented argument: a status field that can only be one of four values cannot arrive as a fifth. It is cheaper to make a bad call impossible in the schema than to detect it afterwards, and it also documents the tool to the model.

Independent calls can be emitted together in one turn and executed concurrently, which collapses several round trips into one. A call that needs another's output cannot be, so a chain is inherently serial — recognizing which of the two you have is what decides whether a loop takes two turns or six.

The setting is per request, so the same tool set can be open-ended on one call and forced to a specific function on the next. Forcing a call is how you guarantee structure from a step that must produce data; forbidding tools is how you get a plain answer out of a session that otherwise keeps reaching for one.

Returning the failure to the model as the tool's result lets it read the message and try a corrected call, which is the behavior you want from a bad argument or a missing record. Throwing instead ends the loop and discards the context, turning a recoverable mistake into a failed run.

Definitions travel with every request and are billed every time, so a rarely used tool is a permanent tax on all traffic rather than a cost paid when it fires. That argues for a small, task-scoped tool set per call — and it is one of the things worth putting early in the prompt so it can be cached.

The defining property is that the next step is chosen by the model rather than by your code: run, observe, decide, repeat until something says stop. Everything difficult about agents follows from that one property — you cannot enumerate the paths in advance, so correctness has to come from bounds and verification instead.

When the steps are known, writing them down is cheaper, faster, debuggable and reproducible; an agent buys flexibility you do not need and pays for it in tokens and unpredictability. The honest test is whether you could draw the flowchart — if you can, build the flowchart.

A loop whose stop condition is the model's own judgment can fail by never stopping, so a done signal is necessary and not sufficient. Hard ceilings on steps, tokens and wall-clock are what make the failure bounded, and each needs a defined behavior on exhaustion rather than an exception nobody handles.

Every iteration resends the accumulated history, so the tenth step costs far more than the first and total spend rises faster than the step count. That is the single biggest surprise on an agent's bill, and it is why compaction, subagents and trimming stale tool output are cost controls rather than tidiness.

A child runs in its own window and returns a summary, so the exploration — dozens of tool results, dead ends, long files — never enters the parent's transcript. The parent stays small and focused; the cost is that everything the child saw is gone, so what the summary must contain has to be specified up front.

Over a long run the original objective is progressively outweighed by everything that has happened since, and the agent optimizes for the local step instead. Restating the goal and the success criteria periodically — and keeping them out of the compactible region — is what holds a long run on target.

A checkable signal gives the loop something to be right or wrong against, so it can stop when the tests pass rather than when it feels finished. Without one, more iterations produce more confident-looking work of unchanged quality — which is why adding a verifier usually beats improving the planning prompt.

Approval should be proportional to how hard the action is to undo: a read needs none, a reversible write needs little, and a deletion, a payment or an outbound message needs an explicit yes. A flat policy fails both ways — it is either an approval prompt nobody reads, or an unsupervised irreversible action.

Everything present competes for attention, so adding marginally relevant material makes the relevant material harder to find rather than merely costing more. A bigger window raises the ceiling without changing that, which is why the discipline is selection — and why "just paste everything" degrades quality as well as price.

Compacting older turns into a summary while keeping the recent ones verbatim preserves immediate detail where it matters and bounds the growth that would otherwise be unbounded. The design question is where the boundary sits, because a turn that just crossed it is available in full one request and paraphrased the next.

Whatever the summary omits is gone permanently, and the things most often omitted are the ones that read as settled — a constraint agreed twenty turns ago, a decision not to take some approach. Decide in advance what must survive and state it, or the compaction quietly reopens questions the run had closed.

Identity, policy and hard requirements belong outside whatever gets compacted, so they cannot be summarized away in a long run. The system field is that place. A constraint living only in an early conversational turn has a lifespan set by the compaction strategy rather than by the task.

A handful of file reads or search results usually outweighs every message in the transcript, and most of it is stale — superseded by a later read, or explored and abandoned. It is therefore the highest-yield thing to trim, and trimming it is safer than trimming conversation because the current state is re-readable.

Writing facts to a store and retrieving the few that matter keeps the window small and the cost flat, while resending everything grows both with the history. It also makes the memory inspectable and editable outside the model — which is what lets a wrong fact be corrected rather than merely argued with.

It changes what the model knows, not how it behaves: freshness, private data and a citable source. Reaching for it to fix tone, format or reasoning quality is a category error — those are prompting and evaluation problems, and adding passages to the context makes them slightly worse by crowding the window.

Small pieces match a query precisely and arrive without the surrounding explanation that makes them usable; large pieces carry the context and dilute the match, so the embedding represents an average of several topics. There is no universally right size — it is set by how self-contained the source material's ideas are.

Cutting at a heading, a section or a function boundary keeps each piece a complete thought, while a fixed character count cuts mid-sentence and mid-argument. The structural split also gives you the metadata for free — which document, which section — that the enrichment step below depends on.

A statement spanning the join between two pieces is fully present in neither, so it is retrievable by neither, and nothing reports the loss. A modest overlap makes boundary content recoverable from at least one side; the cost is duplicated tokens in the index and occasional near-duplicate hits.

Prefixing each piece with the document and section it came from measurably improves retrieval, because the embedding then encodes what the passage is about rather than only what it says. It also gives the model provenance at generation time, which is what makes a citation possible instead of a claim.

Vector search ranks by resemblance, and a passage can be squarely on topic while containing nothing that answers the question — a chapter introduction about exactly your subject that states no fact. That is why a high similarity score is not evidence, and why a grounded prompt still needs permission to say the passages fall short.

Two instructions do the work together: use only the supplied passages, and say so when they are insufficient. The first without the second produces a confident answer assembled from training data anyway, because the model is still being asked to answer — abstention has to be an explicitly acceptable outcome.

A stale index is more dangerous than an empty one, because the answer arrives with a citation attached and looks verified. The failure is invisible from the output, so freshness has to be a monitored property of the pipeline — and any document that changes frequently needs a reindexing path with an alert behind it.

Nearby vectors mean related meaning, which is why a query phrased entirely differently from the document still finds it — the case keyword search cannot handle. The flip side is that nothing guarantees the retrieved neighbor is relevant to your intent, only that it is similar in the representation the model learned.

Similarity is the angle between vectors, so magnitude is discarded — a long document and a short sentence about the same thing can be equally close. That is usually what you want, and it is why normalizing vectors makes cosine and dot product equivalent, which several libraries assume without saying so.

Two models place meaning in differently-shaped spaces, so a distance computed across them is arithmetic on incomparable numbers. It does not error — it returns plausible scores and nonsense rankings, which makes it one of the hardest retrieval bugs to see. The embedding model is part of the index's identity.

Every stored vector was produced by the old model, so upgrading invalidates the entire index rather than degrading it gradually. That makes the choice of embedding model a commitment proportional to corpus size, and it is why the reindexing cost belongs in the decision rather than being discovered afterwards.

Exact search over a large corpus is too slow, so indexes trade a small amount of recall for a large amount of speed — and the parameters controlling that trade are the tuning surface. A recall shortfall is silent: the missing document simply never appears, and the answer is built from whatever did.

Semantic similarity is weak exactly where an exact token is the query — an error code, a product id, a rare surname — because those carry little learned meaning. Lexical search is strong there and weak on paraphrase, so production systems run both and fuse the rankings rather than choosing.

Without a common protocol every application needs bespoke glue for every tool provider, so the integration work grows as a product of the two sets rather than their sum. One interface turns that into writing a server once and having every compliant host use it — the value is the ecosystem, not the wire format.

The host is the application the user interacts with, it runs one client per connection, and each client talks to exactly one server. That one-to-one pairing is what keeps servers isolated from each other, and it is where the trust boundary sits: the host mediates, so a server never speaks directly to the model.

Requests expect a response, notifications do not, and both travel in a defined envelope — so the protocol is transport-independent and debuggable with ordinary tooling. The practical consequence is that a server is an ordinary program reading and writing structured messages, not something requiring a special runtime.

The model decides when to invoke a tool, which is what makes them useful and what makes host approval the control point. Treat the boundary seriously: a tool's description arrives from the server and enters the prompt, so a server you do not trust is supplying text the model will read as instructions.

Resources are addressed by URI and attached because the host chose to attach them, not because the model asked. That distinction matters for reasoning about what can happen: a resource cannot be pulled in by the model mid-conversation, so it carries a different risk profile from a tool the model can call at will.

A prompt is a reusable template the user invokes deliberately — a slash command rather than something the model or the application triggers. Keeping the three categories distinct is the point of the taxonomy: each has a different actor deciding when it happens, and therefore a different place to put a check.

A local subprocess communicates over standard input and output, which needs no network and no authentication because the process boundary is the boundary. A remote server uses HTTP with streaming, at which point authentication, origin checking and transport security all become your responsibility rather than the operating system's.

Writing down what a good answer looks like, in prose, before choosing any metric is what keeps the metric from defining the goal. Skipping it produces a suite that measures whatever was easy to measure — length, keyword presence, schema validity — and a system that improves on those while getting worse at the task.

A fixed, versioned set of inputs with expected outputs is what makes two runs comparable at all; a set that quietly grows or gets edited between runs turns every comparison into an unfalsifiable claim. Version it like code, and record which version a result came from alongside the score.

Exact match, schema validation and assertions are deterministic, free and unambiguous, so wherever the task admits one, use it — a judge model adds cost, variance and a second thing to validate. Reserve model grading for genuinely open-ended output, and check first whether part of the output can be graded by code.

For output with no single correct form, a model can score against an explicit written rubric — and the rubric is the work: vague criteria produce scores that track fluency. Scoring against named dimensions with stated anchors is far more stable than asking for an overall rating out of ten.

The known distortions are systematic rather than random: longer answers score higher, phrasing resembling the judge's own scores higher, and position in the comparison matters. Each has a mitigation — length-controlled rubrics, swapping the order and averaging — but none of them is removed by asking the judge to be fair.

Agreement with human labels on a sample is what licenses using the judge at all, and it needs re-checking whenever the rubric, the model or the task moves. An unvalidated judge produces a number with unknown relationship to quality, which is worse than no number because it will be acted on.

A single run is one sample from a distribution, so a pass and a fail on the same input can both be true. Reporting a pass rate over repeated runs turns that into information, and it also exposes the cases that are borderline — which are usually more diagnostic than the ones that fail every time.

A prompt, model or tool change can fix the case in front of you and break three you are not looking at, and nothing about the change announces it. Running the suite before shipping is what makes that visible; the suite is only as useful as the cases previous failures were added to it as.

Everything arrives as one token stream, so text fetched from a page, a file or a tool result is not structurally distinguishable from something you wrote. The working rule is that retrieved content is material to report on and never a source of instructions — and that rule has to be enforced by your architecture, because the model cannot enforce it reliably.

The payload need not come from the person typing: it can sit in a web page the agent fetches, a document it summarizes, or a tool response. That is what makes it hard — the attacker never interacts with your system directly, and the user is unaware, so neither party is in a position to notice.

Wrapping untrusted text in tags and saying not to follow it measurably lowers the success rate and does not make the text inert, because the same mechanism that reads the instruction reads the content. Treat it as one probabilistic layer among several, never as the thing that makes an untrusted source safe.

Access to private data, exposure to untrusted content, and a way to send something outward are each harmless alone and a breach together. That framing is useful because removing any one leg closes the path — which is often far easier than making the model injection-proof, and is auditable in a way prompt wording is not.

Scoping credentials and permissions bounds what a successful injection can accomplish, which is the only defense that does not depend on the model behaving correctly. Read-only where possible, narrow scopes, short-lived tokens, and no tool present in the set unless this task actually needs it.

Role-play, hypothetical framing, gradual escalation and claimed authority are surface variations on one move: making the disallowed output look like something else. Recognizing the family matters more than collecting the individual phrasings, because the phrasings are regenerated endlessly and the structure is not.

Input screening, system-prompt rules and output filtering fail in different, largely independent ways, which is exactly why running all three is worth more than perfecting one. It also gives you somewhere to put a detection: a filter that catches nothing is evidence, and only if something is watching it.

Identifiers should be redacted before they enter a prompt rather than after, because everything downstream — logs, traces, eval sets, cached prefixes — inherits whatever the prompt contained. Retention is the part usually missed: a debugging trace with full names in it is a data store nobody classified as one.

Prompting is minutes and reversible, retrieval is hours and additive, tuning is days and pins you to a base model. Working in that order is not caution for its own sake — each step also tells you whether the next is needed, and a surprising amount of what looks like a tuning problem is a prompt that never stated the requirement.

Style, format and consistent task behavior transfer well; specific knowledge does not, and attempting it produces confident recall of training examples that were never true in general. Facts belong in retrieval, where they can be updated without retraining and cited at answer time.

A few hundred consistent, correct examples outperform thousands of noisy ones, because the model learns the inconsistencies just as faithfully as the pattern. That makes labeling discipline the main cost, and it makes a small carefully-reviewed set the right starting point rather than a compromise.

Training on clean curated inputs and serving messy real ones is the classic failure, and it shows as a model that evaluates well and disappoints in use. The training set has to include the typos, the truncations and the badly formatted cases, because those are what it will actually meet.

Without a held-out measure there is no way to distinguish a model that learned the task from one that memorized the examples, and both look excellent on the training data. Build the eval first: it is also what tells you whether tuning was needed, which is the cheaper answer more often than not.

The result is tied to the weights it was trained on, so every upgrade means retraining and re-evaluating rather than changing a model id. In a field where base models improve quickly, that ongoing cost usually outweighs the gain — and it is why the decision deserves a deliberate re-check each time the base moves.

Training a small set of additional weights leaves the base untouched, so one deployment can serve many tasks by swapping adapters instead of hosting a full model each. Training and storage are far cheaper, which changes the economics enough to make per-task specialization practical where full tuning would not be.

Output costs a multiple of input, so the two are not interchangeable when trimming: shaving a long prompt saves less than shortening a verbose answer by the same number of tokens. That asymmetry should drive the optimization order, and it is why asking for brevity is a cost lever and not just a style preference.

Each turn resends the whole history, so cost per turn grows with the conversation rather than with the new message, and a long session's last exchange can cost many times its first. That is the arithmetic behind compaction and caching — and the reason a per-message cost estimate understates a chat product badly.

A pipeline usually contains steps of very different difficulty — classification, extraction, drafting, final reasoning — and paying top price for all of them is a choice rather than a necessity. Routing the easy steps to a smaller model is often the largest single saving available, and it needs an eval per step to be safe.

Generation is sequential, so time scales with tokens produced, while the prompt is processed in parallel — which makes shortening the answer a far bigger latency lever than shortening the input. A request that feels slow is usually being asked for too much text, not being given too much.

Hashing the request and reusing an identical result is safe and unambiguous. Serving the stored answer for a merely similar request is a different thing entirely: two questions can be close in embedding space and have different correct answers, so the failure is returning a confident wrong reply with no trace of why.

A limit response means wait and try again later, with the delay growing between attempts and jitter so a fleet of clients does not retry in unison. Immediate retries make the situation worse for everyone, and a synchronized backoff without jitter simply moves the spike rather than smoothing it.

A malformed request fails the same way every time, so retrying it burns quota and delays the error the caller needs. Overload and transient network faults are the opposite. Classifying by status before retrying is what separates resilience from an expensive loop, and the classification belongs in one shared place.

An unbounded call is an outage waiting for a slow day, so every request needs a deadline and a defined behavior when it expires — a smaller model, a cached answer, a degraded response, or an honest failure. Returning something worse is nearly always better than returning nothing after two minutes.

Logging tokens, latency and cost per call is what lets a regression be attributed to a stage instead of noticed as a larger bill. Aggregate numbers hide exactly the thing you need — which step changed — and the instrumentation has to exist before the regression, not be added while investigating it.

Treating prompts as versioned artifacts, with the model id and parameters recorded alongside, is what makes a behavior change traceable to a specific edit. Without it a prompt is untracked production code, and the question "what changed?" has no answer — which is also what makes replaying an old run impossible.