Using and judging AI systems

The international definition is deliberately mechanical: a machine-based system that, for stated or implied objectives, infers from its input how to generate outputs such as predictions, content, recommendations or decisions, with varying autonomy after deployment. Note what that sentence omits. Nothing about understanding, intent or belief.

The beginner mistake is treating AI as a single thing to be for or against. The same label covers a fraud score on your card, a recommendation feed, and a system that writes paragraphs — different techniques, different failure modes, different stakes.

The second mistake is harder to unlearn: the borrowed vocabulary. Say a colleague reports that "the assistant knew our policy was out of date." What happened is that the text it produced matched patterns in which policies get described as outdated. Nothing consulted the policy. These systems can even assert that they are human — an assertion is output, not evidence.

Precision costs nothing. The model is the trained artifact; the system is that model plus retrieval, tools, interface and the organization around it. Most arguments about "AI" dissolve once people say which they mean.

Rule: describe what the system produced, never what it "knew" — and say whether you mean the model or the whole system around it.

Machine learning splits cleanly in two. In the training stage a model is learned; in the deployment stage it is applied to new inputs to generate outputs. Supervised learning takes labeled examples and minimizes a loss function, with validation and testing before deployment. Unsupervised learning trains on unlabeled data. Reinforcement learning has an agent interact with an environment and learn a policy that maximizes reward.

Chat assistants layer these. Pre-training runs over a very large, low-quality body of text, learning to predict the next token, which yields a base model. Supervised fine-tuning follows on a much smaller set of high-quality example responses. Then human comparisons train a reward model, and reinforcement learning pushes the model toward the responses people preferred. That last layer is why it behaves like an assistant at all.

Which exposes the commonest misconception about these tools: that they learn from you. Correct an assistant, watch it thank you and get the answer right, and nothing about the model has changed — the correction sits in the conversation, not in the weights.

Expensive is literal, too. Energy and carbon vary by whether you are pre-training, fine-tuning or running inference, and generative tasks measure as more intensive than non-generative ones. No agreed method exists for estimating the footprint.

Rule: ask whether a claim is about the training stage or the run in front of you — they have different owners, costs and failure modes.

Three data problems recur, and none is fixable by asking better questions.

First, representation. The data used to build a system may not be a true or appropriate representation of the context it gets used in, and the ground truth may not exist or be available at all.

Second, staleness. Datasets become detached from their original context or go out of date relative to where the system is deployed, which is why these systems need corrective maintenance for data, model and concept drift. Collection stopped on a date; nothing after it is in the model. Say a workplace rule changed two months ago and you ask about it: a confident description of the old rule is the expected output, not a malfunction. Systems that answer correctly about last week are fetching material at request time and placing it into the model's input — it did not learn that, it was handed it.

Third, opacity. Few developers of foundation models publish key details of their training sources, and in most cases no single organization controls all of that data.

Rule: before trusting an answer about anything recent, ask where the system could have got it — if the honest answer is "training", assume it is stale.

Generative models produce outputs that approximate the statistical distribution of their training data; a language model does this by predicting the next token. Everything downstream follows from that one sentence — including the misconception worth killing early, that somewhere inside there is a lookup step. There is not. A recalled fact and an invented one come out of identical machinery.

What the model sees is a single concatenated stream of text: the developer's standing instructions, your messages, and anything the application retrieved and pasted in. Instructions and data do not arrive through separate channels — harmless-looking here, and the central security problem two chapters on.

That stream is also where apparent memory lives. Continuity across a conversation comes from what is assembled into the input each time, not from the model changing between turns. Reproducibility is a recognized open problem, so do not expect one question to return the same words twice.

The same approach generates images, audio and video, and multimodal models take several at once. Generated code compiles and reads idiomatically without being verified — research suggests engineers who over-rely on coding-assistant suggestions may produce less secure code.

Rule: everything the system reads is one stream of text, and everything it writes is a prediction — including the parts that look like a lookup.

Confabulation is the generation and confident presentation of erroneous or false content in response to a prompt, including output that drifts from what you asked or contradicts something said earlier in the same conversation. Colloquially: hallucination, or fabrication.

It follows from the design. Outputs approximate the statistical distribution of the training data; the system predicts the next word. That prediction can produce something factually correct and something factually wrong or internally inconsistent, and nothing in the mechanism distinguishes the two. It concentrates in open-ended requests for long-form answers and in areas needing specialist or highly contextual expertise.

Now the part that catches careful people. Outputs may include confabulated reasoning and citations that appear to justify the answer, which pushes readers into trusting it further. These systems sometimes lay out tidy logical steps toward a conclusion that is wrong. A reference list is not evidence; it is more generated text.

The documented consequence is that people believe false content because of the confident nature of the response — a confabulated summary of a patient's records, for instance, could lead a clinician toward an incorrect diagnosis.

Rule: confidence, fluent reasoning and a neat citation list are all output — treat each as a claim to check, never as evidence that checking happened.

Four habits, in rough order of value. Open the cited source and confirm it says what the answer claims — since citations can themselves be generated, an unopened reference proves nothing. Confirm against something the system did not hand you. Recompute any figure rather than trusting stated arithmetic. And treat asking the model to check itself as very weak evidence, because plausible justification for a wrong answer is exactly the failure mode.

Here is the shape it usually takes. A summary of a policy names a clause and a section number; you open the document and that section is about something else entirely. Nothing about the summary looked wrong. That is why the reference is the first thing to check, not the last.

Capability claims deserve the same treatment. Accuracy figures should be paired with realistic test sets representative of expected conditions, and should show they generalize beyond training conditions. Benchmark datasets themselves can carry label errors, undermining their stability, and measures can be oversimplified, gamed, or blind to differences between affected groups. Risk measured in a laboratory can differ from risk in operation.

Rule: check the claim against the source, never against how sure the answer sounded — and read a score for the conditions behind it before believing it transfers.

Three categories are worth holding separately. Systemic bias sits in datasets, in organizational norms and processes, and in the wider society using the system. Computational and statistical bias sits in datasets and algorithms, often from systematic error caused by non-representative samples. Human-cognitive bias is how a person reads output in order to decide or fill a gap. Each can occur with no prejudice or discriminatory intent anywhere.

What AI adds is speed and scale. A worked instance: prompted for images of chief executives, doctors, lawyers and judges, text-to-image models have been found to underrepresent women, racial minorities and people with disabilities — and to struggle to produce non-stereotyped output even when the prompt explicitly asks for it.

Performance is uneven too. A language model may work less well in non-English languages or particular dialects, and the trap is trusting it to perform the same for everyone — which can leave the underperforming group worse off than using no system at all.

Compounding is the part people miss. Outputs skew uniform; that homogenization flows from foundation models into everything built on them; and training that over-relies on synthetic data can make points vanish from the next model's output distribution.

Rule: ask who this system works less well for, and never read strong overall accuracy as equal performance across groups.

Start with the training set. It may contain personal data, and most developers do not disclose their specific sources — so nobody outside can tell whether their information was in there or how it was collected. Models have been made to reveal sensitive material they memorized, including data appearing in only a small number of training samples.

The less obvious exposure is inference. These systems can correctly deduce sensitive attributes that were never in the training data and never disclosed by the user, by stitching together information from separate sources. Wrong inferences cause harm too, when something untrue but sensitive gets attached to a person. This is why de-identification is a mitigation rather than a guarantee: removing a name does not remove the ability to re-identify by combining signals, and enhanced data aggregation is itself listed among the privacy risks these systems introduce.

Then the mundane version. Pasting a client file into a general-purpose assistant to save ten minutes of summarizing moves that file to a third party, under whatever retention and access terms that service happens to have. Client, patient, employee and student records all carry boundaries a paste ignores.

Rule: before pasting, ask whose data this is and whether you may hand it to a third party — and assume whatever you send can be kept.

The root cause is structural: data and instructions are not supplied in separate channels, so an attacker can smuggle instructions through the data channel — the same shape as injection attacks that have troubled databases for decades.

The mistake is picturing this as a user tricking a chatbot. That is direct prompt injection, the easier half. The harder case is indirect prompt injection, mounted not by the user but by a third party controlling a resource the system reads: a web page, a document, a database entry. Often the user is the one harmed.

Demonstrated techniques are more inventive than people expect: instructions hidden in non-visible parts of a resource, multi-stage injections where the first instruction sends the model to fetch a second, encoded text the model is asked to decode. Researchers have shown a model wired into an email client induced to forward messages to an attacker's inbox, and self-propagating injections telling it to send similar messages to everyone in the contact list.

Agents raise the stakes because they run the loop: prompt the model, act on its output including calling functions, feed results back as context. Current mitigations do not fully protect, so the standing advice is to design as though injection will succeed.

Rule: anything the system reads that you did not write is untrusted input — and the moment it can send, buy, post or delete, put a person in front of that step.

Two things scale here: unintentional false content, particularly where it stems from confabulation, and deliberate deception — synthetic audiovisual content and photorealistic images produced with intent to mislead. One documented episode: a synthetic image purporting to show an explosion at a government building circulated widely and briefly moved the stock market.

The personal version is likelier to reach you. A caller sounds exactly like a family member, says they are in trouble, needs money now. A short clip of that person's voice — often available from something they posted publicly — plus a voice-cloning tool is enough. The instinct to listen harder is the wrong one, because the voice is genuinely right. Hang up and call the person back on a number you already know is theirs, or reach them through someone else. Note the payment channel too: demands for a wire transfer, cryptocurrency, or gift-card numbers are themselves a signal.

Nor will technology settle it for you. No watermarking technique is universally effective and robust; attacks against watermarking succeed at high rates, and theoretical impossibility results exist. An absent mark proves nothing, and a present one can be stripped. The corrosive knock-on is that genuine evidence can now be waved away as fake.

Rule: never authenticate a person or a document by how it looks or sounds — authenticate through a channel you chose, not the one that contacted you.

Automation bias is over-reliance on an automated system, or unjustifiably perceiving its output as higher quality than output from other sources. It is documented as amplifying the other risks: confabulation and bias both get through more easily when the reviewer is deferring. Underneath sits a plain assumption that these systems work, and work well, everywhere — they are often perceived as more objective than people, warranted or not.

Pairing can make things worse rather than better. In perceptual judgment tasks, the AI half of a human-AI interaction has been observed to amplify human bias, producing decisions more biased than either the person or the system alone. Deliberately accounted for, the same pairing can improve on both.

The failure has a recognizable look. A reviewer who approves every draft in ten seconds is not oversight, they are a signature. A useful measurement is how often and why people actually overrule the system; if nobody ever does, the review is decorative.

The opposite error is real too — experts can be needlessly averse and lose genuine benefit. Both are calibration failures.

Rule: decide before you start what would make you reject this output — if you cannot say, you are not reviewing it.

The characteristics are: valid and reliable; safe; secure and resilient; accountable and transparent; explainable and interpretable; privacy-enhanced; and fair with harmful bias managed. Valid and reliable is the foundation — the others cannot compensate for a system that does not work. Accountability presupposes transparency. And trustworthiness is only as strong as its weakest characteristic, so taking them one at a time does not add up to it.

The trade-offs are documented, not hypothetical. Interpretability can pull against privacy; predictive accuracy can pull against interpretability; privacy-enhancing techniques can cost accuracy where data is sparse. In the security literature, explainability trades against robustness to attack, and systems optimized for accuracy alone tend to score worse on both robustness and fairness. Maximizing everything at once may not be possible.

Internationally agreed principles land in the same place: inclusive growth and well-being; human rights and democratic values, fairness and privacy included; transparency and explainability; robustness, security and safety; accountability.

So the adoption question is never "which is best". A system that is more accurate but cannot explain itself may be wrong for a decision someone must be able to contest, and fine where nobody will. Tools bought from third parties may be opaque, with risk tolerances unlike yours.

Rule: name the characteristic that matters most for this specific use before comparing anything — "better" is not a property a system has on its own.