A QE Perspective on Gen AI
A short guide to understanding AI well enough to trust it correctly, written for quality engineers.
What "understanding AI" means here
It means understanding, at a working level, what a model is doing when it answers you, what kind of mistakes it makes, and how much confidence any given output deserves. That's the same skill you already apply to a system under test. You're just pointing it at a new kind of system. Nothing in the foundations is specific to QE, and the same ideas apply to any use of AI; the later sections show what they look like in testing work.
How these systems work
Large language models are pattern predictors. Given some text, a model estimates which next word is most likely, over and over, using patterns learned from enormous amounts of text during training. Nothing in that process checks the output against reality as it generates it. The model isn't looking anything up unless a tool explicitly does that for it.
Two consequences follow from that mechanism, and the rest of this guide builds on them:
- Fluent is not the same as correct. A model produces confident, well-formed language whether or not the underlying claim is true.
- Outputs are probabilistic, not deterministic. The same prompt can produce different answers on different runs. That changes how you write a test around one.
How next-word prediction works
Most of today's models are built on the transformer architecture, and the mechanism at its core is called attention. Text is broken into tokens (roughly word-sized pieces), and for each token the model computes how relevant every other token in the input is to it (which words matter for interpreting "it" in a sentence, say). That weighting happens across many stacked layers, each refining the picture, and the final layer turns the result into a probability distribution over what token comes next. One gets sampled, it's appended to the input, and the whole process runs again for the next token. There's no separate step where the model checks that distribution against the world. The probabilities come entirely from patterns in training data.
Why they sound human
Models are trained on human-written text, so they produce first-person language, hedges, apologies, and expressions of confidence. There's no internal state behind them; that's just the conversational pattern in the data they learned from. An agent that says "I checked the logs and I'm confident this passed" is generating a plausible-sounding sentence, not reporting on a persistent mental process it has. Multi-turn or multi-step agents make this easier to fall for: each new response that references "what I did earlier" reads as continuity, but that continuity is only as real as what's been logged and passed back in as input. Treat an agent's self-description of its own actions the same way you'd treat any other unverified claim: check the underlying log or artifact, not the sentence describing it.
Memory has to be designed, piece by piece
"Memory" in an agent is shorthand for several distinct responsibilities a system has to be engineered for separately. Know the categories, because each one fails differently and needs to be checked differently.
| Type | What it holds |
|---|---|
| Working | Whatever's in play for the current interaction (system instructions, recent messages, tool outputs), bounded by the context window. |
| Semantic | Stable facts and preferences meant to persist across conversations, like a name or a time zone. |
| Episodic | A record of specific past events, such as what a previous test run failed on and why, kept to inform future actions. |
| Procedural | The rules and workflows the agent is meant to follow, such as validation steps and when to ask for human approval. |
| External | Information living outside the model entirely, in a database, document store, or inbox, pulled in on demand. |
| Parametric | General knowledge baked into the model's trained weights. No retrieval step, just what it "already knows." |
| Prospective | A future task the agent is meant to act on later, triggered by a time or condition rather than run immediately. |
The QE angle: each type implies a different thing to verify. Working memory fails when context gets truncated or crowded out before it should. Semantic and episodic memory fail when facts persist that shouldn't, or don't persist when they should. External memory failures are ordinary integration bugs. The retrieval step is a normal system dependency, and it fails in normal ways. Treating "the agent's memory" as one thing misses all of this. Check each type against its own failure mode.
Where they break
Every failure mode below is a variation on the same root cause: the model is producing something plausible-sounding, not something verified.
None of this makes the tools unusable. It defines the shape of the review process they require, which overlaps with your existing job.
Why this matters for QE
Quality engineering already carries the right instincts for working with AI: treating unverified output as unverified, looking for the failure case before trusting the happy path, and not letting a component's own self-report stand in for independent evidence. AI output calls for that same discipline, and it matters more here, since it fails by sounding right rather than by throwing an error.
AI shows up in QE in two different roles:
| AI as subject under test | AI as a tool inside your process |
|---|---|
| Your product ships an AI feature (a chatbot, a recommendation engine, an AI-assisted search). You need to test its behavior the way you'd test anything else, plus the new failure classes it introduces. | You use an AI assistant to write test cases, summarize logs, or draft automation. Its output becomes an input to your process and needs the same scrutiny as any other unverified input. |
This guide focuses mainly on the second: using AI well as part of how you do QE, while staying alert to the first whenever it's part of your scope.
AI inside the test process
Test case and scenario drafting
AI is useful for generating a first pass of edge cases from a spec or user story. It's fast, and breadth is what pattern-matching is good at. Treat the output as a draft list to prune and verify, not a finished suite. It will confidently include cases that don't apply to your system and miss ones that require knowledge it doesn't have, like undocumented business rules.
Log and failure triage
Summarizing a long test run's logs to point you at likely root causes is a strong use case, because the model is pattern-matching against text you can verify line by line. Checking a suggested root cause against the actual log line it's citing, rather than the summary of it, catches most of the errors that slip through.
Automation code
Review AI-generated test scripts the way you'd review a human's first draft: check that the assertions assert something meaningful, not just that the code runs. A generated test that never fails is a common and easy-to-miss failure mode of its own.
Autonomous test agents
Newer tools let an AI agent explore an application, generate steps, and execute them with no human writing the individual actions, such as clicking through a UI or driving an API on its own. That autonomy removes the usual per-step review, so verification has to move to the boundaries: what the agent was authorized to touch, what evidence it logs for each action it took, and how its results get audited afterward rather than trusted in the moment.
Using AI as a test oracle
This is the highest-risk use. Asking a model "does this output look correct?" combines two unverified things, its judgment and the thing being judged, and non-determinism means the same check can pass one run and fail the next with no code change. When it's unavoidable, pinning the model version and logging the full response turns a pass into a lead worth confirming rather than a verdict.
A verification mindset for AI output
The habit that transfers most from QE to working with AI: an AI's output is a claim, not a fact, until something independent of the model has checked it. For anything a model gives you, ask:
- What would I check to falsify this? If there's no answer, treat the claim as unverified.
- Does this cite something checkable? A file, a line number, a log entry you can open yourself.
- Would this answer change on a re-run? If the question is close to a coin flip for the model, don't let one run stand as your evidence.
- What does it cost if it's wrong? A wrong suggested test name is cheap. A wrong "this is safe to release" is not, so scale your review effort accordingly.
A working adoption checklist
A starting point for introducing AI tools into a QE workflow without introducing new, invisible defects.
- Every AI-generated test case is run and its result inspected before it's added to the suite verify
- AI-suggested root causes are checked against the raw log line, not just the summary verify
- AI is not used as the sole pass/fail oracle for release-blocking decisions high risk
- Model version and prompt are logged wherever AI output feeds a test artifact, for reproducibility verify
- Generated automation code is reviewed for meaningful assertions, not just successful execution verify
- Anyone relying on AI output knows what a hallucinated result would look like in this specific context team readiness
Keep practicing
Your sense of where AI goes wrong needs upkeep, the same way your sense of where a codebase breaks does. The fastest way to build it is the most familiar one: use the tools on real work, look for where they're wrong, and keep a running list of the failure patterns you see most often.