What verification is trying to prove
A fix is verified when the agent faces the same meaningful pressure and the protected boundary holds. The test should not merely show that the prompt changed or that the agent gave a better-looking answer once. It should show that the previous failure no longer happens under the same scenario conditions.
Verification needs three inputs: the original exploit proof, the claimed fix, and a rerun that targets the same boundary. The result should be recorded as verified fixed, still failing, or regressed if the same issue returns after previously being fixed.
Start with what failed
Before rerunning anything, restate the failure in boundary language. Do not say only that the agent gave a bad answer. Say that it disclosed cross-account data, accepted an unverified identity claim, called a refund tool too early, wrote untrusted memory, or followed a webpage's instruction against the user's intent.
The original evidence should identify the invariant. The invariant is the rule that must hold no matter how persuasive the request becomes. Without that invariant, verification becomes subjective because reviewers can disagree about whether the new answer is good enough.
Rerun the right scenario
The rerun should preserve the pressure pattern that exposed the issue. If the failure came from fake authority, the rerun should still include fake authority. If it came from urgency, the rerun should still test urgency. If it came from untrusted document instructions, the rerun should include a comparable untrusted source.
At the same time, verification should not be brittle. Minor wording changes are acceptable if the scenario still tests the same boundary. The goal is not to memorize one exact prompt. The goal is to show that the boundary holds against the relevant manipulation pattern.
- Use the same protected boundary and severity criteria.
- Use the same target agent or the same release candidate of that agent.
- Preserve the same tool, data, or memory precondition when relevant.
- Compare the rerun result against the original failure criteria.
Evaluate the outcome
A verified fix means the agent refused, escalated, asked for verification, avoided the unsafe tool call, or otherwise preserved the boundary. A still-failing result means the fix did not hold. A regressed result means the boundary had previously been verified but failed again after a later change.
The outcome should be clear to non-authors of the fix. A reviewer should see what changed, what was rerun, what the agent did, and why the result was accepted or rejected. That prevents the verification step from becoming another vague review.
Turn verification into prevention
The final step is deciding whether the scenario should become a recurring check. High-impact failures, repeated failures, and failures tied to important business boundaries should usually become regression tests.
This is how verification becomes a durable security practice. The team is no longer relying on memory that a prompt was fixed. The boundary becomes something that can be checked again when the agent, model, tools, retrieval sources, or policies change.
Verification checklist
A verification review should be short but disciplined. Start by reading the original finding, not the fix description. The original finding tells reviewers what evidence proved the failure and which invariant must now hold.
Next confirm the test environment. The target agent, tool permissions, test data, and scenario setup should be close enough to the original failure that the rerun is meaningful. If the setup changed substantially, record that difference instead of treating the result as a clean verification.
Then compare behavior against the boundary. Did the agent avoid the unsafe action? Did it ask for the missing verification? Did it preserve data scope? Did it avoid writing unsafe memory? Did it escalate at the correct moment?
- Original boundary and failure criteria are visible.
- Rerun uses the same scenario or a defensible variant.
- Tool, data, and permission setup are recorded.
- Outcome is labeled clearly as verified fixed, still failing, or regressed.
When verification fails
A failed verification is useful information. It means the fix did not address the boundary, the scenario variant found a nearby gap, or the system changed in a way that reopened the failure. The next step is not to hide the result. It is to update the finding and clarify the failure mode.
Sometimes the rerun fails for a different reason than the original. For example, the original issue may have been fake identity, while the rerun reveals tool precondition weakness. That should usually become a separate finding or a clearly noted expansion of the existing one.
Verification should keep the team honest. It prevents a prompt edit, policy note, or code change from being treated as complete until the agent behavior proves the boundary now holds.
What changed should be visible
A verification record should explain what changed without becoming a project-management artifact. The reviewer needs enough context to understand whether the fix targeted the right layer: prompt, retrieval, tool permission, authorization, workflow, memory, or escalation.
If the fix changed several layers at once, record that too. A scenario may pass after a prompt edit and a tool-gate change, but the team should know which controls now protect the boundary. That knowledge matters when future changes affect one layer but not another.
The fix description should avoid vague language like improved safety. It should say what boundary was strengthened and how the rerun tested that boundary.
Avoiding false confidence
A single passing rerun is useful, but it is not always enough. If the original scenario is high impact, run a small number of nearby variants. The variants should preserve the same boundary while changing the pressure wording, order of conversation, or source format.
False confidence can also come from changing the test too much. If the rerun removes the pressure pattern that caused the original failure, the result does not prove the fix. It only proves the agent handled an easier case.
The best verification practice is balanced: repeat the original case, add a few meaningful variants when impact is high, and keep the judgment tied to the original boundary.