synexasMATRIX
Request a demo
INDUSTRY ANALYSIS

Keep the test fixed.
Then judge the control.

A new agent control should face the evaluation that exposed the problem, not a more convenient replacement. Microsoft’s run-assert-eval example offers a useful discipline: preserve the test, review enforcement and inspect both prohibited behavior and legitimate work. Its changing applicability counts and remaining failures also show why a favorable aggregate cannot settle acceptance.

Generated editorial illustration of blank draft papers with ruled marks, a separate closed dark evidence folio and a magnifier on a daylight desk.
Generated editorial illustration: blank draft papers, a separate evidence folio and a magnifier suggest the distinction between proposed requirements and evidence. This is not a real test, audit, customer record or security evidence.
AT A GLANCE

A better score needs
A traceable comparison.

Evaluation scope

Specify one prohibited behavior and the legitimate work that must remain possible. Keep the behavior definition, cases and judging approach fixed while assessing the intended control intervention. That preserves test identity; it does not establish every safety property.

Comparable evidence

Separate exploration from comparison. Discovering additional risks can improve the next evaluation, but quietly replacing cases during a control assessment changes the question. Keep new coverage identifiable rather than folding it into an apparent before-and-after result.

Review ownership

Generated policy remains a proposal. People must review its decision rule, enforcement boundary and connection to the agent before relying on a rerun. A plausible rule is not evidence that the intended execution path actually uses it.

Acceptance decision

Ask for case-level outcomes, applicability decisions and unresolved failures, not just a headline score. Record prohibited-behavior failures separately from failures to complete legitimate work, then give a named owner the acceptance decision.

WHAT TO CHECK

Start with a failure.
Define what the test asks.

The important artifact in Microsoft’s worked example is not the generated policy alone. It is the evaluation that travels with it into the next run. Microsoft introduced run-assert-eval on September 24, 2026, describing a workflow that connects risk discovery, measurable behavior requirements, runtime controls and reevaluation. Its Command Line article says the behavior definition, cases and judging approach remain constant while the policy is the intended intervention. For an enterprise reviewer, that creates a concrete question: can the team identify the exact evaluation reused and explain what changed around it? A fresh set of easier cases could produce a better result without answering whether the original failure was addressed. Preserving the test is therefore a condition for interpreting the intervention, not a shortcut to proving causality.

The distinction matters when discovery and acceptance share a workflow. Discovery asks what else might go wrong; an intervention assessment asks how the agent behaves against a specified requirement after a reviewed change. Both are useful, but they produce different evidence. Our proposed acceptance practice is to retain the original evaluation as an identifiable record and add newly discovered coverage separately. Define prohibited behavior narrowly enough that a failure can guide a targeted decision. Alongside it, state what authorized, legitimate work should still succeed. Otherwise, a control that simply refuses more requests can look effective while making the agent less useful. Microsoft explicitly separates those outcome categories; neither should disappear into a single favorable score.

WHAT TO CHECK

Freeze the evaluation.
Reconcile what was applicable.

Microsoft reports reusing the baseline evaluation’s cached material for the governed run. Yet the headline baseline and governed aggregates have different counts of applicable conversations. Case-level reconciliation is needed before interpreting those aggregates. The fixed test set and the applicable subset are not interchangeable concepts: keeping cases available does not, by itself, explain which conversations qualified for each reported measure. Ask which cases were applicable in each run, why any classification changed and how each outcome maps back to its case. Do not read the headline figures as an identical-denominator paired comparison or draw a numerical before-and-after chart from them. A consistent judge helps preserve the measurement approach, but it does not make the judge infallible or resolve applicability differences. The useful comparison begins with traceable cases, not an assumption that two totals describe the same population.

40 and 34 count conversations—not failures

Original agent run

40 conversations were counted as relevant to Microsoft’s customer-data disclosure check.

Run with the proposed control

34 conversations were counted as relevant to that check in the run with the proposed control.

What is being counted
Conversations judged relevant to this particular check—not data disclosures, successful tasks or a safety score.
What these totals do not show
The totals differ. This article has not established which included cases match between the two runs.
What to inspect next
Check which cases were included, why they were relevant and which failures remain before drawing a safety conclusion.
Source context from Microsoft’s worked example, not a test we conducted. These totals do not establish a matched-case comparison or a safety improvement.
WHAT TO CHECK

Generation proposes.
People review enforcement.

The source explicitly says generation is not automatic approval: the policy, its enforcement description, intervention point and target wiring need review before the governed run. That is a substantive handoff, not administrative cleanup. Reviewers need to understand what decision is enforced, where it takes effect and whether the evaluated agent actually reaches that boundary. A rerun cannot compensate for an unclear enforcement contract. Our proposed review record would distinguish generated material from the version people accepted for evaluation, identify the intended control change and disclose other changes that could affect interpretation. If the team must alter cases or judging criteria, retain that as a new evaluation version rather than quietly calling it the original comparison. The objective is an inspectable decision trail, not a claim that automation has approved its own output.

WHAT TO CHECK

Keep the failures visible.
Make acceptance explicit.

Microsoft’s worked example still contains prohibited-behavior failures after the control is applied. Those remaining failures belong in the decision record, not beneath a general claim that the problem is solved. The reported sample also has no observed legitimate-work violations in the governed results; that is not proof that legitimate work is universally unaffected. Before accepting a control, ask: which requirement does this evidence address, which cases remain unresolved and who owns the next action? What legitimate-work expectations were tested, and what relevant work was outside the sample? Can the reviewer inspect applicability decisions and distinguish an observed refusal from a successful authorized outcome? The owner should record whether to accept the evaluated scope, narrow it or request further work, with reasons. These are proposed acceptance questions, not findings from a deployment we tested. The accompanying process diagram organizes those checks; it does not represent measured success.

From a defined requirement to an owned decision

  1. Define expectations
  2. Freeze the evaluation
  3. Review enforcement
  4. Compare traceable cases
  5. Retain failures and decide

Specify one prohibited behavior and the authorized work that should still succeed.

Keep the behavior definition, cases and judging approach fixed; identify later additions separately.

People review the generated policy, enforcement boundary and agent wiring before relying on the rerun.

Assess the intended control change against the preserved test; reconcile applicability per case before interpreting aggregates.

Record prohibited-behavior and legitimate-work failures separately, keep unresolved cases and name the decision owner.

Recommended review checks informed by Microsoft’s evaluation workflow. This process is not a measured outcome; applicability still needs case-level reconciliation and remaining failures stay visible.
SOURCES

Two Microsoft publications.
One bounded evidence base.

The Command Line article supplies the method and reported worked example. The September 24 Azure announcement places run-assert-eval in Microsoft’s broader agent-governance workflow; it is not independent corroboration. We have not executed the product or tested a real deployment. Read this as analysis of vendor-reported evaluation evidence and proposed acceptance questions, not proof of general safety, a real-world incident report or an approval to deploy.

Let’s make AI useful.
For your actual work.

Tell us about a job your team repeats, not your confidential files. We’ll show you how Matrix handles work like it.

Request a demo
DOWNLOAD

Security brief

Tell us where to send it.