synexasMATRIX
Request a demo
INDUSTRY ANALYSIS

The AI answer includes a confidence percentage.
What can you tell colleagues?

You are preparing an internal briefing about when a supplier says a service change begins. An AI assistant supplies a definite date and a reassuring confidence percentage. Forwarded together, they could make colleagues think the date has been checked and the number measures its reliability. Before circulation, the briefing owner needs to separate what the assistant says, what any relevant evaluation established and what the supplier’s authorized material actually supports. The percentage is a question to examine, not a substitute for that material.

Generated editorial illustration of a blank cream card in a clear stand on a wooden desk, beside a blue glass pebble. Behind them are blank sheets, a translucent sleeve, a metal paper clip and a black-rimmed magnifying lens.
AT A GLANCE

Keep three things separate.
Confidence, correctness and evidence.

Evaluation scope

The assistant’s stated confidence describes its answer. It is not automatically an observed correctness rate or support for the briefing’s factual claim.

Comparable evidence

Calibration asks whether stated confidence tracks observed correctness across relevant cases. It does not guarantee that this particular sentence is right.

Review ownership

Name the briefing owner. That person checks the number’s stated basis and brings unresolved factual questions to the responsible subject or record owner.

Acceptance decision

Do not circulate an unestablished percentage as measured assurance, invent an approval threshold or replace it with an unsupported label such as certain.

WHAT TO CHECK

Keep the answer and its number.
Ask what each represents.

Preserve the wording being reviewed, including the date, any condition and the confidence percentage. Ask what the number claims to describe: this answer, the assistant’s general knowledge or something else? An explanation supplied by the assistant can identify a claimed basis; it does not independently establish that basis. For the proposed supplier briefing, distinguish the date in the answer from the date and conditions in the supplier’s notice. A polished answer might omit a condition without changing its confident tone. The briefing owner should inspect the authorized original rather than ask only whether the assistant still stands by its answer.

Three different records can sit behind the word confidence. The assistant states a percentage. An evaluation observes which answers were correct on a defined set. A source passage supports, qualifies or fails to support the briefing’s actual claim. None can silently replace another. Calibration concerns the relationship between the first two across cases: do answers assigned similar confidence turn out correct at a corresponding frequency? Even good calibration leaves individual errors. Conversely, an answer can be correct without establishing that its attached percentage is calibrated.

WHAT TO CHECK

A number can carry information.
Its demonstrated conditions matter.

The 2022 self-evaluation study offers positive evidence with important conditions. Its abstract reports good calibration for the larger models tested on diverse multiple-choice and true-or-false questions when formatted appropriately. It also reports encouraging results from having models propose answers and then assess the probability that a proposed answer is correct. That evaluation is not the same operation as appending any confidence percentage to a conversation. Separately, the study reports difficulty calibrating predictions about knowing an answer on new tasks. Those are different questions, not a universal finding that assistants know when they are right. The 2023 confidence study provides a further counterweight to dismissing every percentage. It examined historical models fine-tuned with human feedback: training adjusted using people’s judgments of responses. Its returned abstract and introduction report that expressed confidence was often better calibrated than the models’ conditional probabilities on three historical question-answering benchmarks. Those conditional probabilities concern how likely the model is to produce a particular answer given the question or other supplied text; the reported comparison includes estimates obtained by sampling model responses. They are not independent checks of factual truth. The returned comparison also distinguishes ways of asking for confidence; the number’s origin matters. Better calibration in that comparison is not an increase in factual accuracy, nor validation of today’s product, prompt, question or briefing workflow. These studies support investigating a number’s basis, not borrowing a research result to certify your answer.

A proposed reading order, not an approval flow

  1. Answer + number
  2. Stated basis
  3. Source passage
  4. Open question
  5. Briefing owner

Keep the original wording, percentage and conditions. A self-report is not a measured correctness rate.

What does the number describe? Ask for relevant evidence linking confidence to observed correctness; a historical benchmark does not validate this answer.

Read authorized source material for the actual claim. Its support is separate from the assistant’s confidence.

Record what remains unclear. Ask the responsible subject or record owner a specific factual question.

The owner carries evidence and uncertainty into the handoff. Arrows show reading order, not approval or tested success; this proposed aid is unvalidated.

WHAT TO CHECK

Ask about the demonstrated basis.
Ask separately about the work fact.

The briefing owner can ask whoever supplied the evaluation claim: which relevant cases were checked, how was correctness determined, and did the check compare stated confidence with observed correctness? Did it use the same kind of task and way of requesting confidence? An overall accuracy result alone does not answer those questions. If no applicable evaluation is available, retain that gap rather than deriving assurance from the paper titles. Then ask the responsible subject or record owner a factual question. For the supplier notice: does this date apply to the service and audience described in the briefing, and is it conditional on anything in the notice? Include the disputed sentence and authorized source location. The person explaining evaluation evidence and the person resolving the work fact may be different people.

WHAT TO CHECK

Hand over the evidence and open question.
Do not manufacture certainty.

We propose a short reading note: the original answer and percentage, its stated basis, the relevant source passage, what that passage supports and the specific unresolved question with its follow-up owner. The figure shows the order of reading, not a classifier, tested intervention or approval procedure. Before circulation, the briefing owner compares the final wording with that note. If the date is supported but the percentage’s meaning is not established, do not describe the number as a measured chance that the date is right. Where retaining it is useful, identify it as the assistant’s unvalidated self-report. Do not replace it with certain or verified without support. If the source leaves the date conditional, keep the condition. If the applicable date remains unclear, keep the affected claim unresolved and ask the record owner the specific question. No reply is not confirmation. The briefing owner retains responsibility for the handoff under existing review rules; neither a high percentage nor this proposed reading aid grants permission to rely on the claim.

SOURCES

Two historical studies and their histories.
Not a test of your briefing.

The first study was submitted on July 11, 2022; its history ends at v4, November 21, 2022, with a comment about added references and fixed typos. The second was submitted on May 24, 2023; its v2 is dated October 24, 2023 and marked camera-ready. Four links represent two studies, not four replications. Both histories and bounded versioned excerpts were read for this article. The first excerpt ends during the introduction; the second includes the introduction and first comparison table, then truncates during related work. Unread sections, later follow-ups and an exhaustive correction or retraction search are not covered. No current assistant, supplier notice, internal briefing or workplace outcome was tested.

Let’s make AI useful.
For your actual work.

Tell us about a job your team repeats, not your confidential files. We’ll show you how Matrix handles work like it.

Request a demo
DOWNLOAD

Security brief

Tell us where to send it.