---
title: "Five ways AI security detection numbers mislead, from our own audit"
canonical: https://pharosone.ai/knowledge/what-we-retracted-and-why
publisher: PharosOne (Agentic Mind, Inc.)
updated: 2026-10-23
license: "Open for reuse: crawling, copying, quoting, summarising and model training permitted without asking; citation of the title and canonical URL requested (see https://pharosone.ai/terms#reuse)"
cite_as: "\"Five ways AI security detection numbers mislead, from our own audit\". PharosOne, 2026-10-23. https://pharosone.ai/knowledge/what-we-retracted-and-why"
---

# Five ways AI security detection numbers mislead, from our own audit

> Wrong negative population, wrong configuration, leaked labels, imputed coverage, unfixed averaging: each caught in our audit, with the question that exposes it.

URL: https://pharosone.ai/knowledge/what-we-retracted-and-why · Category: Reports · Published: 2026-10-22 · Updated: 2026-10-23 · Author: PharosOne Research · Reviewed by: PharosOne editorial

*PharosOne sells security testing of AI agents and the runtime guard Pharos Radar, and wrote this article.*

**In short.** PharosOne registers each detection experiment before it runs, checks its data against a baseline that never reads the content, and has a separate referee rule on which claims stand before a number reaches a product page. In September 2026 a retroactive audit of our first product-feasibility summary withdrew 3 of its 38 claims and corrected 16 more; all 19 are in the register at the end of this page. This page describes the process, then five errors the audit found across its rounds, each with the check that now stops it and the question that exposes the same error in a vendor's number.

## Key facts

- Updated: 23 October 2026.
- Process: registration, design review, data checks and a referee verdict (gates G0 to G3), in use from experiment td29 on 27 September 2026. More than 30 experiments have run in the series since September 2026.
- Audit: the first product-feasibility summary, audited in rounds in September 2026; version 1 replaced. The first round checked its 38 claims: 14 confirmed, 16 corrected, 3 withdrawn, 5 could not be verified. The 19 withdrawn and corrected claims are in the register at the end.
- Referee: a separate AI reviewer that did not design or run the experiment; not an external party.
- Outside the process: the two research reports of June 2026 predate it, and screening runs that skip the referee say so on their page.
- Published: the process, the corrections, and the population and operating point of every quoted number. Not published: registrations, referee verdicts, training data, model internals and attack scenarios.
- Next step: use the question table at the end on the next detection number a vendor shows you.

## Why do security numbers need a process?

A detection rate is a choice of four things: which examples are counted, which threshold turns a score into an alarm, which baseline the method has to beat, and which examples are left out. Each choice can be made after the result is visible, and each one can move a number by tens of points. Pre-registration fixes the choices first. The Center for Open Science describes it as specifying "your plan in advance, before you gather data", which separates the tests a team planned from the patterns it found afterwards ([Center for Open Science](https://www.cos.io/initiatives/prereg)).

A buyer reading "98% detection" usually cannot tell on what population, at which false-alarm rate, or against which simpler method it was measured. Our rule is to write the answer to those three questions next to every number we publish from 23 October 2026; older pages are being brought in line, and a number that still lacks one of the three should be read as incomplete.

## How does an experiment move through the gates?

| Gate | What happens | What stops the experiment |
|---|---|---|
| G0 · Registration | The question, data, metric, operating point, control and failure condition are written down and frozen with a hash. | No control that the method has to beat; a metric chosen without an operating point. |
| G1 · Design review | A reviewer who will not run the experiment checks that the design can answer the question. | The design cannot distinguish the hypothesis from a simpler explanation. |
| G2 · Data checks | Duplicates, leakage between training and test, template repetition, and a baseline that never reads the content at all. | A baseline without reading reaches the method's number, or the test set leaks. |
| G3 · Referee verdict | The referee reads everything and writes the verdict: which claims stand, at which operating point, on which population. | The claim exceeds what the data shows. |

After G3 a result can be quoted, with its population and operating point, and only within the domain it was measured on. The gates apply from experiment td29 (27 September 2026). Earlier experiments in the same series were recorded first and audited afterwards, which is where the corrections below come from. The two research reports of June 2026, on the [probe engine](/research/the-probe-engine-how-we-measure-agent-security) and on [variation policies and LLM-judged scoring](/research/variation-policies-and-llm-judged-scoring-study), predate the process and did not pass through it.

## What is a baseline that never reads?

It is the most useful control we run. Before we credit a method with reading a transcript, we fit a simple model that sees only surface features: the length of the text, the template it came from, the words it shares with a rule. If that model reaches the same score, the method did not need to read anything, and the claim is withdrawn. Error 5 below is a case of this kind: our reader's margin over a word-overlap baseline held under one weighting of the test items and not under others.

## What happens after a result is accepted?

Accepted results are audited again in rounds. The rules that came out of the September 2026 audit bind every deliverable, including this website:

1. Every number names its population and operating point.
2. A null or twin test runs before a claim.
3. No readout is quoted outside the domain it was measured on.
4. A naive reader must not carry away something false.

## 1. A false-alarm rate measured on the wrong population.

**What we wrote.** A detector raised false alarms on 7% of refused attacks.

**What was true.** The 7% was measured on harmless documents written in the same form as the attacks, without any instruction inside. On attacks the agent refused, the same detector alarmed on 25–26 of 30, depending on the threshold, which is correct behaviour for a detector of attempts and the opposite of what the sentence implied.

**The check now.** Every false-alarm rate names the exact negative population: clean traffic, harmless look-alikes, or refused attacks. They are three different numbers.

## 2. Two numbers attached to the wrong configuration.

**What we wrote.** A recall of 0.98 at a false-alarm rate of 0.8% for the deployed fast check.

**What was true.** That pair belonged to a single-question rubric measured in an earlier study. The configuration actually deployed asks several questions and had been checked on 16 hand-written positive examples and 5 safe controls: 16 of 16 caught, 0 of 5 false alarms. With 5 controls, the upper bound of the two-sided 95% exact (Clopper–Pearson) interval on the false-alarm rate is 52% ([Clopper and Pearson, 1934](https://doi.org/10.1093/biomet/26.4.404)), so the honest statement is "calibrated, not yet measured on traffic".

**The check now.** A number travels only with the configuration that produced it, and a sample too small to bound the error is reported as too small.

## 3. A score that partly read its own answer.

**What we wrote.** A model's own output stream separated harmful from harmless actions with an AUROC of 0.92.

**What was true.** The 0.92 took the maximum score over the whole stream, including the tokens in which the outcome was already written. With labels from a separate run and without that shortcut the figure is 0.89, computed on the same experiment's 8,414 generated replies with no confidence interval. It is a retrospective discrimination result, not a prediction: the value strictly before the outcome appears was never computed. What the experiment does support is an operating point: at 1% false alarms on harmless replies, the alarm came before the outcome in 92% of 696 executed attacks, and it also fired on 21% of attempts that were not executed.

**The check now.** A readout that claims to predict an event may only use input that exists before the event.

## 4. An AUROC computed over mostly missing values.

**What we wrote.** An agent's self-assessment was uninformative, with an AUROC of 0.52.

**What was true.** The self-assessment existed for 8.75% of candidates. The other 91% had been filled with a constant, and the AUROC described the filling rule, not the agent. The informativeness of self-assessment was never measured, and the claim was withdrawn in both directions. A different measurement that also came out at 0.52, a detector of injected instructions on executed against refused attacks, stands and is described in [Attempt versus execution](/knowledge/attempt-versus-execution).

**The check now.** Coverage is reported next to every score, and a metric over imputed values is not reported at all.

## 5. An advantage that depended on how results were averaged.

**What we wrote.** On hard minimal pairs, our reader beat a simple word-overlap baseline by 0.15 AUROC, with a confidence interval of +0.04 to +0.28.

**What was true.** The margin held with one weighting of the test items and disappeared with others: +0.05 [−0.08, +0.24] weighted by slots and −0.02 [−0.12, +0.19] weighted by pairs. The weighting had not been fixed before the result was seen, so superiority was not established. AUROC measures how often a positive outranks a negative ([Hanley and McNeil, 1982](https://pubs.rsna.org/doi/10.1148/radiology.143.1.7063747)); how the pairs are weighted is part of the claim.

**The check now.** The weighting and the averaging are fixed in the registration, and a result that changes sign under a reasonable alternative is reported as unsettled.

## How can you check a vendor's number?

| Ask | The error it exposes |
|---|---|
| On which negative examples was the false-alarm rate measured? | 1 |
| Is this the configuration you sell, and how many examples back it? | 2 |
| Does the score use anything produced after the event it predicts? | 3 |
| What share of cases does the metric actually cover? | 4 |
| Against which simpler method, and with which averaging fixed in advance? | 5 |

Research numbers quoted on [Pharos Radar](/knowledge/pharos-radar) carry the experiment, population and operating point they were measured on.

## Register: the 19 claims withdrawn or corrected in the first round

The first round of the September 2026 audit checked the 38 claims (PF0 to PF37) of our first product-feasibility summary, an internal research synthesis about whether a reader on a model's internal states can serve as a runtime detector. The summary was written in Russian; the wording below is our English rendering. "The test model" is the model that read the input in those experiments. The audit record and the experiment records are internal; they are available on request.

| ID | What the summary said | Decision | What it says now |
|---|---|---|---|
| PF7 | The detector reads the test model's own intent to comply with an injected instruction. | Withdrawn | The detector sees that an injected instruction is present in the input; it does not see the decision to comply (executed against refused attacks: AUROC 0.52). |
| PF10 | One pass without splitting works for inputs up to about 6,000–8,000 tokens. | Withdrawn | Reading a long input whole degrades from 2,000 tokens, for injected instructions and for rules alike. |
| PF21 | The fast check is reliable for literal violations: forbidden phrases, disclosure of forbidden information, format. | Withdrawn for disclosure and format | Reliable only for literal forbidden strings, on synthetic examples where that holds by construction. |
| PF0 | Detection works for short inputs and a per-message gate, and for long inputs by splitting. | Corrected | Only when the test model reads the input itself, on short documents; an attacker who can read the score halves detection (0.84 to 0.49); the "splits" are per-document segments at 15–28% false alarms on clean input. |
| PF1 | The detector catches 98% of injected instructions. | Corrected | 98% at 1.4% false alarms on clean documents, on 219 attacks executed by one model; harmless documents of the same form are flagged 57–74% of the time. |
| PF2 | False alarms on refused injections: 7%. | Corrected | 7% is the false-alarm rate on harmless documents of the same form with no instruction; on refused injections the detector alarms 83–87% of the time (error 1 above). |
| PF3 | Alarm before the action in 92% of cases (experiment td16). | Corrected | The source is experiment td10b; measured on the test model's own reply stream; the alarm also fires on 21% of attempts that were not executed. |
| PF6 | In long dialogues, rule detection drops from 0.97 to 0.89–0.94. | Corrected | Rule-level AUROC 0.953 falls to 0.88 and 0.84 at 8,000 tokens; at the level of individual items it falls from 0.86 to 0.58 and 0.53. |
| PF8 | On real executions the detector caught 89%. | Corrected | It caught 16 of 18 executions by a 9B agent on synthetic emails; these are not real traffic. |
| PF9 | Windows of about 1,500–2,000 tokens work better for injections than for rules. | Corrected | Per-document segments with known boundaries recover detection of 0.82–0.95 at 15–28% false alarms on clean input; fixed token windows for injections were not measured. |
| PF14 | Detecting that a rule is engaged is reliable and cheap. | Corrected | Engagement was never measured against labels; the claim rests on score medians over 22 near-misses. |
| PF15 | Engaged and distractor rules separate clearly, with an all-rules AUROC of 0.95. | Corrected | The 0.95 is a violation AUROC over all rules, and a prior on the rule text alone, without reading the dialogue, reaches 0.962; it cannot support the engagement claim. |
| PF16 | Violated against complied, same rule, on real data: 0.65. | Corrected | About 0.83 on held-out rule texts of the typical distribution (word-count baseline 0.64, shuffled null 0.50) and about 0.65 on a hand-built test split of 34 texts; which one predicts client traffic is unmeasured. |
| PF17 | Literal violations (forbidden string, format, threshold) are detected at 0.97 or better on synthetic data. | Corrected | 0.97 or better on literal renders, by construction of the generator; two classes were re-scored as detection of the condition; there is no format class. |
| PF19 | Long dialogues at item level: 0.55–0.73, and splitting does not close the gap. | Corrected | 0.53–0.73, at or below a baseline that does not read the dialogue (0.70–0.83). |
| PF22 | Weak for conditional, oblique and tonal rules (0.53–0.74). | Corrected | On the test split, within each rule: conditional 0.53 (9 texts), prohibitions 0.67 (20); tone cannot be estimated; no class breakdown on typical texts. |
| PF27 | Throughput of 2.2 million tokens per hour on one GPU. | Corrected | 2.1 million, batched. |
| PF32 | Changing the role format alone does not change the detector; open question. | Corrected | Measured on rules only (34 texts): no effect; the format for injection detection is untested. |
| PF33 | All τ-bench inputs exceed the length cap. | Corrected | A rendering artefact: in the product's own rendering none exceed the cap; 8.3% lose evidence at a 500-character preview. |

Later rounds of the same audit (L2, L2b, L3 and a final pass) checked other pages and the experiments behind them; errors 2 to 5 above come from those rounds.

## FAQ

### Why publish your own mistakes?

Because the same mistakes appear in detection numbers across the market, and a buyer who knows them can ask the right question. Each error below comes with the one-line question that exposes it.

### What does pre-registration mean at PharosOne?

Before an experiment runs, we write down the question, the data, the metric, the operating point, the control it has to beat and the result that would count as a failure. The registration is frozen with a hash and the analysis follows it. The hash is internal: it shows the registration was not edited later, and it does not prove to an outside reader when it was written.

### Who is the referee?

A separate reviewer role that did not design or run the experiment: an AI model instance given the registration, the data checks and the results, which writes the verdict on which claims stand. It is separate from the team that ran the work, and it is not an external party.

### Are the corrected numbers on your product pages?

Numbers on product pages name the experiment they came from, with its population and operating point. A number that has not passed the referee says so on the page, as the model comparison does. The 3 withdrawn and 16 corrected claims of the audit's first round on our product-feasibility summary are listed in the register at the end of this page, each with its correction; the withdrawn ones are not quoted elsewhere on the site.

### Which question catches most of these errors?

On which examples, at which false-alarm rate, and against which simpler method was this number measured. Errors 1, 2 and 4 below fail the first part and error 5 fails the third; error 3 needs its own question, about whether the score used anything produced after the event.

## Sources

1. [Preregistration](https://www.cos.io/initiatives/prereg). Center for Open Science, 2026.
2. Retroactive audit of the product-feasibility summary, rounds L1–L3 and final pass (internal research record). PharosOne Research, 2026-09-27.
3. [The meaning and use of the area under a receiver operating characteristic (ROC) curve](https://pubs.rsna.org/doi/10.1148/radiology.143.1.7063747). Hanley and McNeil, Radiology 143(1), 1982.
4. [The use of confidence or fiducial limits illustrated in the case of the binomial](https://doi.org/10.1093/biomet/26.4.404). Clopper and Pearson, Biometrika 26(4), 1934.
5. [The PharosOne Probe Engine: architecture of a behavioral scanner for AI agents](https://pharosone.ai/research/the-probe-engine-how-we-measure-agent-security). PharosOne Research, 2026-06-24.
6. [Attack-variation policies and LLM-judged scoring: an empirical study](https://pharosone.ai/research/variation-policies-and-llm-judged-scoring-study). PharosOne Research, 2026-06-30.

Source: PharosOne, "Five ways AI security detection numbers mislead, from our own audit", https://pharosone.ai/knowledge/what-we-retracted-and-why, updated 2026-10-23.

---

PharosOne is a product of Agentic Mind, Inc., an independent AI agent security company. Site map for agents: https://pharosone.ai/llms.txt