---
title: "How AI assistants choose sources: a four-model simulated-search pilot"
canonical: https://pharosone.ai/knowledge/how-ai-search-assistants-choose-sources
publisher: PharosOne (Agentic Mind, Inc.)
updated: 2026-10-23
license: "Open for reuse: crawling, copying, quoting, summarising and model training permitted without asking; citation of the title and canonical URL requested (see https://pharosone.ai/terms#reuse)"
cite_as: "\"How AI assistants choose sources: a four-model simulated-search pilot\". PharosOne, 2026-10-23. https://pharosone.ai/knowledge/how-ai-search-assistants-choose-sources"
---

# How AI assistants choose sources: a four-model simulated-search pilot

> Four API models searched fixed result pools 32 times: they chose pages from title and description, cited 89% of what they opened, and flagged vendor pages.

URL: https://pharosone.ai/knowledge/how-ai-search-assistants-choose-sources · Category: Reports · Published: 2026-10-22 · Updated: 2026-10-23 · Author: PharosOne Research · Reviewed by: PharosOne editorial

*PharosOne sells security testing and monitoring for AI agents and wrote this article. The pages under test were four of our own articles; every other page in the result lists came from other publishers.*

**In short.** In a simulated-search pilot on 22 October 2026, four models called through their APIs as search assistants (OpenAI GPT-6.1 Sol, Google Gemini 4 Argon, Anthropic Claude Sonnet 5.5 and xAI Grok 4.7) searched fixed pools of real pages, decided which ones to open from the title and a short description, and cited 85 of the 96 pages they opened. This measures the models in our harness; consumer products such as ChatGPT or Gemini search with their own engines and may choose differently. Once a page was read, original data and a direct answer won: pages the models scored as original were recommended in 83% of judgements, pages they scored as promotional in 26%. The sample is small (32 searches on 8 queries), so the differences between vendors are directions for the full run to check.

## Key facts

- Updated: 23 October 2026. Pilot run on 22 October 2026; simulated search through the models' APIs, not the consumer products.
- Assistants: `openai/gpt-6.1-sol` on Azure, `google/gemini-4-argon` on Google Vertex, `anthropic/claude-sonnet-5.5` on Amazon Bedrock, `x-ai/grok-4.7` on xAI, all through OpenRouter with the provider pinned and the vendor's default temperature.
- Sample: 4 articles, 2 queries each, 1 seed: 32 search runs, 32 blind critiques plus 32 in reversed order, 16 adversarial reviews, 4 open-web control searches.
- Pool: each query's target page plus 4–5 real pages that a search engine returned for the same query, fetched on 22 October 2026.
- Opening decides citation: 89% of opened pages were cited (85/96, 95% interval 81–93%).
- Position: no effect on blind full-text ranking (correlation 0.01, 95% interval −0.18 to 0.21).
- Cost: $0.040–0.065 per answer, depending on the model.
- Next step: read [How we audit our own AI security numbers](/knowledge/what-we-retracted-and-why).

## What do the terms mean?

**Opened.** The assistant called `open()` on the page and read its text.

**Cited.** The page's URL appears among the sources of the final answer. Citation rank is the order of first appearance; 1 is the first source cited.

**Recommended as a vendor.** The answer names the page's publisher as a company the user should consider. A separate extractor model read each answer and recorded this; it does not score quality.

**Flagged.** After opening the page, the assistant called it promotional, unsupported or wrong in its answer.

**Blind.** The model was not told which page was under test, and the pages came in random order. Blindness covers the target and the order only: our pages appeared under their normal URLs and with their own text, so a model could still see that PharosOne published them.

## How do GPT, Gemini, Claude and Grok differ as search assistants?

Each model ran the same 8 searches. "Target page" is the page under test, one of four PharosOne articles. Brackets are 95% Wilson intervals.

| Per model, pilot of 22 October 2026 | GPT-6.1 Sol (OpenAI) | Gemini 4 Argon (Google) | Claude Sonnet 5.5 (Anthropic) | Grok 4.7 (xAI) |
|---|---|---|---|---|
| Opened the target page | 5/8 [31–86%] | 4/8 [22–78%] | 4/8 [22–78%] | 4/8 [22–78%] |
| Cited the target page | 5/8 [31–86%] | 4/8 [22–78%] | 4/8 [22–78%] | 3/8 [14–69%] |
| Mean citation rank when cited | 1.8 (n=5) | 1.5 (n=4) | 1.0 (n=4) | 1.0 (n=3) |
| Recommended the publisher as a vendor | 2/8 [7–59%] | 4/8 [22–78%] | 0/8 [0–32%] | 1/8 [2–47%] |
| Flagged the target page, when opened | 3/5 | 4/4 | 4/4 | 2/4 |
| Tool calls per answer (cap 8) | 6.75 | 7.25 | 4.9 | 7.0 |
| Pages opened per answer | 3.0 | 3.6 | 2.75 | 2.6 |
| Sources cited per answer | 2.75 | 3.75 | 2.75 | 2.0 |
| Opened pages that were cited | 22/24 | 29/29 | 22/22 | 12/21 |
| Trust score, target pages / other pages (blind, 1–5) | 3.0 / 3.1 | 4.0 / 3.9 | 3.0 / 3.2 | 3.0 / 3.3 |
| Mean rank of the target page in blind critique (pools of 5–6) | 2.9 | 2.25 | 2.0 | 2.6 |
| Issues per page in the adversarial review | 60 (GPT-6.1 Sol Pro) | 9 | 24 (Claude Opus 5.5) | 28 |
| Cost per answer | $0.040 | $0.065 | $0.056 | $0.046 |

Every interval in the first four rows overlaps, so the pilot cannot rank the four assistants. Three patterns are worth checking at full size. Gemini was the most generous: it gave the target pages the highest trust score of the four and recommended their publisher in half of its answers. Claude placed the target pages first whenever it cited them and never recommended the publisher; it quoted them as what "the vendor says". Grok searched the most (34 searches in 8 answers, against 17 for Claude), cited the fewest sources and cited 4 pages it never opened.

## Which pages does an assistant open?

In these fixed pools, mostly the ones whose title and description matched the question. The pilot did not vary titles independently of the rest of the page, so this is an observation; it does not measure the effect of a title on its own. The search tool returned each result as a title, a URL and the first 35 words of the page's meta description. The models opened about 3 results per answer and cited almost only what they opened: 85 of 96 opened pages were cited, and 5 of 90 citations went to unopened pages. The decision to open is the gate.

Our four test pages show the pattern:

- A price report with data we collected ourselves ([What an AI agent pentest costs in 2026](/knowledge/ai-agent-pentest-cost-2026)) was opened in 8 of 8 searches.
- A product page was opened in 4 of 4 searches that asked for exactly that kind of product, and in 1 of 4 searches that asked a broader how-to question.
- A threat explainer was opened in 4 of 4 searches for "what is indirect prompt injection and how do I test for it". For "can a strong system prompt stop indirect prompt injection in a RAG agent?" it was opened in 0 of 4, although its FAQ answers that question in one line. Neither its title nor its description mentioned it.
- A note on a new PHP vulnerability was opened in 0 of 8 searches, including runs where it was the first result. The models opened the advisories and news reports that first published the facts, such as [Tenable's FAQ](https://www.tenable.com/blog/php2shell-cve-2026-109618-cve-2026-109619-frequently-asked-questions-about-critical-php-remote).

Rewrites of public material lose to the original source at this step. For the broader monitoring question, all four models opened [Simon Willison's post on the lethal trifecta](https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/) and three cited it first.

## What makes an assistant recommend a page it has read?

Original information and a neutral tone. In the blind critique each model read the full text of every page in the pool, labelled A–F in random order, and scored each page from 1 to 5 for usefulness, trust, originality and marketing tone. 176 page judgements in total, 44 per model.

- Pages scored 4–5 for originality were recommended in 68 of 82 judgements (83%, 95% interval 73–90%) and were the best page in 25 of 32 critiques. Pages scored 1–2 were recommended in 8 of 38 (21%, 11–36%).
- Pages scored 4–5 for marketing tone were recommended in 19 of 72 (26%, 18–38%). Pages scored 1–2 were recommended in 48 of 54 (89%, 78–95%).
- Marketing tone and trust moved against each other: correlation −0.68 across all judgements, between −0.58 (GPT) and −0.77 (Grok) per model.

These intervals treat each page judgement as independent. They come from 32 critiques, so the real uncertainty is wider. The same model scored a page's originality and tone and decided whether to recommend it, so these are associations within one judgement and do not show that a neutral tone causes a recommendation. Our price report, built on data we collected, was ranked the best page in 8 of 8 critiques. Our threat explainer, which retells public sources and ends with a sales section, was ranked best in none and scored 4.25 for marketing tone against 2.2 for the other pages in its pool.

## How do assistants treat a page written by a vendor?

They identify it and tell the user. When the models opened our pages, they flagged them as promotional or unsupported in 13 of 17 cases (76%, 95% interval 53–90%). Every model that opened the price report attributed its numbers to the vendor and noted that they could not be checked independently. The publisher was recommended as a vendor in 7 of 32 answers (22%, 11–39%), unevenly across models (table above).

In blind critique our pages scored higher for marketing tone than the other pages in the same pools (3.5 against 3.0) and about the same for trust (3.25 against 3.4). Most of the other pages were vendor blogs as well. The price report now opens with a one-line disclosure of who wrote it and what PharosOne sells; whether a disclosure changes the flag rate is one of the questions for the full run.

## Do assistants catch contradictions between pages?

Yes, when they read the related pages. In the adversarial pass one stronger model per vendor read the target page together with the pages it links to on the same site and listed every weak claim, unsupported number and contradiction. Across the four pages, 16 issues were raised on the same sentence by all four vendors, and 51 by three or four. One sentence, about when our runtime guard returns its verdict, was worded differently on the product page and on the other three articles; reviewers from all four vendors flagged the mismatch. The pages now use one wording.

The vendors differ in how much they report: from 9 issues per page (Gemini 4 Argon) to 60 (GPT-6.1 Sol Pro), many of them minor. Counting the issues several vendors agree on is more useful than raw totals. Claude Opus 5.5 declined to review the PHP vulnerability note through the provider's content filter; Claude Sonnet 5.5 reviewed it instead.

## Does the position in the results matter?

Little. In the blind critique we ran every judgement a second time with the page order reversed. The change in a page's position did not predict the change in its rank: correlation 0.01 (95% bootstrap interval −0.18 to 0.21; 32 paired judgements, 176 page pairs), and the same page was best in both orders in 24 of 32 pairs (75%, 58–87%). While browsing, the first result was opened in 22 of 32 searches (69%, 51–82%) and the other positions in 41–62%: a weak preference that did not rescue the PHP note when it was listed first.

## What should you change on a page written for AI search?

1. Put the answer to the reader's question in the title and the meta description. The assistant decides whether to open the page from those lines; an answer that sits only in the FAQ is not found.
2. Publish something only you have: data you collected, a comparison with its method, a decision you can explain. A retelling of public sources loses to the source.
3. Say in the first line who wrote the page and what you sell. The assistants identified vendor authorship on their own and told the user.
4. Keep sales copy out of explainers. All four blind reviewers singled out the sales section of our threat explainer, and it was recommended in 2 of 8 critiques.
5. Check every number and claim against the other pages of the site. Reviewers read the linked pages and flag any sentence that says something different.
6. Give every number its population, date and source. Unsupported claims and numbers (115 of 486 issues) and contradictions within or across pages (114) were the two largest groups of issues the adversarial reviewers raised.

## Method, sample and limitations

The test ran on 22 October 2026 in content-eval, an evaluation harness PharosOne built for its own pages. For each query we assembled a pool: the target page and 4–5 real pages that a search engine returned for the same query, fetched and cleaned to visible text with one word cap (6,000 words) for all pages. Each model then ran four passes.

1. Browsing (blind). A search-assistant system prompt with two tools: `search()` returns the pool in an order shuffled per query and model; `open()` returns a page's text. At most 8 tool calls, then an answer with citations. Claude Haiku 5.5 extracted from each transcript what was taken from the target page, whether the publisher was recommended and whether the page was flagged.
2. Critique (blind). The full text of every page, labelled A–F in random order; scores, a recommendation per page and a ranking.
3. Critique in reversed order, to measure position bias.
4. Adversarial review (not blind). One stronger model per vendor reads the target page with its related pages: GPT-6.1 Sol Pro, Gemini 4 Argon, Claude Opus 5.5 and Grok 4.7.

An open-web control asked Perplexity Sonar Pro the same queries on the live web. It did not find pharosone.ai for any of them; the site was not indexed at the time. All text found in pages that addressed a model (14 passages, all quoted attack examples inside articles about prompt injection) was recorded and never followed. The pilot cost $11.25, of which $1.65 paid for the 32 browsing answers.

Limitations:

- These are the models behind AI search assistants, run through our harness with one system prompt and one result list. ChatGPT, Gemini, Claude and Grok as consumer products have their own search engines, ranking and prompts, and may choose differently.
- `search()` returned the same pool whatever the model typed. A real engine would return different results to a reworded query.
- Every target page came from one publisher on one subject, AI security. Our pages were clean text; the other pages kept some page furniture after cleaning, which slightly favours ours.
- Pages that render only with JavaScript, such as OWASP's GenAI project site, could not be cleaned and were left out of the pools.
- 8 searches per model and one seed. Per-model rates have intervals 40–60 points wide.
- The extractor sometimes credited claims to a page the model had not opened; we count claims only when the page was cited and flags only when it was opened.
- The pages were tested as they stood on 22 October 2026. Several have been edited since in response to the findings.

## What comes next?

A larger run on revised versions of the library's articles is planned. Its scope, queries and sample size are not fixed yet, and the pilot's intervals say nothing about how precise it will be. If it runs, this page will be updated with its numbers and the version of each page it tested.

## FAQ

### How does an AI search assistant decide which page to open?

From the title, the URL and a short description in the result list. In our blind pilot of 22 October 2026 (32 searches, four models) the search tool showed a title, a URL and the first 35 words of each page's description, and the models opened 3 of 5 or 6 results per answer on average. A page whose title and description did not mention the question was not opened, even when its FAQ answered it word for word.

### Do AI assistants cite pages they have not opened?

Rarely. In the pilot, 85 of the 96 pages the models opened were cited (89%, 95% interval 81–93%), and 5 of 90 citations went to pages the model had not opened, 4 of them from Grok 4.7.

### Does the position in the search results change which source an AI assistant cites?

Little. When the models read every page in full, reversing the order did not change the ranking (correlation 0.01, 95% interval −0.18 to 0.21, 32 paired judgements). While browsing, the first result was opened in 69% of searches against 41–62% for the other positions.

### Do AI assistants trust content written by a vendor?

They recognise it and say so. When the models opened the vendor-written pages under test, they flagged them as promotional or unsupported in 13 of 17 cases, and recommended the publisher as a vendor in 7 of 32 answers. Pages scored as promotional in blind review were recommended 26% of the time, against 89% for pages scored as neutral.

### Which AI assistant recommends vendors most often?

In the pilot, Gemini 4 Argon recommended the publisher of the test pages in 4 of 8 answers, GPT-6.1 Sol in 2, Grok 4.7 in 1 and Claude Sonnet 5.5 in none. With 8 answers per model the intervals overlap, so this is a direction for the full run to check.

## Sources

1. content-eval pilot: blind SERP arena, 4 articles, 8 queries, 4 vendors (internal dataset). PharosOne Research, 2026-10-22.
2. [The lethal trifecta for AI agents: private data, untrusted content, and external communication](https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/). Simon Willison, 2025-06-16.
3. [php2shell (CVE-2026-109618, CVE-2026-109619): Critical PHP RCE FAQ](https://www.tenable.com/blog/php2shell-cve-2026-109618-cve-2026-109619-frequently-asked-questions-about-critical-php-remote). Tenable, 2026-10-21.
4. [How we audit our own AI security numbers](https://pharosone.ai/knowledge/what-we-retracted-and-why). PharosOne Research, 2026-10-23.
5. [What an AI agent pentest costs in 2026](https://pharosone.ai/knowledge/ai-agent-pentest-cost-2026). PharosOne Research, 2026-10-22.

Source: PharosOne, "How AI assistants choose sources: a four-model simulated-search pilot", https://pharosone.ai/knowledge/how-ai-search-assistants-choose-sources, updated 2026-10-23.

---

PharosOne is a product of Agentic Mind, Inc., an independent AI agent security company. Site map for agents: https://pharosone.ai/llms.txt