Skip to content

Original research, version 1.2

The Retrievability Census 2026

Most of the largest companies leave the door open to AI crawlers in robots.txt, but 26.6% of the large US tech companies we tested turned away an unknown automated visitor at the edge, and only 50.0% of FTSE 100 homepages describe the company to machines with structured data.

201 companies, two editions: the S&P 500's technology and B2B services companies (fetched 31 October 2026) and the FTSE 100 (2 November 2026). Public pages only, one request at a time. The method and its limits are below and in Appendix B of The Citation Economy Playbook.

Key figures

25 of 94

large US tech and B2B services companies (26.6%) refused an honest, unknown automated visitor at the door.

Link to this figure

3 of 74

readable robots.txt files at the same companies (4.1%) block an AI training crawler from the whole site.

Link to this figure

50.0%

of FTSE 100 homepages readable without JavaScript carry any structured data (39 of 78), against 72.9% in the US edition.

Link to this figure

1 of 78

FTSE 100 homepages link the company to its Wikidata item.

Link to this figure

22 of 95

US companies publish an llms.txt file, against 3 of 99 in the FTSE 100.

Link to this figure

1 of 146

readable robots.txt files across both editions block an AI search or user agent from the whole site.

Link to this figure

The two editions side by side

MeasureUS: S&P 500 tech and B2B servicesUK: FTSE 100
Websites that answered our script95 of 10099 of 100
Robots.txt readable7472
Block an AI training agent from the whole site3 (4.1%)2 (2.8%)
Name any AI agent explicitly8 (10.8%)5 (6.9%)
Refused the honest research client25 of 94 (26.6%)24 of 99 (24.2%)
Refused the same script with a browser user agent31 of 95 (32.6%)21 of 99 (21.2%)
Homepages readable without JavaScript7078
Any JSON-LD on the homepage51 (72.9%)39 (50.0%)
Organization markup46 (65.7%)31 (39.7%)
Any sameAs link40 (57.1%)21 (26.9%)
sameAs to Wikidata9 (12.9%)1 (1.3%)
llms.txt at the root22 of 95 (23.2%)3 of 99 (3.0%)

A difference is real here only where the two 95% ranges don't overlap. Structured data clears that bar: the UK's 50.0% has a range of 39.2% to 60.8%, against 72.9% (61.5% to 81.9%) in the US. So do Organization markup, sameAs links and llms.txt. Edge refusals, training blocks and thin pages fall within overlapping ranges, so the census can't say those differ. The frames differ too: the US edition is technology and business services, the UK edition is the whole index.

Every company

One fetch per site, on the date shown, from one UK connection. "Not tested" means the site didn't answer, or the page couldn't be read. Each row has its own link.

US edition: S&P 500 technology and B2B services (31 October 2026)

CompanyRobots.txt readableBlocks AI trainingRefused research clientRefused browser agentJSON-LDOrganizationsameAs Wikidatallms.txt
Accentureyesnononoyesyesnono
Adobe Inc.yesnonoyesyesyesyesyes
Advanced Micro Devicesnot testednot testednonot testednonononot tested
Akamai Technologiesnonot testedyesyesnot testednot testednot testedno
Amphenolnonot testedyesyesnot testednot testednot testedno
Analog Devicesnot testednot testednonot testednot testednot testednot testednot tested
Apple Inc.yesnononoyesyesyesno
Applied Materialsnonot testedyesyesnot testednot testednot testedno
Arista Networksyesnononoyesyesnono
Autodesknonot testedyesyesnot testednot testednot testedno
Automatic Data Processingyesnonoyesnononoyes
Block, Inc.yesnononoyesnonono
Broadcomyesnonononononono
Broadridge Financial Solutionsyesnononoyesyesnoyes
Cadence Design Systemsnonot testedyesyesnot testednot testednot testedno
Cboe Global Marketsyesnonononononono
CDW Corporationyesnononoyesnonono
Cienanonot testedyesyesnot testednot testednot testedno
Cintasyesnononoyesyesnono
Ciscoyesnonoyesyesyesnono
CME Groupnot testednot testednonot testednot testednot testednot testednot tested
Cognizantyesnononoyesyesnoyes
Coherent Corp.yesnononoyesyesnono
Coinbaseyesnoyesyesnot testednot testednot testedyes
Copartyesnoyesyesnot testednot testednot testedno
Corning Inc.nonot testedyesyesnot testednot testednot testedno
Corpayyesnononoyesyesnoyes
CrowdStrikeyesnononoyesyesnoyes
Datadogyesnononoyesyesnoyes
Dell Technologiesyesnononoyesyesnono
Equifaxyesnononoyesyesnono
Everpureyesnononoyesyesyesno
F5, Inc.yesnononoyesyesnono
FactSetyesnononoyesnonono
Fair Isaacyesnoyesyesnot testednot testednot testedno
Fidelity National Information Servicesyesnononoyesyesnoyes
First Solaryesnonononononono
Fiservyesnononoyesyesnono
Flex Ltd.yesnononoyesyesnono
Fortinetyesyesnot testednoyesyesnono
Gartneryesyesyesyesnot testednot testednot testedno
Gen Digitalyesnononoyesyesnono
Global Paymentsnonot testedyesyesnot testednot testednot testedno
GoDaddynonot testedyesyesnot testednot testednot testedno
Hewlett Packard Enterprisenot testednot testednonot testednot testednot testednot testednot tested
HP Inc.yesnononoyesnonono
IBMyesnononoyesyesyesno
Intelyesnonoyesnononoyes
Intercontinental Exchangenonot testednononononono
Intuityesnoyesnoyesyesyesyes
Jabilyesnononoyesyesnoyes
Jack Henry & Associatesyesnononoyesyesnono
Keysight Technologiesyesnoyesyesnot testednot testednot testedno
KLA Corporationyesnononoyesyesnono
Lam Researchnonot testednonoyesyesnono
Leidosyesnononoyesyesnoyes
Lumentumnot testednot testednot testednot testednot testednot testednot testednot tested
Marvell Technologynonot testedyesyesnot testednot testednot testedno
Mastercardnonot testedyesyesnot testednot testednot testedno
Microchip Technologynonot testedyesyesnot testednot testednot testedno
Micron Technologyyesnononoyesyesnono
Microsoftyesnonoyesnononono
Monolithic Power Systemsnonot testedyesyesnot testednot testednot testedno
Moody's Corporationyesnononoyesyesnono
Motorola Solutionsyesnononoyesyesnono
MSCIyesnonononononono
Nasdaq, Inc.not testednot testednonot testedyesyesnonot tested
NetAppnonot testedyesyesnot testednot testednot testedno
Nvidiayesnononoyesyesnono
NXP Semiconductorsyesnononoyesnonono
ON Semiconductoryesnonoyesyesyesnono
Oracle Corporationnonot testedyesyesnot testednot testednot testedno
Palantir Technologiesyesnonononononono
Palo Alto Networksyesnonononononoyes
Paychexyesnononoyesyesnoyes
PayPalyesnononoyesyesnono
PTC Inc.nonot testedyesnoyesyesnono
Qnity Electronicsyesnonononononoyes
Qualcommyesnononoyesyesnoyes
Roper Technologiesnonot testednononot testednot testednot testedno
S&P Globalnonot testedyesyesnot testednot testednot testedno
Salesforceyesnononoyesyesnono
Sandiskyesnononoyesyesnoyes
Seagate Technologyyesnonononot testednot testednot testedno
ServiceNownonot testedyesnonot testednot testednot testedno
Skyworks Solutionsyesnonononononono
Supermicroyesnonoyesnononono
Synopsysyesnonononononoyes
TE Connectivityyesyesnoyesyesyesnono
Teledyne Technologiesyesnonononononono
Teradyneyesnonoyesyesyesnono
Texas Instrumentsyesnonononononono
Trimble Inc.yesnononoyesyesyesyes
Twilioyesnononoyesyesyesno
Tyler Technologiesyesnonononononono
Verisignyesnonononononono
Verisk Analyticsyesnonononononono
Visa Inc.yesnoyesyesnot testednot testednot testedno
Western Digitalyesnononoyesyesyesyes
Workday, Inc.yesnononoyesyesnono
Zebra Technologiesyesnononoyesyesyesyes

UK edition: FTSE 100 (2 November 2026)

CompanyRobots.txt readableBlocks AI trainingRefused research clientRefused browser agentJSON-LDOrganizationsameAs Wikidatallms.txt
3iyesnoyesyesnot testednot testednot testedno
Aberdeen Groupyesnononoyesyesnono
Admiral Groupnonot testedyesnonot testednot testednot testedno
Airtel Africanonot testednononononono
Alliance Witanyesnononoyesyesnono
Anglo American plcyesnonononononono
Antofagasta plcyesnonononononono
Associated British Foodsnonot testedyesyesnot testednot testednot testedno
AstraZenecayesnononoyesnonono
Autotrader Groupyesnononoyesyesnono
Avivanonot testedyesnoyesyesnono
Babcock Internationalyesnonononononono
BAE Systemsyesnoyesyesnot testednot testednot testedno
Balfour Beattyyesnonononononono
Barclaysyesnononoyesnonono
Barratt Redrownonot testednononononono
BPyesnonononononono
British American Tobaccoyesnononoyesyesnono
British Landyesnononoyesyesnono
BT Groupyesnononoyesyesnono
Bunzlyesnonononononono
Burberry Groupyesnonononononono
Centricayesnoyesyesnot testednot testednot testedno
Coca-Cola Europacific Partnersyesnononoyesyesnono
Coca-Cola HBCnonot testedyesyesnot testednot testednot testedno
Compass Groupnonot testedyesyesnot testednot testednot testedno
Computacenteryesnononoyesyesnono
Convatecyesnonoyesyesnonono
Croda Internationalyesnononoyesyesnoyes
DCC Energyyesnonononononono
Diageoyesnononoyesyesyesno
Diplomayesnononoyesyesnono
EasyJetnonot testedyesyesnot testednot testednot testedno
Endeavour Miningyesnonononononono
Experianyesnonononononono
F & C Investment Trustnonot testednononononono
Fresnillo plcyesnonononononono
Games Workshopyesnoyesyesnot testednot testednot testedno
Glencoreyesnonononononono
GSK plcyesnononoyesyesnono
Haleonnonot testedyesyesnot testednot testednot testedno
Halma plcyesnononoyesnonono
Hiscoxyesnononoyesyesnono
Howdens Joineryyesyesnononononono
HSBCyesnononoyesnonono
ICGyesnononoyesyesnono
IG Groupyesnononoyesnonoyes
IHG Hotels & Resortsyesnonononononono
IMInonot testedyesyesnot testednot testednot testedno
Imperial Brandsnonot testedyesyesnot testednot testednot testedno
Informayesnonononononono
International Airlines Groupyesnonononononono
Interteknonot testednononononono
Investecyesnoyesyesnot testednot testednot testedno
Ithaca Energyyesnonononononono
JD Sportsyesnononoyesyesnono
Kingfisher plcyesnononoyesyesnono
Land Securitiesnonot testednononononono
Legal & Generalnot testednot testednonot testednot testednot testednot testednot tested
Lion Finance Groupyesnonononononono
Lloyds Banking Groupyesnononoyesyesnono
London Stock Exchange Groupyesnononoyesnonono
LondonMetric Propertyyesnononoyesyesnono
M&Gnonot testednononononono
Marks & Spencernonot testednonoyesnonono
Melrose Industriesnonot testedyesyesnot testednot testednot testedno
Metlen Energy & Metalsyesnonononononono
National Grid plcyesnoyesyesnot testednot testednot testedno
NatWest Groupyesnononoyesyesnono
Next plcnonot testedyesnoyesyesnono
Pearson plcyesnononoyesyesnono
Pershing Square Holdingsnonot testedyesyesnot testednot testednot testedno
Polar Capital Technology Trustyesnononoyesyesnono
Prudential plcyesnonononononono
Reckittyesnononoyesyesnono
RELXyesnononoyesyesnono
Rentokil Initialnonot testedyesyesnot testednot testednot testedno
Rio Tintoyesnonononononono
Rolls-Royce Holdingsyesnonononononono
Sage Groupyesnoyesyesnot testednot testednot testedno
Sainsbury'snonot testedyesyesnot testednot testednot testedno
Scottish Mortgage Investment Trustnonot testednonoyesyesnono
Segroyesnonononononono
Severn Trentyesnonononononono
Shell plcyesnonononononono
Smith & Nephewnonot testednononononono
Smiths Groupnonot testedyesnonononono
Spirax Groupyesnonononononono
SSE plcyesnonononononono
St. James's Placeyesnononoyesyesnono
Standard Charteredyesnonononononono
Standard Lifenonot testednonoyesyesnono
Tescoyesnonononononono
Tritax Big Box REITnonot testedyesyesnot testednot testednot testedno
Unilevernonot testedyesyesnot testednot testednot testedno
United Utilitiesyesnonononononono
Vodafone Groupyesnononoyesyesnono
Weiryesnononoyesyesnoyes
Whitbreadyesnononoyesyesnono
WPPyesyesnononononono

How it was measured

Limits

Download the data

Reuse. The data is free to reuse, in any medium, with credit to Sotiris Spyrou and a link to this page. Cite the version and the date, because every number will move.

Cite this page

Using a figure or a passage from this page in an article, a course, a forum answer or AI-generated text? Please credit Sotiris Spyrou and link to this page, so your readers can check the source and the date it was reviewed. Copy a ready-made citation:

Plain text

Markdown

HTML