Original research, version 1.2
The Retrievability Census 2026
Most of the largest companies leave the door open to AI crawlers in robots.txt, but 26.6% of the large US tech companies we tested turned away an unknown automated visitor at the edge, and only 50.0% of FTSE 100 homepages describe the company to machines with structured data.
201 companies, two editions: the S&P 500's technology and B2B services companies (fetched 31 October 2026) and the FTSE 100 (2 November 2026). Public pages only, one request at a time. The method and its limits are below and in Appendix B of The Citation Economy Playbook.
Key figures
25 of 94
large US tech and B2B services companies (26.6%) refused an honest, unknown automated visitor at the door.
Link to this figure3 of 74
readable robots.txt files at the same companies (4.1%) block an AI training crawler from the whole site.
Link to this figure50.0%
of FTSE 100 homepages readable without JavaScript carry any structured data (39 of 78), against 72.9% in the US edition.
Link to this figure1 of 146
readable robots.txt files across both editions block an AI search or user agent from the whole site.
Link to this figureThe two editions side by side
| Measure | US: S&P 500 tech and B2B services | UK: FTSE 100 |
|---|---|---|
| Websites that answered our script | 95 of 100 | 99 of 100 |
| Robots.txt readable | 74 | 72 |
| Block an AI training agent from the whole site | 3 (4.1%) | 2 (2.8%) |
| Name any AI agent explicitly | 8 (10.8%) | 5 (6.9%) |
| Refused the honest research client | 25 of 94 (26.6%) | 24 of 99 (24.2%) |
| Refused the same script with a browser user agent | 31 of 95 (32.6%) | 21 of 99 (21.2%) |
| Homepages readable without JavaScript | 70 | 78 |
| Any JSON-LD on the homepage | 51 (72.9%) | 39 (50.0%) |
| Organization markup | 46 (65.7%) | 31 (39.7%) |
| Any sameAs link | 40 (57.1%) | 21 (26.9%) |
| sameAs to Wikidata | 9 (12.9%) | 1 (1.3%) |
| llms.txt at the root | 22 of 95 (23.2%) | 3 of 99 (3.0%) |
A difference is real here only where the two 95% ranges don't overlap. Structured data clears that bar: the UK's 50.0% has a range of 39.2% to 60.8%, against 72.9% (61.5% to 81.9%) in the US. So do Organization markup, sameAs links and llms.txt. Edge refusals, training blocks and thin pages fall within overlapping ranges, so the census can't say those differ. The frames differ too: the US edition is technology and business services, the UK edition is the whole index.
Every company
One fetch per site, on the date shown, from one UK connection. "Not tested" means the site didn't answer, or the page couldn't be read. Each row has its own link.
US edition: S&P 500 technology and B2B services (31 October 2026)
| Company | Robots.txt readable | Blocks AI training | Refused research client | Refused browser agent | JSON-LD | Organization | sameAs Wikidata | llms.txt |
|---|---|---|---|---|---|---|---|---|
| Accenture | yes | no | no | no | yes | yes | no | no |
| Adobe Inc. | yes | no | no | yes | yes | yes | yes | yes |
| Advanced Micro Devices | not tested | not tested | no | not tested | no | no | no | not tested |
| Akamai Technologies | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Amphenol | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Analog Devices | not tested | not tested | no | not tested | not tested | not tested | not tested | not tested |
| Apple Inc. | yes | no | no | no | yes | yes | yes | no |
| Applied Materials | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Arista Networks | yes | no | no | no | yes | yes | no | no |
| Autodesk | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Automatic Data Processing | yes | no | no | yes | no | no | no | yes |
| Block, Inc. | yes | no | no | no | yes | no | no | no |
| Broadcom | yes | no | no | no | no | no | no | no |
| Broadridge Financial Solutions | yes | no | no | no | yes | yes | no | yes |
| Cadence Design Systems | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Cboe Global Markets | yes | no | no | no | no | no | no | no |
| CDW Corporation | yes | no | no | no | yes | no | no | no |
| Ciena | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Cintas | yes | no | no | no | yes | yes | no | no |
| Cisco | yes | no | no | yes | yes | yes | no | no |
| CME Group | not tested | not tested | no | not tested | not tested | not tested | not tested | not tested |
| Cognizant | yes | no | no | no | yes | yes | no | yes |
| Coherent Corp. | yes | no | no | no | yes | yes | no | no |
| Coinbase | yes | no | yes | yes | not tested | not tested | not tested | yes |
| Copart | yes | no | yes | yes | not tested | not tested | not tested | no |
| Corning Inc. | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Corpay | yes | no | no | no | yes | yes | no | yes |
| CrowdStrike | yes | no | no | no | yes | yes | no | yes |
| Datadog | yes | no | no | no | yes | yes | no | yes |
| Dell Technologies | yes | no | no | no | yes | yes | no | no |
| Equifax | yes | no | no | no | yes | yes | no | no |
| Everpure | yes | no | no | no | yes | yes | yes | no |
| F5, Inc. | yes | no | no | no | yes | yes | no | no |
| FactSet | yes | no | no | no | yes | no | no | no |
| Fair Isaac | yes | no | yes | yes | not tested | not tested | not tested | no |
| Fidelity National Information Services | yes | no | no | no | yes | yes | no | yes |
| First Solar | yes | no | no | no | no | no | no | no |
| Fiserv | yes | no | no | no | yes | yes | no | no |
| Flex Ltd. | yes | no | no | no | yes | yes | no | no |
| Fortinet | yes | yes | not tested | no | yes | yes | no | no |
| Gartner | yes | yes | yes | yes | not tested | not tested | not tested | no |
| Gen Digital | yes | no | no | no | yes | yes | no | no |
| Global Payments | no | not tested | yes | yes | not tested | not tested | not tested | no |
| GoDaddy | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Hewlett Packard Enterprise | not tested | not tested | no | not tested | not tested | not tested | not tested | not tested |
| HP Inc. | yes | no | no | no | yes | no | no | no |
| IBM | yes | no | no | no | yes | yes | yes | no |
| Intel | yes | no | no | yes | no | no | no | yes |
| Intercontinental Exchange | no | not tested | no | no | no | no | no | no |
| Intuit | yes | no | yes | no | yes | yes | yes | yes |
| Jabil | yes | no | no | no | yes | yes | no | yes |
| Jack Henry & Associates | yes | no | no | no | yes | yes | no | no |
| Keysight Technologies | yes | no | yes | yes | not tested | not tested | not tested | no |
| KLA Corporation | yes | no | no | no | yes | yes | no | no |
| Lam Research | no | not tested | no | no | yes | yes | no | no |
| Leidos | yes | no | no | no | yes | yes | no | yes |
| Lumentum | not tested | not tested | not tested | not tested | not tested | not tested | not tested | not tested |
| Marvell Technology | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Mastercard | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Microchip Technology | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Micron Technology | yes | no | no | no | yes | yes | no | no |
| Microsoft | yes | no | no | yes | no | no | no | no |
| Monolithic Power Systems | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Moody's Corporation | yes | no | no | no | yes | yes | no | no |
| Motorola Solutions | yes | no | no | no | yes | yes | no | no |
| MSCI | yes | no | no | no | no | no | no | no |
| Nasdaq, Inc. | not tested | not tested | no | not tested | yes | yes | no | not tested |
| NetApp | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Nvidia | yes | no | no | no | yes | yes | no | no |
| NXP Semiconductors | yes | no | no | no | yes | no | no | no |
| ON Semiconductor | yes | no | no | yes | yes | yes | no | no |
| Oracle Corporation | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Palantir Technologies | yes | no | no | no | no | no | no | no |
| Palo Alto Networks | yes | no | no | no | no | no | no | yes |
| Paychex | yes | no | no | no | yes | yes | no | yes |
| PayPal | yes | no | no | no | yes | yes | no | no |
| PTC Inc. | no | not tested | yes | no | yes | yes | no | no |
| Qnity Electronics | yes | no | no | no | no | no | no | yes |
| Qualcomm | yes | no | no | no | yes | yes | no | yes |
| Roper Technologies | no | not tested | no | no | not tested | not tested | not tested | no |
| S&P Global | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Salesforce | yes | no | no | no | yes | yes | no | no |
| Sandisk | yes | no | no | no | yes | yes | no | yes |
| Seagate Technology | yes | no | no | no | not tested | not tested | not tested | no |
| ServiceNow | no | not tested | yes | no | not tested | not tested | not tested | no |
| Skyworks Solutions | yes | no | no | no | no | no | no | no |
| Supermicro | yes | no | no | yes | no | no | no | no |
| Synopsys | yes | no | no | no | no | no | no | yes |
| TE Connectivity | yes | yes | no | yes | yes | yes | no | no |
| Teledyne Technologies | yes | no | no | no | no | no | no | no |
| Teradyne | yes | no | no | yes | yes | yes | no | no |
| Texas Instruments | yes | no | no | no | no | no | no | no |
| Trimble Inc. | yes | no | no | no | yes | yes | yes | yes |
| Twilio | yes | no | no | no | yes | yes | yes | no |
| Tyler Technologies | yes | no | no | no | no | no | no | no |
| Verisign | yes | no | no | no | no | no | no | no |
| Verisk Analytics | yes | no | no | no | no | no | no | no |
| Visa Inc. | yes | no | yes | yes | not tested | not tested | not tested | no |
| Western Digital | yes | no | no | no | yes | yes | yes | yes |
| Workday, Inc. | yes | no | no | no | yes | yes | no | no |
| Zebra Technologies | yes | no | no | no | yes | yes | yes | yes |
UK edition: FTSE 100 (2 November 2026)
| Company | Robots.txt readable | Blocks AI training | Refused research client | Refused browser agent | JSON-LD | Organization | sameAs Wikidata | llms.txt |
|---|---|---|---|---|---|---|---|---|
| 3i | yes | no | yes | yes | not tested | not tested | not tested | no |
| Aberdeen Group | yes | no | no | no | yes | yes | no | no |
| Admiral Group | no | not tested | yes | no | not tested | not tested | not tested | no |
| Airtel Africa | no | not tested | no | no | no | no | no | no |
| Alliance Witan | yes | no | no | no | yes | yes | no | no |
| Anglo American plc | yes | no | no | no | no | no | no | no |
| Antofagasta plc | yes | no | no | no | no | no | no | no |
| Associated British Foods | no | not tested | yes | yes | not tested | not tested | not tested | no |
| AstraZeneca | yes | no | no | no | yes | no | no | no |
| Autotrader Group | yes | no | no | no | yes | yes | no | no |
| Aviva | no | not tested | yes | no | yes | yes | no | no |
| Babcock International | yes | no | no | no | no | no | no | no |
| BAE Systems | yes | no | yes | yes | not tested | not tested | not tested | no |
| Balfour Beatty | yes | no | no | no | no | no | no | no |
| Barclays | yes | no | no | no | yes | no | no | no |
| Barratt Redrow | no | not tested | no | no | no | no | no | no |
| BP | yes | no | no | no | no | no | no | no |
| British American Tobacco | yes | no | no | no | yes | yes | no | no |
| British Land | yes | no | no | no | yes | yes | no | no |
| BT Group | yes | no | no | no | yes | yes | no | no |
| Bunzl | yes | no | no | no | no | no | no | no |
| Burberry Group | yes | no | no | no | no | no | no | no |
| Centrica | yes | no | yes | yes | not tested | not tested | not tested | no |
| Coca-Cola Europacific Partners | yes | no | no | no | yes | yes | no | no |
| Coca-Cola HBC | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Compass Group | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Computacenter | yes | no | no | no | yes | yes | no | no |
| Convatec | yes | no | no | yes | yes | no | no | no |
| Croda International | yes | no | no | no | yes | yes | no | yes |
| DCC Energy | yes | no | no | no | no | no | no | no |
| Diageo | yes | no | no | no | yes | yes | yes | no |
| Diploma | yes | no | no | no | yes | yes | no | no |
| EasyJet | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Endeavour Mining | yes | no | no | no | no | no | no | no |
| Experian | yes | no | no | no | no | no | no | no |
| F & C Investment Trust | no | not tested | no | no | no | no | no | no |
| Fresnillo plc | yes | no | no | no | no | no | no | no |
| Games Workshop | yes | no | yes | yes | not tested | not tested | not tested | no |
| Glencore | yes | no | no | no | no | no | no | no |
| GSK plc | yes | no | no | no | yes | yes | no | no |
| Haleon | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Halma plc | yes | no | no | no | yes | no | no | no |
| Hiscox | yes | no | no | no | yes | yes | no | no |
| Howdens Joinery | yes | yes | no | no | no | no | no | no |
| HSBC | yes | no | no | no | yes | no | no | no |
| ICG | yes | no | no | no | yes | yes | no | no |
| IG Group | yes | no | no | no | yes | no | no | yes |
| IHG Hotels & Resorts | yes | no | no | no | no | no | no | no |
| IMI | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Imperial Brands | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Informa | yes | no | no | no | no | no | no | no |
| International Airlines Group | yes | no | no | no | no | no | no | no |
| Intertek | no | not tested | no | no | no | no | no | no |
| Investec | yes | no | yes | yes | not tested | not tested | not tested | no |
| Ithaca Energy | yes | no | no | no | no | no | no | no |
| JD Sports | yes | no | no | no | yes | yes | no | no |
| Kingfisher plc | yes | no | no | no | yes | yes | no | no |
| Land Securities | no | not tested | no | no | no | no | no | no |
| Legal & General | not tested | not tested | no | not tested | not tested | not tested | not tested | not tested |
| Lion Finance Group | yes | no | no | no | no | no | no | no |
| Lloyds Banking Group | yes | no | no | no | yes | yes | no | no |
| London Stock Exchange Group | yes | no | no | no | yes | no | no | no |
| LondonMetric Property | yes | no | no | no | yes | yes | no | no |
| M&G | no | not tested | no | no | no | no | no | no |
| Marks & Spencer | no | not tested | no | no | yes | no | no | no |
| Melrose Industries | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Metlen Energy & Metals | yes | no | no | no | no | no | no | no |
| National Grid plc | yes | no | yes | yes | not tested | not tested | not tested | no |
| NatWest Group | yes | no | no | no | yes | yes | no | no |
| Next plc | no | not tested | yes | no | yes | yes | no | no |
| Pearson plc | yes | no | no | no | yes | yes | no | no |
| Pershing Square Holdings | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Polar Capital Technology Trust | yes | no | no | no | yes | yes | no | no |
| Prudential plc | yes | no | no | no | no | no | no | no |
| Reckitt | yes | no | no | no | yes | yes | no | no |
| RELX | yes | no | no | no | yes | yes | no | no |
| Rentokil Initial | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Rio Tinto | yes | no | no | no | no | no | no | no |
| Rolls-Royce Holdings | yes | no | no | no | no | no | no | no |
| Sage Group | yes | no | yes | yes | not tested | not tested | not tested | no |
| Sainsbury's | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Scottish Mortgage Investment Trust | no | not tested | no | no | yes | yes | no | no |
| Segro | yes | no | no | no | no | no | no | no |
| Severn Trent | yes | no | no | no | no | no | no | no |
| Shell plc | yes | no | no | no | no | no | no | no |
| Smith & Nephew | no | not tested | no | no | no | no | no | no |
| Smiths Group | no | not tested | yes | no | no | no | no | no |
| Spirax Group | yes | no | no | no | no | no | no | no |
| SSE plc | yes | no | no | no | no | no | no | no |
| St. James's Place | yes | no | no | no | yes | yes | no | no |
| Standard Chartered | yes | no | no | no | no | no | no | no |
| Standard Life | no | not tested | no | no | yes | yes | no | no |
| Tesco | yes | no | no | no | no | no | no | no |
| Tritax Big Box REIT | no | not tested | yes | yes | not tested | not tested | not tested | no |
| Unilever | no | not tested | yes | yes | not tested | not tested | not tested | no |
| United Utilities | yes | no | no | no | no | no | no | no |
| Vodafone Group | yes | no | no | no | yes | yes | no | no |
| Weir | yes | no | no | no | yes | yes | no | yes |
| Whitbread | yes | no | no | no | yes | yes | no | no |
| WPP | yes | yes | no | no | no | no | no | no |
How it was measured
- Each company's official website came from its Wikidata item.
- Robots.txt was parsed under RFC 9309 for 15 AI agents plus Googlebot and Bingbot as controls.
- The homepage was requested twice: once by an honest research client that named the study, and once by the same script with a browser user agent. A refusal is a 401, 403, 429 or 503, or a page marked as a bot challenge.
- Visible words, JSON-LD, Organization markup and sameAs links were read from the raw HTML without running JavaScript.
- llms.txt counted only if it returned a non-HTML file beginning with a markdown heading.
- Version 1.2 includes three detector fixes found by an independent check of the UK data; re-running the US data with them changed no US number.
Limits
- Our client wasn't a verified crawler: a firewall may admit the real GPTBot and still refuse a script like ours.
- One fetch per site, on one day, from one UK connection; homepages only.
- Sites that didn't answer are left out of every base except the Wikidata ones, so refusal rates are a floor.
- Large listed companies only. Nothing here describes small or private firms.
Download the data
Reuse. The data is free to reuse, in any medium, with credit to Sotiris Spyrou and a link to this page. Cite the version and the date, because every number will move.
Cite this page
Using a figure or a passage from this page in an article, a course, a forum answer or AI-generated text? Please credit Sotiris Spyrou and link to this page, so your readers can check the source and the date it was reviewed. Copy a ready-made citation:
Plain text
Markdown
HTML