How walled is the web? Checked daily.
The rules say come in. A third of front doors say no.
We check 170 major sites every day: what their robots.txt tells 24 AI agents, and what happens when an honestly labeled bot knocks. Every status comes with receipts, back to Jan 2023.
13of 159
robots.txt files that block all 12 major AI agents
48of 150
homepages that refused or challenged our honestly labeled bot
47of 159
files that block at least one training crawler, such as GPTBot
39of 159
files that name a user-directed agent, such as ChatGPT-User
Out of the 159 sites whose robots.txt we could read (144 live today, 15 from an Internet Archive copy at most three months old), and the 150 homepages we could ask. For the other 11 sites we have no readable robots.txt.
Checked 19 Oct 2026, 13:46 UTC170 sites24 AI agentsHistory from Jan 2023
By category
Who blocks training crawlers
Sites in each category whose robots.txt blocks at least one training crawler today. Show the full wall
01 The wall
Every site, every AI agent, month by month
Each column is a site and each row is an AI agent. A dark cell means robots.txt keeps that agent out of the whole site. Press play to watch the wall go up since Jan 2023, rebuilt from the Internet Archive's copies of each file.
Archive copies: first capture each month, carried forward up to three months. Current month: today's Checker run. How this works
02 The Index
All 170 sites
Status summarizes the evidence on each site's page. Training, Search and Agents show how robots.txt treats each kind of AI bot: blocked partial allowed. Door is what happened when our honestly labeled Checker asked for the homepage. Click a column to sort; filters are kept in the link.
170 sites
| Training · Search · Agents | |||||
|---|---|---|---|---|---|
| Amazonnamedamazon.com | Retail | Partial | Training: PartialSearch: PartialAgents: Partial | Open | Jun 2026 |
| Walmartwalmart.com | Retail | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Targettarget.com | Retail | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Best Buybestbuy.com | Retail | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | May to Oct 2026 |
| eBayebay.com | Retail | Partial | Training: PartialSearch: PartialAgents: Allowed | Refused | May to Oct 2026 |
| Etsyetsy.com | Retail | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Challenged | · |
| Costcocostco.com | Retail | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| The Home Depothomedepot.com | Retail | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Lowe'slowes.com | Retail | Unknown | Training: UnknownSearch: UnknownAgents: Unknown | Refused | · |
| Wayfairwayfair.com | Retail | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Macy'smacys.com | Retail | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Refused | May 2026 |
| Nordstromnordstrom.com | Retail | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | May to Oct 2026 |
| IKEAikea.com | Retail | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | Sep 2025 |
| Sheinshein.com | Retail | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Temutemu.com | Retail | Allowed | Training: PartialSearch: AllowedAgents: Allowed | Open | Sep 2025 |
| AliExpressaliexpress.com | Retail | Partial | Training: PartialSearch: PartialAgents: Allowed | No answer | Oct 2026 |
| Chewychewy.com | Retail | Partial | Training: AllowedSearch: PartialAgents: Allowed | Refused | Sep 2025 |
| Nikenike.com | Retail | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Instacartinstacart.com | Retail | Partial | Training: BlockedSearch: PartialAgents: Partial | Not testedrobots.txt bars our bot | May 2026 |
| Krogerkroger.com | Retail | Unknown | Training: UnknownSearch: UnknownAgents: Unknown | Not testedrobots.txt did not answer | · |
| Sephorasephora.com | Retail | Unknown | Training: UnknownSearch: UnknownAgents: Unknown | Refused | · |
| Zapposzappos.com | Retail | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Allbirdsallbirds.com | Retail | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Gymsharkgymshark.com | Retail | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Brooklinenbrooklinen.com | Retail | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Rothy'srothys.com | Retail | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Death Wish Coffeedeathwishcoffee.com | Retail | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| ColourPopcolourpop.com | Retail | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Bombasbombas.com | Retail | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Refused | · |
| Fashion Novafashionnova.com | Retail | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Delta Air Linesnameddelta.com | Travel | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| United Airlinesunited.com | Travel | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Not testedrobots.txt did not answer | · |
| American Airlinesaa.com | Travel | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Refused | Aug 2025 |
| Southwest Airlinessouthwest.com | Travel | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Alaska Airlinesalaskaair.com | Travel | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| JetBluejetblue.com | Travel | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| British Airwaysbritishairways.com | Travel | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Not testedrobots.txt did not answer | · |
| Lufthansalufthansa.com | Travel | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Refused | · |
| Air Canadaaircanada.com | Travel | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Marriottmarriott.com | Travel | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Refused | Feb 2025 |
| Hiltonhilton.com | Travel | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Refused | · |
| Hyatthyatt.com | Travel | Unknown | Training: UnknownSearch: UnknownAgents: Unknown | Refused | · |
| IHG Hotelsihg.com | Travel | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Refused | Nov 2025 |
| Airbnbairbnb.com | Travel | Partial | Training: PartialSearch: AllowedAgents: Allowed | Open | Oct 2026 |
| Booking.combooking.com | Travel | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Expediaexpedia.com | Travel | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Refused | Sep 2025 |
| KAYAKkayak.com | Travel | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | Apr 2023 |
| Tripadvisortripadvisor.com | Travel | Partial | Training: PartialSearch: PartialAgents: Allowed | Challenged | Feb 2026 |
| Vrbovrbo.com | Travel | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Refused | Sep 2025 |
| Hotels.comhotels.com | Travel | Unknown | Training: UnknownSearch: UnknownAgents: Unknown | Not testedrobots.txt did not answer | Mar 2026 |
| Amtrakamtrak.com | Travel | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| JPMorgan Chasenamedchase.com | Finance | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Charles Schwabnamedschwab.com | Finance | Blocked | Training: UnknownSearch: UnknownAgents: Unknown | Refused | · |
| Bank of Americabankofamerica.com | Finance | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Wells Fargowellsfargo.com | Finance | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Citiciti.com | Finance | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Capital Onecapitalone.com | Finance | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | May to Oct 2026 |
| American Expressamericanexpress.com | Finance | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| U.S. Bankusbank.com | Finance | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Discoverdiscover.com | Finance | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Fidelityfidelity.com | Finance | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Vanguardvanguard.com | Finance | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| E*TRADEetrade.com | Finance | Unknown | Training: UnknownSearch: UnknownAgents: Unknown | Not testedrobots.txt did not answer | · |
| Robinhoodrobinhood.com | Finance | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| SoFisofi.com | Finance | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Challenged | · |
| PayPalpaypal.com | Finance | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Venmovenmo.com | Finance | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | Jul 2024 |
| Coinbasecoinbase.com | Finance | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Challenged | · |
| Yelpnamedyelp.com | Local and reviews | Paid | Training: BlockedSearch: BlockedAgents: Blocked | Not testedrobots.txt bars our bot | · |
| OpenTableopentable.com | Local and reviews | Unknown | Training: UnknownSearch: UnknownAgents: Unknown | Not testedrobots.txt did not answer | Aug 2025 |
| DoorDashdoordash.com | Local and reviews | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Challenged | · |
| Uber Eatsubereats.com | Local and reviews | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Challenged | · |
| Grubhubgrubhub.com | Local and reviews | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Angiangi.com | Local and reviews | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Challenged | Aug 2024 |
| Thumbtackthumbtack.com | Local and reviews | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | Apr 2024 |
| Nextdoornextdoor.com | Local and reviews | Blocked | Training: BlockedSearch: BlockedAgents: Blocked | Not testedrobots.txt bars our bot | Sep 2023 |
| Zillowzillow.com | Local and reviews | Partial | Training: PartialSearch: PartialAgents: Allowed | Open | Apr 2026 |
| Realtor.comrealtor.com | Local and reviews | Partial | Training: PartialSearch: AllowedAgents: Partial | Refused | Nov 2024 |
| Redfinredfin.com | Local and reviews | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Craigslistcraigslist.org | Local and reviews | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | Dec 2025 to Jul 2026 |
| Ticketmasterticketmaster.com | Local and reviews | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | Jul 2025 |
| Eventbriteeventbrite.com | Local and reviews | Partial | Training: PartialSearch: AllowedAgents: Allowed | Open | · |
| Indeedindeed.com | Local and reviews | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Challenged | Apr to Sep 2026 |
| Glassdoorglassdoor.com | Local and reviews | Partial | Training: PartialSearch: PartialAgents: Allowed | Challenged | May to Oct 2026 |
| The New York Timesnytimes.com | News | Blocked | Training: PartialSearch: PartialAgents: Partial | Challenged | Apr 2026 |
| The Washington Postwashingtonpost.com | News | Partial | Training: PartialSearch: PartialAgents: Partial | No answer | Aug 2025 |
| The Wall Street Journalwsj.com | News | Partial | Training: PartialSearch: PartialAgents: Partial | Not testedrobots.txt bars our bot | Jun 2026 |
| The Guardiantheguardian.com | News | Partial | Training: PartialSearch: PartialAgents: Partial | Open | · |
| BBCbbc.com | News | Partial | Training: PartialSearch: PartialAgents: Partial | Open | Jan 2026 |
| CNNcnn.com | News | Partial | Training: PartialSearch: PartialAgents: Partial | Open | Oct 2026 |
| Reutersreuters.com | News | Partial | Training: BlockedSearch: PartialAgents: Partial | Not testedrobots.txt bars our bot | Sep 2026 |
| Associated Pressapnews.com | News | Partial | Training: PartialSearch: PartialAgents: Partial | Challenged | Aug 2025 |
| Bloombergbloomberg.com | News | Partial | Training: PartialSearch: PartialAgents: Partial | Challenged | Oct 2026 |
| Financial Timesft.com | News | Partial | Training: PartialSearch: PartialAgents: Partial | Challenged | May 2025 |
| The Atlantictheatlantic.com | News | Partial | Training: PartialSearch: PartialAgents: Partial | Refused | Apr 2026 |
| The New Yorkernewyorker.com | News | Partial | Training: PartialSearch: PartialAgents: Partial | Open | Oct 2025 |
| NPRnpr.org | News | Partial | Training: PartialSearch: PartialAgents: Partial | Not testedrobots.txt did not answer | Oct 2026 |
| USA Todayusatoday.com | News | Blocked | Training: PartialSearch: PartialAgents: Blocked | Open | Apr to Jun 2026 |
| Fox Newsfoxnews.com | News | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Axiosaxios.com | News | Partial | Training: PartialSearch: PartialAgents: Allowed | Challenged | May to Jul 2026 |
| Politicopolitico.com | News | Blocked | Training: PartialSearch: PartialAgents: Partial | Challenged | Sep 2025 |
| The Vergetheverge.com | News | Partial | Training: PartialSearch: PartialAgents: Partial | Open | Jun 2026 |
| WIREDwired.com | News | Partial | Training: PartialSearch: PartialAgents: Partial | Open | Oct 2025 |
| The Economisteconomist.com | News | Partial | Training: PartialSearch: PartialAgents: Partial | Challenged | Aug 2026 |
| Los Angeles Timeslatimes.com | News | Partial | Training: PartialSearch: AllowedAgents: Allowed | Open | Aug 2025 |
| Mediummedium.com | News | Partial | Training: PartialSearch: PartialAgents: Allowed | Challenged | Sep 2025 |
| Substacksubstack.com | News | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Redditreddit.com | Social | Blocked | Training: BlockedSearch: BlockedAgents: Blocked | Not testedrobots.txt bars our bot | Feb 2026 |
| Xx.com | Social | Blocked | Training: BlockedSearch: BlockedAgents: Blocked | Not testedrobots.txt bars our bot | Sep 2026 |
| Facebookfacebook.com | Social | Blocked | Training: BlockedSearch: BlockedAgents: Blocked | Not testedrobots.txt bars our bot | Oct 2026 |
| Instagraminstagram.com | Social | Blocked | Training: BlockedSearch: BlockedAgents: Blocked | Not testedrobots.txt bars our bot | Jan 2025 to Apr 2026 |
| LinkedInlinkedin.com | Social | Partial | Training: BlockedSearch: PartialAgents: Blocked | Not testedrobots.txt bars our bot | Mar to Jul 2026 |
| TikToktiktok.com | Social | Blocked | Training: PartialSearch: PartialAgents: Partial | Open | Oct 2025 |
| YouTubeyoutube.com | Social | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Pinterestpinterest.com | Social | Blocked | Training: BlockedSearch: PartialAgents: Blocked | Not testedrobots.txt bars our bot | Oct 2026 |
| Quoraquora.com | Social | Partial | Training: PartialSearch: PartialAgents: Partial | Challenged | · |
| Tumblrtumblr.com | Social | Partial | Training: PartialSearch: PartialAgents: Allowed | Open | Sep 2024 |
| Discorddiscord.com | Social | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Threadsthreads.com | Social | Blocked | Training: BlockedSearch: BlockedAgents: Blocked | Not testedrobots.txt bars our bot | Sep 2024 to May 2025 |
| Blueskybsky.app | Social | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Stack Overflowstackoverflow.com | Social | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Challenged | Feb 2026 |
| Twitchtwitch.tv | Social | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | May to Jul 2026 |
| GitHubgithub.com | Developer | Allowed | Training: PartialSearch: AllowedAgents: Allowed | Open | Oct 2026 |
| GitLabgitlab.com | Developer | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| MDN Web Docsdeveloper.mozilla.org | Developer | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| npmnpmjs.com | Developer | Unknown | Training: UnknownSearch: UnknownAgents: Unknown | Challenged | · |
| PyPIpypi.org | Developer | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Vercelvercel.com | Developer | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | May 2026 |
| Stripe Docsdocs.stripe.com | Developer | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | May 2026 |
| Cloudflare Docsdevelopers.cloudflare.com | Developer | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | Oct 2025 |
| Supabasesupabase.com | Developer | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | May 2026 |
| Hugging Facehuggingface.co | Developer | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Next.jsnextjs.org | Developer | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Sveltesvelte.dev | Developer | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Mintlifymintlify.com | Developer | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Dockerdocker.com | Developer | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Anthropicanthropic.com | Developer | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| OpenAIopenai.com | Developer | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Challenged | · |
| Wikipediawikipedia.org | Reference | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Britannicabritannica.com | Reference | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Challenged | · |
| Internet Archivearchive.org | Reference | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| IMDbimdb.com | Reference | Blocked | Training: BlockedSearch: PartialAgents: Partial | Not testedrobots.txt bars our bot | May to Sep 2026 |
| Merriam-Webstermerriam-webster.com | Reference | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Challenged | · |
| Fandomfandom.com | Reference | Unknown | Training: UnknownSearch: UnknownAgents: Unknown | Challenged | · |
| USA.govusa.gov | Government | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| IRSirs.gov | Government | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Social Security Administrationssa.gov | Government | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Refused | · |
| CDCcdc.gov | Government | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| NIHnih.gov | Government | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Challenged | · |
| Medicare.govmedicare.gov | Government | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| HealthCare.govhealthcare.gov | Government | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| GOV.UKgov.uk | Government | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | May 2026 |
| Canada.cacanada.ca | Government | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| FTCftc.gov | Government | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Mayo Clinicmayoclinic.org | Healthcare | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Refused | · |
| Cleveland Clinicclevelandclinic.org | Healthcare | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Johns Hopkins Medicinehopkinsmedicine.org | Healthcare | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Challenged | · |
| Kaiser Permanentekp.org | Healthcare | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Mount Sinaimountsinai.org | Healthcare | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | Oct 2026 |
| UnitedHealthcareuhc.com | Healthcare | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Refused | · |
| Aetnaaetna.com | Healthcare | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Cignacigna.com | Healthcare | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Anthemanthem.com | Healthcare | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| Humanahumana.com | Healthcare | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | Oct 2026 |
| CVS Healthcvs.com | Healthcare | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | May 2024 |
| Walgreenswalgreens.com | Healthcare | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
| WebMDwebmd.com | Healthcare | Partial | Training: PartialSearch: AllowedAgents: Allowed | Open | Sep 2026 |
| GoodRxgoodrx.com | Healthcare | Partial | Training: AllowedSearch: AllowedAgents: Allowed | Challenged | Dec 2024 |
| Zocdoczocdoc.com | Healthcare | Unknown | Training: UnknownSearch: UnknownAgents: Unknown | Challenged | Feb 2026 |
| MyChart (Epic)mychart.org | Healthcare | Allowed | Training: AllowedSearch: AllowedAgents: Allowed | Open | · |
03 Recent changes
What moved lately
Oct 2026 · archiveAirbnb More blockingNow blocked: GPTBot, ClaudeBot, Applebot-Extended, MistralAI-Training and AI2Bot.Oct 2026 · archive · provisionalAliExpress More blockingNow blocked: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider and Amazonbot.Oct 2026 · archiveBloomberg Less blockingNow partly blocked: OAI-SearchBot, Claude-SearchBot, ChatGPT-User and Claude-User.Oct 2026 · archive · provisionalCNN Less blockingNow allowed: Google-Extended.Oct 2026 · archive · provisionalFacebook More blockingNow blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot and Amazonbot.May to Oct 2026 · archive · provisionalGlassdoor More blockingNow blocked: ClaudeBot.
04 How to read this
robots.txt is only half the story
Training9 tracked
Collects pages to train AI models. Blocking it asks that content be kept out of future model training; it does not stop live AI answers.
Today 12 of 159 readable files block all of them and 47 block at least one. 53 name one or more by name.
Search and answers8 tracked
Indexes or fetches pages so an AI product can cite them in answers. Blocking it usually removes the site from that product's citations.
Today 7 of 159 readable files block all of them and 41 block at least one. 51 name one or more by name.
User-directed agents7 tracked
Fetches a page because a person asked an AI to, in real time. This is the lane the "war on agents" is about. Several operators say these fetchers may not follow robots.txt.
Today 10 of 159 readable files block all of them and 33 block at least one. 39 name one or more by name.
5 of the 7 user-directed agents we track come with an operator statement that they may not follow robots.txt, because a person asked for the page (OpenAI, Perplexity, Google, Meta, Amazon). Anthropic and Mistral AI say or imply that theirs do. So for the agents at the center of the "war on agents", a robots.txt line is often a request rather than a lock. The tools that can stop them are bot-management software, login walls and terms of service. That is why the Index also records what happens when an honest bot knocks, and collects reports from people whose agents hit a wall.
05 Let one in, keep the rest out
10 sites let in one or two companies by name
These sites name one or two AI companies' crawlers in robots.txt and allow them, while blocking agents from at least three other companies. Only agents a file names count, so a company simply missing from the file does not put a site on this list.
| Site | Lets in | Keeps out | Opened |
|---|---|---|---|
| Axiosaxios.com | Google Google-Extended | 3 agents from 3 other companies | Mar 2026 |
| The New York Timesnytimes.com | Amazon Amazonbot | 16 agents from 9 other companies | Oct 2025 |
| The Vergetheverge.com | Amazon Amazonbot | 15 agents from 11 other companies | May 2025 |
| The Atlantictheatlantic.com | OpenAI GPTBot, OAI-SearchBot, ChatGPT-User DuckDuckGo DuckAssistBot |
13 agents from 9 other companies | Jul 2024 |
| The Wall Street Journalwsj.com | OpenAI GPTBot, OAI-SearchBot, ChatGPT-User | 21 agents from 11 other companies | Jun 2024 |
| Airbnbairbnb.com | Meta Meta-ExternalAgent Perplexity PerplexityBot |
5 agents from 5 other companies | before 2023, or no record |
| Tripadvisortripadvisor.com | Perplexity PerplexityBot | 8 agents from 8 other companies | before 2023, or no record |
| Pinterestpinterest.com | DuckDuckGo DuckAssistBot | 23 agents from 11 other companies | before 2023, or no record |
| Quoraquora.com | Google Google-Extended Amazon Amazonbot |
9 agents from 5 other companies | before 2023, or no record |
| IMDbimdb.com | Amazon Amazonbot, Amzn-SearchBot, Amzn-User DuckDuckGo DuckAssistBot |
20 agents from 10 other companies | before 2023, or no record |
Ran into a wall?
Tell us what your agent hit: a login wall, a challenge, a refusal, or "worked fine".
Report it →Run a store?
See what your robots.txt tells each AI agent, and whether that is what you meant.
Check your site →Every Monday
One number, the week's notable changes, and one chart. Built from the data.
Get the newsletter →