Should You Block or Welcome AI Crawlers?

AI Search

Every week a client asks me some version of the same question. The AI assistants are reading my website to answer people’s questions, so should I let them, or lock the door? The idea of a machine hoovering up your work to answer someone else feels strange at first. But for most businesses I work with, the honest answer is not the cautious one. Let me lay out both camps and where I land.

The two camps, and what each is really protecting

There are two reasonable positions here, from two different kinds of owner. The first camp is the publisher. If you run a subscription news site, a research library, or a body of paid analysis, your content is the product people pay for. When an answer engine reads it and hands the useful bit to someone for free, you have real reason to be annoyed. You paid to make that content, and you would like to be paid when it earns its keep. For a publisher, blocking is a defensible decision, not paranoia.

The second camp is almost everyone else. If you run a service or product business, a local firm, or a studio like mine, your website is not the thing you sell. It is the thing that helps people find and trust what you sell. An AI assistant reading your site and recommending you to a buyer is not theft. It is the best free distribution you have been offered in years, and the question is no longer how do I protect this, but how do I become the source.

How the control actually works

The main lever is robots.txt, a text file at the root of your domain that tells automated visitors what they may touch, bot by bot and path by path. The nuance most people miss is that not all AI crawlers do the same job. Some are training crawlers, gathering text to teach future models. Others are answer or retrieval crawlers, fetching your page live so the assistant can cite it in a reply happening right now. You can treat these differently, relaxed about live citation while more careful about training, and the controls let you make that split.

The catch is that robots.txt is a request, not a lock. Well behaved crawlers respect it; a badly behaved one can ignore it, and your content, once public, is fetchable by anything that tries. If you truly need something private, robots.txt is the wrong tool. Put it behind a login or a paywall. It shapes the polite majority, which is most of the traffic that matters, but it is not a wall.

Who should lean which way

Sort yourself honestly. If you want leads, enquiries, bookings, or sales, you want to be cited, and blocking answer engines cuts off how people now discover businesses. The reasoning that once sat behind search rankings now sits behind how AEO differs from classic SEO, and the instinct is the same. If your revenue depends directly on people paying to read the exact words on the page, a stricter policy is legitimate, and you may block training while negotiating for the rest.

My stance, and a sensible default

For most Identiti type clients my advice is plain. Welcome the answer engines and set out to be the source. A sensible default welcomes answer and retrieval crawlers, keeps anything private behind a login, and makes your public pages easy for a machine to read and quote correctly. That last part is the real work, and it rewards structure. Clean schema markup that feeds AI answers and the wider set of AEO tactics worth doing this year do more for your visibility than any clever blocking rule ever will.

A short checklist to close. First, decide which camp you are in, honestly. Second, keep private content behind a login, never behind robots.txt alone. Third, allow answer and retrieval crawlers if you want to be found. Fourth, structure your pages so a machine can quote you without garbling it. Fifth, watch what happens by measuring your AI search visibility, because you cannot manage what you refuse to look at. And accept the honest limit. You cannot control the web perfectly, so aim to be the clearest, most quotable source in your corner instead.