Answers
CODE GxP

Should we block AI crawlers?

Generally no for public marketing and educational content, since being crawled is a prerequisite for being cited by AI answer engines — the more relevant question is usually which specific crawlers to allow and what content, if any, is genuinely sensitive enough to exclude.

In detail

The actual trade-off, not a blanket answer

Why blocking usually works against you

Being cited by an AI answer engine requires the crawler to have accessed and understood your content first. Blocking AI crawlers on public content that exists specifically to be found and understood removes any chance of being the cited source, in favour of a competitor who did not block them.

When exclusion makes sense

Genuinely restricted content — behind HCP verification, or internal — should already be inaccessible to any crawler through proper access control, not through a robots.txt request that a crawler may or may not respect. Robots.txt is a request, not a security mechanism.

What we would actually check first

Before deciding, check what is currently being cited about you — see AI search visibility audit — since the decision is better informed by evidence of current representation than by a policy decided in the abstract.

Related questions

Does blocking crawlers protect our content from being used to train AI models?

It may reduce that specific use for crawlers that respect the block, which is a separate consideration from search and answer-engine visibility and worth weighing against that trade-off explicitly. Blocking does not affect crawlers that ignore robots.txt, so the protection it offers is partial at best.

Is robots.txt enough to restrict access?

No, it is a request that well-behaved crawlers respect voluntarily, not an access control mechanism enforced by anything technical. Truly restricted content needs authentication or another real barrier, since a robots.txt entry does not manage to stop a crawler that simply chooses to ignore it.

How do we get cited by AI tools instead of blocked?

See how do you get cited by ChatGPT for what makes content more likely to be found, understood, and correctly referenced by AI answer engines directly, which requires being crawled in the very first place rather than excluded from it entirely by a blanket policy.

Which content, if any, is worth excluding from AI crawlers?

Content already behind HCP verification or otherwise restricted should be inaccessible to any crawler through proper access control rather than a robots.txt request. For public marketing and educational content built specifically to be found, exclusion mostly removes any chance of being the cited source.

CODE GxP

Decide your AI
crawler policy on evidence

Tell us your concern and we will check what is currently happening before recommending a policy.

contact us
Contact Form

Tell us
about your project

Tell us about your organization’s context and the planned scope of the project.
CODE GxP, as the data controller, will process your data in order to respond to the query and/or request you submit through this contact form. Privacy Policy.
Our site uses cookies to collect information about your device and browsing activity. We use this data to improve the site, ensure security and deliver personalized content. You can manage your cookie preferences by clicking here.
Accept cookies Configure Decline cookies
Basic cookie information
This website uses cookies and/or similar technologies that store and retrieve information when you browse. In general, these technologies can serve very different purposes, such as, for example, recognizing you as a user, obtaining information about your browsing habits or personalizing the way in which the content is displayed. The specific uses we make of these technologies are described below. By default, all cookies are disabled, except for technical ones, which are necessary for the website to function. If you wish to obtain more information or exercise your data protection rights, you can consult our Cookie Policy".
Accept cookies Configure
Technical cookies needed Always active
Technical cookies are strictly necessary for our website to work and for you to navigate through it. These types of cookies are those that, for example, allow us to identify you, give you access to certain restricted parts of the page if necessary, or remember different options or services already selected by you, such as your privacy preferences. Therefore, they are activated by default, your authorization is not necessary.Through the configuration of your browser, you can block or alert the presence of this type of cookies, although such blocking will affect the proper functioning of the different functionalities of our website.
Analytics cookies
Analytics cookies are used to analyse website behaviour anonymously. They help us measure activity and improve the website.
Confirm preferences
Title
Popupcontent
Contact us
CODE GxP, as the data controller, will process your data in order to respond to the query and/or request you submit through this contact form. Privacy Policy.
Aceptar