On 30 October 2026, Google announced plans to support a shared AI-training preference in robots.txt. The proposal comes from work around the IETF AI Preferences group and is intended to let a website express a broad training choice across participating AI providers instead of maintaining a separate rule for every company. Google says it aims to support the shared control with other providers in 2027, while its existing Google-Extended control continues to operate.
This is a future-facing preference, not a universal switch that is already enforced everywhere. It also does not make crawling, Google Search indexing, AI training and live answer grounding the same thing. A rushed robots.txt edit can block useful discovery without achieving the intended policy outcome. This guide gives publishers, ecommerce brands, franchise networks and business websites a practical readiness process. The proposal is still evolving, so technical implementation should be checked against current provider and IETF documentation before deployment.
1. Start with what was actually announced
Google announced an intention to support a shared policy token in robots.txt and described a 2027 target. That is different from saying every AI provider reads or enforces it today. Treat the announcement as a governance and implementation planning signal. Keep current controls in place until the providers you care about publish active support, precedence and product coverage.
2. Separate crawling, Search, training and grounding
Crawling is fetching a page, Search indexing is eligibility for search results, training changes model parameters, and grounding uses current information to help form an answer. One organisation may use different crawlers and rules for these purposes. Write the business objective for each purpose separately before touching robots.txt. A brand may want Search visibility while limiting model training, or may want selected pages eligible for cited AI answers.
3. Create a content inventory before a policy
List the content types on the site: public articles, product pages, help material, user submissions, gated research, partner assets, legal documents, customer portals and internal search results. Add owner, licence, commercial value, sensitivity and update frequency. A site-wide instruction is difficult to evaluate when the team does not know which material it owns or has permission to share.
4. Define the business decision and accountable owner
Ask whether the objective is rights protection, licensing leverage, privacy, traffic, attribution, brand visibility or a negotiated partner arrangement. Assign one decision owner with input from editorial, marketing, technology and legal teams. Robots.txt should express an approved policy, not become the place where a developer makes a business decision alone.
5. Understand the proposed shared token
The current Internet-Draft uses a policy token named `__usage_ai-training` inside the familiar robots.txt group structure. Its purpose is to express a training-use preference, not to identify one particular crawler. The draft remains changeable and is not a final IETF standard. Copying an example today without checking the final implementation date and syntax can create false confidence.
6. Keep provider-specific rules during transition
Google says existing company-specific controls will continue alongside the shared signal. If the website already uses Google-Extended or other documented AI crawler rules, record the exact reason, date and owner before changing them. During a transition, a provider-specific instruction may still be the only active signal. Avoid deleting working rules merely because a broader proposal has been announced.
7. Review Google-Extended separately
Google-Extended is a control related to whether site content may help improve certain Gemini models and whether it may be used for grounding in Gemini experiences. Google says that when the future shared token opts out and Google-Extended is not explicitly specified, it plans to interpret the shared preference as a Google-Extended disallow. An explicit Google-Extended choice may therefore have important visibility consequences; verify the current documentation before deciding.
8. Protect normal Google Search crawling
Do not confuse Google-Extended with Googlebot. Google documents that the Google-Extended product token does not affect inclusion or ranking in Google Search. A broad `User-agent: *` block, an accidental Googlebot disallow or a server-level denial is a different action and can damage Search visibility. Have an SEO and developer review every proposed robots.txt change together.
9. Decide whether path-level exceptions are needed
The draft model can express different preferences for different paths, but path rules create maintenance risk. A publisher might treat public press material differently from licensed reports; a retailer might separate product facts from customer-account areas. Use exceptions only when URL structure and ownership are stable. Never place personal, confidential or paid content on a public URL and rely on robots.txt as access control.
10. Fix access control before crawler policy
Robots.txt is publicly visible and does not authenticate a visitor. Sensitive material belongs behind proper authorization, not behind a disallow line. Audit staging sites, downloadable files, query-generated pages and forgotten subdomains. Remove public exposure, apply access control and then decide how legitimate public content should be treated for Search and AI use.
11. Build a staging and validation routine
Test the intended robots.txt file in a non-production environment and compare it line by line with production. Confirm the file returns a successful response, uses plain text, sits at the correct host root and does not redirect unexpectedly. Check important Search pages with the current Google tools and inspect server logs after release. A syntax-valid file can still express the wrong business policy.
- Snapshot the current robots.txt file and record the reason for each group.
- List representative URLs that should remain searchable, trainable or excluded.
- Test every rule against those URLs before deployment.
- Schedule a post-release crawl, index and log review.
12. Document conflicts and precedence
Write down what should happen if the shared preference and a provider-specific rule differ. Google has described one planned interpretation, but every provider may publish its own implementation detail. Contracts, licensing arrangements and authenticated data-sharing can also sit outside a public preference file. Keep a provider matrix with current status, source link, effective date and approved business choice.
13. Include rights, licensing and privacy review
An AI-training preference is not a substitute for copyright notices, contracts, privacy controls or data minimisation. Confirm that images, guest articles, testimonials, research and partner material are covered by the rights the business actually holds. Ask qualified legal and privacy professionals how the chosen signal relates to contracts and applicable law. Avoid claiming that robots.txt guarantees deletion of previously collected data.
14. Measure discovery before and after changes
Record organic Search clicks, AI-assistant referrals, cited-answer visibility where measurable, crawling patterns and content licensing outcomes before rollout. Annotation of the implementation date helps later analysis. Expect incomplete attribution: an AI answer can influence a user without sending a click. Combine analytics with brand-search trends, lead source notes and a sample of customer conversations.
15. Prepare communications for stakeholders
Tell editorial, marketing, product and leadership what the rule controls and what it does not. Publish an internal decision note with scope, rationale, owner, implementation status and review date. If the organisation publicly states an AI-content policy, keep the statement accurate and aligned with the technical configuration. Do not promise that all providers honour a draft signal before they do.
16. Follow a 30-day readiness plan
Days 1–7: inventory content, licences, current crawler rules and business objectives. Days 8–14: decide Search, training and grounding preferences with the appropriate owners. Days 15–21: draft the future configuration, provider matrix, URL test set and rollback plan. Days 22–30: validate existing controls, fix accidental exposure and schedule a review for provider implementation updates.
17. Use a deploy, wait or redesign decision
Deploy only when the relevant provider confirms active support, the syntax is current and the organisation has approved the consequences. Wait when the proposal or provider implementation remains incomplete. Redesign when the site structure cannot express the desired policy safely. The strongest outcome is not the fastest edit; it is a documented choice that preserves intended discovery while respecting content rights.
Key takeaways
Put the framework into action
- Google plans to support a shared AI-training preference in robots.txt in 2027; it should not be treated as universally active today.
- Crawling, Search indexing, model training and live answer grounding are different controls and business decisions.
- Audit content rights and current provider rules, test representative URLs and preserve Google Search access before deploying any new signal.
Research sources
Sources & further reading
This article provides general business education, not legal, tax or financial advice. Adapt the framework to your market and consult qualified professionals where required.