SEO 8 min read

A site's visibility in AI: llms.txt, sitemap and WebMCP

A site's visibility in AI: llms.txt, sitemap and WebMCP starts with separating the functions you are checking. Access to content, indexing in Google and executing an action in the browser...

2026-06-12

A site's visibility in AI: llms.txt, sitemap and WebMCP starts with separating the functions you are checking. Access to content, indexing in Google and executing an action in the browser have different requirements.

For a site's visibility in AI, check the specific service you care about. An llms.txt file does not guarantee that the page will be cited.

A site's visibility in AI: llms.txt, sitemap and WebMCP

In this guide, we distinguish Google's recommendations from the llms.txt proposal and from WebMCP interfaces. The verification procedure is a GOAI editorial recommendation, with no promises about indexing or source selection.

The cited documentation was consulted on September 14, 2026. For implementation, also check the version of the platform or browser in which the solution will run.

Decide which result you are after

In the GOAI analysis sheet, we recommend naming the target service and the result you can observe. In Google Search Console you can check whether a page is indexed, but this finding does not prove that it is cited in an AI answer.

Access: the service can receive the page's public content.
Indexing: the engine includes the page in its index, according to its own rules.
Citation: the answer being analyzed includes a link to your page.
Action: the agent can use a site function in a compatible environment.

For each check, keep the address, the date and the proof of what you observed. For the reports available in Google, see the Search Console guide and avoid extending the conclusion to other services without separate checks.

What Google says about llms.txt

Google's guide to generative features states that Google Search does not use llms.txt and other special files of this kind for optimization. Keeping them for other systems neither improves nor negatively affects visibility or rankings in Google Search.

Google also explains that a file being discovered or indexed does not mean it receives special treatment. A request for llms.txt in the server log must be interpreted within the limits of this observation.

To appear as a supporting link in AI Overviews or AI Mode, Google's documentation requires the page to be eligible for indexing and for display with a snippet in search. Meeting the conditions does not guarantee inclusion in the answer.

What the llms.txt proposal is

The llms.txt proposal, published by Jeremy Howard in 2024 and revised in August 2026, describes a Markdown file with information and links for agents. The version consulted allows it to be placed at the root of the site or in a section, for the addresses under that path.

The proposed structure includes the project name, a summary and lists of described resources. The document also recommends Markdown versions of pages, with mechanisms for indicating these alternatives.

For a documentation site, llms.txt can be evaluated as an entry point for an agent that uses it. Check the behavior of the chosen tool before presenting this possibility as a working integration.

The GOAI editorial recommendation is to keep the selection of resources limited to public content that is current and useful for the stated purpose. In the maintenance sheet, note who updates the file when the pages it points to change.

To check an llms.txt, pick a question about an included page and see whether the agent consults the indicated resource. In the test sheet, state whether you gave it the file's address directly, because this scenario does not prove that the agent would have discovered it on its own.

How to evaluate llms-full.txt and "AI sitemaps"

If a CMS module generates llms-full.txt, check the module's documentation to see which pages it includes and when it updates them. The file name does not confirm that a particular assistant requests or uses it.

For offers described commercially as an "AI sitemap", ask for the format produced and the documentation of the service that consumes it. In GOAI's recommendation, the check must identify the concrete difference from the XML sitemap or from a link index.

Identify the tool the file is prepared for.
Check that only published and accessible pages are included.
Compare the exported text with the current version in the CMS.
Test the use of the resource in the stated tool and keep the result.

A full export can duplicate information from WordPress pages and may need separate maintenance. If the product does not document a consumer or a way to test it, treat the promised benefit as unconfirmed.

The XML sitemap helps with discovering URLs

According to Google's sitemap documentation, the sitemap provides information about the site's pages and files. It can help with discovering URLs, without guaranteeing that every page will be crawled and indexed.

Google notes that pages that are properly linked to each other can be discovered by navigating the site. Therefore, a sitemap should not be presented as a universal obligation for every site.

To update the sitemap, see the guide on submitting an XML sitemap to Google. In the GOAI check proposed here, compare the URLs in the sitemap with the pages published in the CMS and investigate the differences.

In a CMS that publishes the sitemap automatically, check what happens when an article is withdrawn or changes its URL. We recommend keeping this check in the publishing procedure, so that the sitemap and internal links reflect the pages you want to make accessible.

Robots.txt controls crawling, without protecting accounts

Google's guide to robots.txt explains the file's role in controlling crawling by bots. Blocking crawling is not an authentication method, nor a guarantee that the URL will not appear in search.

A private WordPress area must be protected through the mechanisms of the application and the server. Avoid using a robots.txt rule as a substitute for permission checks.

In the technical audit, check the response served at the robots.txt address and the rules it contains. The fact that the PHP application generates it does not by itself prove an access or caching problem.

For every rule you change, save the previous configuration and the reason for the change. If the problem concerns the page appearing in Google, continue with the indexing checks.

Separate search bots from training bots

Anthropic's documentation assigns different roles to its bots. ClaudeBot collects content that may contribute to training, Claude-SearchBot supports search, and Claude-User fetches pages at users' request.

A rule for ClaudeBot should not automatically be described as a decision about access for all Claude functions. Record each agent separately, along with the purpose for which you allow or restrict access.

In Google's documentation on crawlers, Google-Extended is a product identifier for certain uses of content in Gemini. These include training and using content to support answers in the products listed in the documentation.

Google-Extended is not a separate HTTP identifier of a bot and does not control inclusion or ranking in Google Search. For each provider, check the exact name and the effect of the rule before publishing the configuration.

Check the protections in front of the server

Cloudflare AI Crawl Control provides tools for observing and controlling AI bot access. When analyzing a domain, check the actual settings and the events available in the account.

GOAI's recommendation is to compare the Cloudflare event with the server log for the same URL and period. That way you can investigate whether the request was stopped before it reached the application.

Changing the agent name in a test request does not reproduce the identity of the real bot. Anthropic publishes information for verifying source addresses, and for other providers their official method must be consulted.

In the verification sheet, also keep the content of the response received, not only the HTTP code. A login page or a browser verification page must be told apart from the article you are trying to access.

If you change a Cloudflare rule, repeat the same check and compare the result with the previously saved situation. Note exactly which rule was changed and keep the option of rolling back, to avoid attributing the result to another change made at the same time in the server or the application.

WebMCP enables actions in the browser

The WebMCP document of September 10, 2026 describes interfaces through which web applications offer tools to AI agents. The document has the status of a community group draft and states that it is not a W3C standard.

Chrome's overview of WebMCP describes a declarative interface for HTML forms and an imperative one for JavaScript interactions. These mechanisms concern an agent's use of site functions.

The specification consulted does not define the /.well-known/webmcp file as a mandatory discovery mechanism. Creating a resource with this name does not prove that the site offers working WebMCP tools.

In a GOAI test project, we recommend starting with a function that reads information, such as filtering a catalog. Check separately the compatible browser, the accepted parameters and the result returned to the agent.

For functions that submit forms or modify data, document the authorization and the confirmation of the action in the application. A successful WebMCP test shows that the interaction works, without confirming that the site is indexed or cited.

When evaluating WebMCP, also include the case where the agent supplies an incomplete or unsupported parameter. In the log, record the message received and confirm that the application does not perform an action different from the one described, before using the function on real data.

Keep the content verifiable

For CMS pages, we recommend answers that indicate the source and the conditions of application. The guide on original information in SEO explains how to document the contribution without made-up scores.

Structured data must match the information published, with no promises of selection in AI answers. For the current status of FAQPage, see the article with examples of frequently asked questions.

In the WordPress page's editorial sheet, check whether the service name and its conditions are consistent with those in the exported resources. If llms.txt or a full file keeps an old offer, fix the generation source and repeat the export before the next test.

Close the verification with separate evidence

In the proposed GOAI report, each conclusion must be tied to the test performed and the environment used. Avoid turning an observed citation into a promise for all the answers of an AI service.

Record the accessible URLs and any confirmed blocks.
Note the indexing findings from Search Console separately.
Keep the question, the date and the source cited in the AI answer checked.
Document the WebMCP tools tested and the result of each action.

To check bot access and indexing, see GOAI's technical SEO service. For content analysis and setting evidence-backed fixes, check the activities included in the SEO audit.

Source references

Dorel Tănase

Original author at GOAI. Published here in English with permission.

Share this article

Back to Blog