How Llms.txt And Robots.txt Affect AI Crawlers
Existing reputation helps disproportionately. A brand with review volume, press history and consistent details is starting from a partly assembled record. A brand with none of that is building identity from scratch, and identity work is slow because it depends on re-crawling sources you do not control.
Log the conditions with every run, including which assistant, llm seo which mode, whether web access was enabled and the date. When a result moves sharply, the conditions log is usually what tells you whether the world changed or your setup did.
It held because the results page was a list of destinations and nothing else. Reaching the top of that list meant being the first destination offered. As the page filled with features that answer in place, being first in the list stopped meaning being first on the screen, and it now sometimes means being below the answer.
Get the Basics Right Before Anything Clever Once access is confirmed, check that content actually exists for a crawler to read. Load your important pages with JavaScript disabled. If your specifications, pricing, service areas or contact details vanish, they are effectively absent from this channel regardless of how permissive your robots file is.
One caution for anyone reporting this upward. Do not present it as the end of search, because it is not, and the overstatement will be remembered when organic traffic is still the largest line in the report a year later. Present it as a change in what a position buys, which is both accurate and sufficient to justify a change in where content effort goes.
The decision that almost never makes sense for a commercial business is blocking the agents that fetch pages when composing answers. That is the mechanism by which you get recommended, and turning it off is the equivalent of declining to be listed anywhere, taken quietly, usually by accident.
Put someone's name against this. Crawler rules sit between marketing, development and whoever administers the content delivery network, which in most organisations means nobody checks them. The failures documented here are not difficult to find, they are simply nobody's job, and a quarterly review taking half an hour prevents the most complete form of invisibility available.
On Third Party Tracking Tools Several tools now offer to monitor this at scale, and they save real time once your prompt set runs into the hundreds. They are worth buying for trend lines and for coverage you cannot manually sustain.
One to Three Months: Listings and Corrections Claiming a directory profile, correcting an address, fixing a miscategorisation and responding to reviews all take effect once the platform publishes the change and the page is re-crawled.
Report frequency rather than presence. Being named in one run out of five is a genuinely different situation from being named in five out of five, and a report that collapses both to mentioned has thrown away the useful part.
Record the conditions alongside the results: which assistant, which model version if visible, whether web access was on, the date and the run number. When a result changes sharply, the conditions log is usually what tells you whether the world changed or your setup did.
Days One to Fourteen: Find Out Where You Stand Somebody writes fifty questions your buyers would ask, in their words. They run each one three times across the two or three assistants your customers use, from a signed out session, and record the full answers and every source cited.
In that setting your ranking is one input among several to a retrieval step, and often not a decisive one. Ahrefs found in July 2025, across 15,000 long-tail prompts, that around 80 percent of cited pages did not rank for the original query at all, with about 12 percent in the top ten.
The pages that earn citations are consistent across industries: an honest comparison of the options including where you are not the right choice, a plain definition page for the thing you sell, a specifications page with real numbers, and a pricing page that says something concrete.
Watch the source list as closely as the mention rate, because it usually moves first. New citations from a directory you corrected are a leading indicator, and they typically appear a month or two before any change in whether you are recommended.
Where Analytics Can and Cannot Help Referral traffic from assistant domains does show up in analytics, and it is worth segmenting into its own report. Treat the numbers as a floor rather than a count, since some assistants strip referrer information and some traffic arrives looking direct.
Also check the assumption underneath your own targets. Many teams still carry ranking goals inherited from a period when position and traffic moved together. A target expressed as positions gained is now measuring something that no longer reliably converts into visits, and leaving it in place quietly directs effort toward the metric rather than the outcome.