How Llms.txt And Robots.txt Affect AI Crawlers
Entity Coherence Before a model can recommend you it has to be confident that the scattered mentions of your name refer to one company. That confidence comes from consistency across the details that identify you.
Direct Answers Beat Positioning When a model composes a recommendation it needs sentences it can attribute. Positioning language supplies none. A paragraph about being a trusted leader committed to excellence contains no attachable claim, so it is passed over in favour of a competitor who wrote down their turnaround time.
Structured data attracts a particular kind of over-investment. Teams implement a dozen schema types, validate them all, and conclude the job is done, having spent most of their effort on markup that changes nothing about how a machine understands the business.
Treat markup as something with a maintenance cost rather than a one off implementation. Prices change, people leave, products are discontinued, and structured data quietly keeps asserting the old version long after the visible page has been updated. Adding a schema review to whatever process already updates your pages costs minutes and prevents the most damaging failure mode, which is confidently stating something that is no longer true.
This is the least interesting subject in the discipline and the one that most often explains a total absence from generated answers. A brand can do everything else correctly and remain invisible because a line in a text file, or a setting nobody remembers enabling, is turning the relevant crawlers away.
Deciding Whether to Block Anything There is a legitimate argument for restricting training crawlers, particularly for publishers whose archive is the product. That is a commercial and editorial decision and it deserves a real discussion rather than a default.
The better approach is to keep them, correct the facts, date them honestly, answer engine optimization and make clear how they relate to the present. A page that says plainly what it documents and when is more useful than one quietly rewritten to look current.
A Numeric Name Is an Entity Problem Names beginning with digits behave differently across the web than names beginning with letters. They get written several ways, they sort strangely in directories, and they collide with unrelated numeric strings in ways that letter based names do not.
What to Do This Quarter Four things, none of which require a budget. Run the ten prompt self audit and find out where you actually stand. Claim and correct every listing on the sources your baseline shows are being cited.
Audit for contradiction before adding anything new. Run your key pages through a validator, then read the output against what the page actually says and against your main directory listings. Contradictions are more damaging than gaps, because they actively undermine confidence in the record.
One reframing helps when presenting this internally. Report the channel as influence rather than acquisition. Acquisition framing invites a comparison against paid media on cost per lead, which this channel will lose on the reported numbers even where it is working, because most of its effect never appears as a referral. Influence framing invites the right question, which is whether more of your market arrives already knowing who you are.
What Structured Data Is Doing Here Markup removes ambiguity. Prose says your company was founded in 2011 and operates in three counties, and a machine has to parse that from language. Structured data states it as a field, with no inference required.
The decision that almost never makes sense for a commercial business is blocking the agents that fetch pages when composing answers. That is the mechanism by which you get recommended, and turning it off is the equivalent of declining to be listed anywhere, taken quietly, usually by accident.
What Ranking Does and Does Not Buy You Ranking still helps, because the retrieval step usually starts with a search. But it buys far less than people assume. Ahrefs examined 15,000 long-tail prompts across four assistants in July 2025 and found roughly 80 percent of cited pages did not rank for the original query at all, with about 12 percent in the top ten.
What robots.txt Controls It is a request, honoured by mainstream crawlers, that certain user agents avoid certain paths. It has no enforcement behind it and it does not secure anything, but the major providers respect it.
What Transfers to an Ordinary Business Three things, and they are the three that most small operators skip. Check that you are readable before assuming you have a content problem, since on a small site an access failure is total rather than partial.
A quick way to find contradictions is to write out your key facts on one sheet, taken from your structured data, then check that sheet against your about page, your main directory listing and your marketplace account. Doing it manually feels crude and it surfaces the conflicts that validators never flag, because a validator checks syntax rather than whether your founding year matches the one you published elsewhere.