How Llms.txt And Robots.txt Affect AI Crawlers

What Not to Do in the Name of Legibility Hidden text intended only for machines fails on every axis. It is detectable, it violates most guidelines, and it produces exactly the uniform low quality signal you were trying to avoid.

Be prepared for the internal objection that this sends people to competitors. Some of it will, and those are mostly people who would not have bought from you anyway. The trade is that the page becomes usable as an impartial source, which is worth considerably more than the small number of poorly matched prospects it redirects, and the sales team usually agrees once they see which enquiries stop arriving.

Look at What They Do About Third Party Sources This is where the real work lives and where weak proposals are thinnest. Ask specifically what they will do about the review platforms, directories, forums and comparison articles that assistants actually cite in your category.

Anything a client cannot argue with is not a report. If you cannot open the document, disagree with a conclusion and point at the evidence that contradicts it, you have been sent a reassurance rather than an analysis.

Ask one final question before signing: what would you tell me if this is not working after six months? The answer reveals whether they have thought about failure, and an agency that has not thought about failure will not recognise it. brand mentions in ai answers

Blocking these is therefore not one decision. Turning away a training crawler is a defensible editorial position. Turning away the agent that fetches pages at answer time removes you from answers entirely, and the two are frequently confused.

Get the Basics Right Before Anything Clever Once access is confirmed, check that content actually exists for a crawler to read. Load your important pages with JavaScript disabled. If your specifications, pricing, service areas or contact details vanish, they are effectively absent from this channel regardless of how permissive your robots file is.

What Honest Reporting Contains The prompt set, versioned and unchanged since last month. The raw answers, kept in full rather than summarised. Which competitors were named. Which sources were cited. What work was done. What moved, and the specific claim about which work caused it.

One organisational habit makes this sustainable. Give the sales and support teams a single place to drop questions as they hear them, with no process attached beyond writing down the question in the customer's words. Anything more elaborate stops being used within a month, and a shared document with fifty verbatim questions in it is worth more than a formal intake process nobody completes.

Set a review cycle, quarterly for fast moving categories and twice a year otherwise. Update the figures rather than the timestamp, and show a real modified date so freshness can be judged honestly. brand mentions in ai answers

Watch the quality of enquiries as well as the count. A common early signal is that conversations start further along, with the prospect already aware of your price band, your typical timeline and what you do not do, because a machine told them before they arrived. That shows up in sales cycle length and in fewer wasted calls long before it shows up in any dashboard.

A Reasonable Sequence Fix rendering first, since content a machine cannot see is the only total failure in the list. Then work through your commercially important pages one at a time, moving the direct answer to the top and replacing the vaguest paragraph with concrete figures.

Vague answers about digital PR are a warning sign. Good answers are concrete: they have read your baseline source list, they know which platforms allow corrections, they have a view on which comparison articles are worth approaching, and they will tell you which ones are out of reach.

Everything else has to be transformed. A brand page has to be reframed as one option among several. A specification sheet has to be weighed against a competitor's. A comparison page needs none of that work, which makes it the cheapest source to use.

Then add the structural markup, then check the whole thing with a reader in mind rather than a crawler. If a page has become harder for a person to use, something has gone wrong and the change should be reversed.

The second is content behind interaction. Accordions, tabs and modals are good interface patterns and their content is sometimes absent from the initial response. Check whether yours is present in the HTML even when collapsed, which is usually a configuration question rather than a design one.

What Makes a Comparison Page Quotable Most vendor comparison pages are unusable, because they are arguments dressed as comparisons. Every row favours the publisher and the conclusion was written first, which is transparent to a reader and produces nothing a model can lift as an impartial claim.

Real questions are messy, specific and frequently uncomfortable. They ask about price, about limitations, about whether you can handle a particular awkward situation. That specificity is exactly what makes an answer quotable, because it matches the shape of a real query rather than a generic one.