A restaurant that blocks Applebot-Extended in its robots.txt keeps its place in Apple Search. Apple has now written that down, adding a line to its Applebot documentation: "Site rules for Applebot-Extended are not considered in ranking for Search." Barry Schwartz reported the change at Search Engine Roundtable on 7 September, ahead of the public release of iOS 27.
Apple runs two crawler names and only one of them crawls
Applebot is the crawler. Apple's documentation says the data it collects powers "the search technology integrated into many user experiences in Apple's ecosystem including Spotlight, Siri, and Safari". If Applebot cannot fetch a menu page, Apple has nothing to draw on when someone asks Siri where to eat nearby.
Applebot-Extended fetches nothing. Apple states it plainly: "Applebot-Extended does not crawl webpages." It is a permission flag on material Applebot has already collected, deciding whether that material may "help train Apple foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools". Opting out is two lines:
User-agent: Applebot-Extended
Disallow: /
And Apple's stated consequence of using them: "Webpages that disallow Applebot-Extended can still be included in search results."
The costly mistake is the wildcard, not the opt-out
Plenty of robots.txt files have been rewritten to keep AI companies out, usually in a hurry and usually with a broad brush. The line that matters is which user agent the disallow lands on. Refuse Applebot-Extended and a site gives up a role in training Apple's models. Refuse Applebot and it gives up the index behind Siri, Spotlight and Safari suggestions.
For an independent restaurant, the second of those is the whole point of having a website. Voice queries about opening hours, cuisine and dishes are answered from a crawled index, and a page Applebot cannot reach is a page Siri cannot offer.
Apple has split its crawler the way Google split Googlebot
Google separated Google-Extended from Googlebot on the same logic: one user agent for search, a second that governs only generative model use. Apple has now matched both the structure and the reassurance that comes with it. Publishers weighing an AI opt-out had been weighing it against an unknown cost to Apple Search. That cost is now documented as zero.
Worth keeping in proportion: this is a documentation statement, not a measured result. Nobody has run a controlled test, and nobody outside Apple can. What has changed is that the answer is written on a support page an operator can point at, rather than inferred.
Nosnippet is the page-level version of the same choice
The documentation names a second control. A publisher can apply the nosnippet meta tag to stop specific content being used in AI-generated outputs, without a site-wide robots.txt rule. That distinction matters for a restaurant site where the calculus differs by page. A long piece about the kitchen's sourcing is one decision; the menu itself is another.
Structured menu data sits on the crawler side of the line
The opt-out Apple has clarified applies to model training. It does not touch what Applebot indexes, which is where a marked-up menu does its work. Restaurant, Menu and MenuItem markup, with prices, sections and dietary flags, remains readable by Apple whether or not the site has disallowed Applebot-Extended.
Apple's page does not spell out how the rebuilt Siri grounds answers in live web content, as distinct from training on it, and that gap is worth watching as iOS 27 rolls out. What the page does settle is narrower and useful: an AI training opt-out and a menu that Apple can read are not in tension.
Which leaves one line in one file worth reading before iOS 27 lands. A disallow on Applebot-Extended is now a documented choice with a known price. A disallow on Applebot is a restaurant removing itself from Apple Search, and nothing in the update softens that.