Free scanTools…

Applebot-Extended user agent

Operated by Apple · Model training · Learning from you

robots.txt token
Applebot-Extended

Applebot-Extended is a robots.txt control token, not a crawler. No request ever arrives with Applebot-Extended in its User-Agent header, so grepping your access logs for it will always return nothing. Apple reads the token from your robots.txt and applies it to content its ordinary crawler has already fetched.

Allow or block Applebot-Extended in robots.txt

Robots.txt applies the most specific matching group, so a group naming Applebot-Extended overrides your User-agent: * rules for this bot alone.

Allow
User-agent: Applebot-Extended
Allow: /
Block
User-agent: Applebot-Extended
Disallow: /

What blocking costs you: Excluded from Apple Intelligence training; Siri/on-device summaries won't learn from you.

To see which AI crawlers your robots.txt allows right now, run your domain through the AI crawler access checker. To check whether AI answers actually cite you, use the AI citation checker.

Apple's other crawlers

Blocking one Apple bot does not block the others — each token is a separate group.

Apple's own documentation: https://support.apple.com/en-us/119829

FAQ

What is the Applebot-Extended user agent?
Applebot-Extended is not a crawler and sends no requests, so no User-Agent header ever contains it. Apple reads the token "Applebot-Extended" from your robots.txt as an opt-out switch, and honours it using the pages its ordinary crawler already fetched. Searching your access logs for it will always come up empty.
Who operates Applebot-Extended and what does it collect?
Applebot-Extended is run by Apple and collects pages as training data for a model. In the AI picture that puts it under "Learning from you": Feeding the models that power future answers.
How do I block Applebot-Extended?
Add a robots.txt group naming the token exactly: User-agent: Applebot-Extended followed by Disallow: /. Robots.txt matches the most specific group, so a Applebot-Extended group overrides whatever your User-agent: * group says. Excluded from Apple Intelligence training; Siri/on-device summaries won't learn from you.
Should I block Applebot-Extended?
That depends on what you lose. Excluded from Apple Intelligence training; Siri/on-device summaries won't learn from you. Sites chasing AI citations usually keep the live-answer and AI-search bots open and make a separate decision about the training crawlers. Run your domain through the AI crawler access checker to see which ones your robots.txt lets in today.
Can Applebot-Extended be spoofed?
Yes. A User-Agent header is self-reported, so any scraper can claim to be Applebot-Extended. The token is fine for reporting and for robots.txt, but if you are rate-limiting or firewalling on it, verify the request IP against Apple's published ranges before you trust the name.

Paste a full User-Agent line and identify any crawler with the AI bot user agent list and lookup, or see every token side by side.