# brikiwik.org — an encyclopedia. Everything here is meant to be found.
#
# The split below is deliberate and is not a blanket AI block.
#
# ALLOWED: crawlers that index in order to ANSWER a question and cite the
# source. That is the whole point of the site — being found and being quoted.
# Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot and the
# user-triggered fetchers are all welcome and are covered by the Allow below.
#
# DISALLOWED: crawlers whose stated purpose is collecting a training corpus.
# Part data here is derived from Rebrickable, whose terms end "No Rebrickable
# content may be used in the training of AI models." Brikiwik does not train
# anything, but publishing openly is the route by which that content would
# reach a training set, and the cheap, reversible step is to say no here.
# Each vendor's own description of its token is quoted beside it.

User-agent: *
Allow: /

Sitemap: https://brikiwik.org/sitemap.xml

# --- slowed, not refused ---

# AhrefsBot feeds Ahrefs' backlink index and its Yep search engine; it plays
# no part in Google ranking. On 23 September 2026 it made about 80,000
# requests in a day, most of them rendered on Cloud Run. Its own page says
# "Crawl-delay is strictly followed when requesting HTML pages", so ten
# seconds keeps it at roughly 8,600 pages a day. A named group replaces the
# * group for that crawler, so the Allow has to be repeated here.
User-agent: AhrefsBot
Allow: /
Crawl-delay: 10

# --- training corpora: no ---

# "used to make our generative AI foundation models more useful and safe"
User-agent: GPTBot
Disallow: /

# "collecting web content that could potentially contribute to their training"
User-agent: ClaudeBot
Disallow: /

# Legacy Anthropic tokens, kept because older crawlers still send them.
# NO VENDOR DESCRIPTION: Anthropic's current crawler documentation lists only
# ClaudeBot, Claude-User and Claude-SearchBot, so there is nothing of theirs
# to quote for these two. Refused on the same reasoning as ClaudeBot above.
User-agent: anthropic-ai
Disallow: /
User-agent: Claude-Web
Disallow: /

# Common Crawl, "a non-profit foundation founded with the goal of
# democratizing access to web information by producing and maintaining an open
# repository of web crawl data". No model is trained by them; the corpus is
# the most widely reused training INPUT on the web, which is the same thing
# arriving by a longer road.
User-agent: CCBot
Disallow: /

# Gemini training AND Gemini grounding — Google bundles the two under one
# token, so this line has a real cost and is taken deliberately. It does not
# touch Googlebot: "Google-Extended does not impact a site's inclusion in
# Google Search nor is it used as a ranking signal in Google Search."
User-agent: Google-Extended
Disallow: /

# Apple: "Applebot-Extended is only used to determine how to use the data
# crawled by the Applebot user agent" — disallowing it opts the site out of
# training Apple's foundation models. Applebot itself (search) is allowed
# below, which is the whole reason this token exists separately.
User-agent: Applebot-Extended
Disallow: /

# ByteDance, TikTok's parent. NO VENDOR DESCRIPTION: ByteDance publishes no
# crawler documentation at all, so there is nothing of theirs to quote. Independent trackers describe it as collecting
# training data for ByteDance's own models, and it is widely reported to
# ignore robots.txt — which is a reason to state the refusal, not to omit it.
User-agent: Bytespider
Disallow: /

# Meta: "training foundation AI models or improving products by indexing
# content directly" — the first half is the reason.
User-agent: meta-externalagent
Disallow: /

# Amazon: "Amazonbot is used to improve our products and services", which
# their documentation goes on to say may include training Amazon AI models.
User-agent: Amazonbot
Disallow: /

# --- answering a reader's question: yes, explicitly ---
# These repeat the Allow above rather than relying on it, because a crawler
# that finds its own token in this file stops reading the wildcard group.

# "surface websites in search results in ChatGPT's search features"
User-agent: OAI-SearchBot
Allow: /

# "When users ask ChatGPT or a CustomGPT a question, it may visit a web page"
User-agent: ChatGPT-User
Allow: /

# "navigates the web to improve search result quality for users"
User-agent: Claude-SearchBot
Allow: /

# "When individuals ask questions to Claude, it may access websites"
User-agent: Claude-User
Allow: /

# Perplexity: "designed to surface and link websites in search results", and
# their documentation states it is not used to crawl for foundation models.
User-agent: PerplexityBot
Allow: /

# Google's search crawler. Their crawler documentation gives no one-line
# definition; what it does say is that preferences addressed to this token
# "affect Google Search (including Discover and all Google Search features)".
# Google-Extended above is the training token, and is refused.
User-agent: Googlebot
Allow: /

# Microsoft: "crawling is how Bingbot discovers new and updated pages and
# content to add to Bing's search index".
User-agent: Bingbot
Allow: /

# Apple: powers "the search technology integrated into many user experiences
# in Apple's ecosystem including Spotlight, Siri, and Safari". Apple's own
# docs say this data may also train their foundation models UNLESS
# Applebot-Extended is disallowed — which it is, above. The pair is the point.
User-agent: Applebot
Allow: /

# DuckDuckGo: "DuckDuckBot is a web crawler for DuckDuckGo."
User-agent: DuckDuckBot
Allow: /