# /api/* is machine plumbing, not content. # # These routes back the site's own rendering and its in-page XHR. They return # JSON, they are linked from no page, and none of them is a URL anyone should # land on from a search result. Crawling them spends budget that belongs to the # 588 city pages instead. Googlebot was taking /api/decisions repeatedly. # # WARNING: REPEATED IN EVERY GROUP ON PURPOSE. robots.txt is most-specific-group-wins: # a crawler matching its own `User-agent:` block reads ONLY that block and never # sees the rules under `*`. A single Disallow under `*` would silently exempt # all the named crawlers below. If you add a group, add the Disallow with it. # # The same is true of `Content-Signal:`. It is a per-group directive, so it is # restated in every group that a named crawler reads. A signal written only # under `*` would be invisible to GPTBot, ClaudeBot, and the rest. # # This is a crawl-budget hint, not a control: /api/ is enforced in middleware, # which is what actually stops a client that ignores this file. # # THE EXCEPTIONS: six URLs under /api/ are documents, not plumbing, and every # group that disallows /api/ must Allow them back. Four are human-readable # documentation pages (/api/reference, /api/batch-decisions, /api/eligibility, # /api/idempotency) that share the namespace by accident of routing -- they are # the same pages middleware exempts in OPEN_EXACT_PATHS. The other two are # /api/llms.txt (the API's own agent-readable doc) and /api/widget/rates (the # one keyless, CORS-open endpoint, declared as a Feed in schema-map.xml below). # We link all six from llms.txt, index.md, and the .well-known documents, so # disallowing them means pointing crawlers at documents and then refusing to # serve them. They carry no corpus data; the guarded JSON proxies # (/api/credit-unions, /api/meta, /api/decisions, /api/benchmarks, /api/rates, # /api/geo, /api/stats, ...) stay disallowed, which is what the block was for. # # The Allow lines are placed BEFORE `Disallow: /api/` in each group. Modern # parsers use longest-match-wins so order does not matter to them, but a naive # first-match parser gets the right answer this way. # # NOT added to the CCBot / Bytespider / DataForSeoBot / MJ12bot groups: those # say `Disallow: /` for the whole site and have no /api/ rule to correct. An # Allow there would punch six holes in a deliberate full-site refusal. # /for-credit-unions/claim is a form, not content. Every profile page links a # unique pre-filled variant (?cu=...&state=...), so crawlers see ~3,700 "pages" # that are all the same form -- GPTBot fetched it 3,871 times in the 24h after # launch (2026-07-31), more than every rate page combined. The claim CTA is for # the human CU employee already on their profile page; a crawler's budget # belongs on the rate pages we want indexed and cited. Canonical already dedups # it for Google; this stops the fetches. # CONTENT SIGNAL (contentsignals.org) # # search=yes index it, cite it, link back. That is the trade this whole # file makes: bandwidth for reach. # ai-train=no do not use it as model training corpus. Training consumes the # corpus and returns nothing -- no citation, no referral, no # visit. It is the same test DataForSEO fails below. # # Note this is deliberately NOT `ai-input=no`. Retrieval-augmented answering is # the surface we most want to be in: an assistant fetching a page to answer a # person's question cites us, and that is reach. User-agent: * Content-Signal: search=yes, ai-train=no Allow: / Allow: /api/reference Allow: /api/batch-decisions Allow: /api/eligibility Allow: /api/idempotency Allow: /api/llms.txt Allow: /api/widget/rates Disallow: /api/ Disallow: /for-credit-unions/claim # Answer engines - explicitly allowed # # Two shapes here, and both are wanted. A "-Bot"/"-SearchBot" builds the index an # assistant retrieves from; a "-User" fetcher is a PERSON who just asked their # assistant about a credit union, and the fetch exists to cite us in the answer. # Blocking either removes us from the answer. User-agent: GPTBot Content-Signal: search=yes, ai-train=no Allow: / Allow: /api/reference Allow: /api/batch-decisions Allow: /api/eligibility Allow: /api/idempotency Allow: /api/llms.txt Allow: /api/widget/rates Disallow: /api/ Disallow: /for-credit-unions/claim User-agent: OAI-SearchBot Content-Signal: search=yes, ai-train=no Allow: / Allow: /api/reference Allow: /api/batch-decisions Allow: /api/eligibility Allow: /api/idempotency Allow: /api/llms.txt Allow: /api/widget/rates Disallow: /api/ Disallow: /for-credit-unions/claim User-agent: ChatGPT-User Content-Signal: search=yes, ai-train=no Allow: / Allow: /api/reference Allow: /api/batch-decisions Allow: /api/eligibility Allow: /api/idempotency Allow: /api/llms.txt Allow: /api/widget/rates Disallow: /api/ Disallow: /for-credit-unions/claim User-agent: ClaudeBot Content-Signal: search=yes, ai-train=no Allow: / Allow: /api/reference Allow: /api/batch-decisions Allow: /api/eligibility Allow: /api/idempotency Allow: /api/llms.txt Allow: /api/widget/rates Disallow: /api/ Disallow: /for-credit-unions/claim User-agent: Claude-User Content-Signal: search=yes, ai-train=no Allow: / Allow: /api/reference Allow: /api/batch-decisions Allow: /api/eligibility Allow: /api/idempotency Allow: /api/llms.txt Allow: /api/widget/rates Disallow: /api/ Disallow: /for-credit-unions/claim User-agent: Claude-SearchBot Content-Signal: search=yes, ai-train=no Allow: / Allow: /api/reference Allow: /api/batch-decisions Allow: /api/eligibility Allow: /api/idempotency Allow: /api/llms.txt Allow: /api/widget/rates Disallow: /api/ Disallow: /for-credit-unions/claim User-agent: Claude-Web Content-Signal: search=yes, ai-train=no Allow: / Allow: /api/reference Allow: /api/batch-decisions Allow: /api/eligibility Allow: /api/idempotency Allow: /api/llms.txt Allow: /api/widget/rates Disallow: /api/ Disallow: /for-credit-unions/claim User-agent: anthropic-ai Content-Signal: search=yes, ai-train=no Allow: / Allow: /api/reference Allow: /api/batch-decisions Allow: /api/eligibility Allow: /api/idempotency Allow: /api/llms.txt Allow: /api/widget/rates Disallow: /api/ Disallow: /for-credit-unions/claim User-agent: PerplexityBot Content-Signal: search=yes, ai-train=no Allow: / Allow: /api/reference Allow: /api/batch-decisions Allow: /api/eligibility Allow: /api/idempotency Allow: /api/llms.txt Allow: /api/widget/rates Disallow: /api/ Disallow: /for-credit-unions/claim User-agent: Perplexity-User Content-Signal: search=yes, ai-train=no Allow: / Allow: /api/reference Allow: /api/batch-decisions Allow: /api/eligibility Allow: /api/idempotency Allow: /api/llms.txt Allow: /api/widget/rates Disallow: /api/ Disallow: /for-credit-unions/claim User-agent: Google-Extended Content-Signal: search=yes, ai-train=no Allow: / Allow: /api/reference Allow: /api/batch-decisions Allow: /api/eligibility Allow: /api/idempotency Allow: /api/llms.txt Allow: /api/widget/rates Disallow: /api/ Disallow: /for-credit-unions/claim User-agent: Applebot-Extended Content-Signal: search=yes, ai-train=no Allow: / Allow: /api/reference Allow: /api/batch-decisions Allow: /api/eligibility Allow: /api/idempotency Allow: /api/llms.txt Allow: /api/widget/rates Disallow: /api/ Disallow: /for-credit-unions/claim User-agent: DeepSeekBot Content-Signal: search=yes, ai-train=no Allow: / Allow: /api/reference Allow: /api/batch-decisions Allow: /api/eligibility Allow: /api/idempotency Allow: /api/llms.txt Allow: /api/widget/rates Disallow: /api/ Disallow: /for-credit-unions/claim User-agent: ora-agent Content-Signal: search=yes, ai-train=no Allow: / Allow: /api/reference Allow: /api/batch-decisions Allow: /api/eligibility Allow: /api/idempotency Allow: /api/llms.txt Allow: /api/widget/rates Disallow: /api/ Disallow: /for-credit-unions/claim # Explicitly allowed Meta and Amazon general-purpose agents. These remain # independently named so their traffic can be audited without refusing it. User-agent: meta-externalagent User-agent: Amazonbot Content-Signal: search=yes, ai-train=no Allow: / Allow: /api/reference Allow: /api/batch-decisions Allow: /api/eligibility Allow: /api/idempotency Allow: /api/llms.txt Allow: /api/widget/rates Disallow: /api/ Disallow: /for-credit-unions/claim # Training-only crawlers - not welcome # # Same test as the resellers below: does anything come back? These two collect # for a training corpus and operate no surface that cites a source or sends a # reader. CCBot feeds Common Crawl, which is redistributed wholesale as training # data; Bytespider is ByteDance's model crawler. Neither indexes for a search # result a person clicks, so the bandwidth-for-reach trade this file makes # everywhere else simply does not apply. # # This is the enforcement of the `ai-train=no` signal stated in every group # above, made explicit for the two crawlers whose declared purpose IS training. # Both previously read `Allow: /`. User-agent: CCBot User-agent: Bytespider Content-Signal: search=no, ai-train=no Disallow: / # Resellers - not welcome # # The trade this file makes everywhere else is bandwidth for reach: a search or # AI crawler indexes the corpus, cites it, and sends people back. A data # reseller crawls it, sells it to its own customers, and sends nothing. # # DataForSEO took 4,822 requests in 24h (2026-07-28), 4,140 of them on # /credit-unions/*, making it the single largest consumer of the rate corpus -- # more than double the scraping operation we spend our time blocking. It has # never sent a referral. # # It was crawling honestly under a policy that said Allow: / to everyone, so # this is stated here BEFORE it is enforced in middleware. A crawler that # declares itself and obeys robots.txt should be turned away by robots.txt. User-agent: admino4ka-credit-unions User-agent: AhrefsBot User-agent: AIWebIndex User-agent: Barkrowler User-agent: DotBot User-agent: LinkupBot User-agent: SemrushBot User-agent: SERankingBacklinksBot User-agent: ShapBot Content-Signal: search=no, ai-train=no Disallow: / User-agent: DataForSeoBot Disallow: / # MJ12bot (Majestic) is the same trade, measured 2026-07-29: 332 requests in # 14h, every one allowed, concentrated on /credit-unions/*. Majestic sells a # backlink/SEO index to its subscribers; it does not index for a search surface # a person uses, so nothing comes back -- no citation, no referral, no visit. # Stated here BEFORE middleware enforces it, same as DataForSEO. User-agent: MJ12bot Disallow: / # Sitemap Sitemap: https://rateapi.dev/sitemap.xml # Schema Map (NLWeb Schema Feeds) # The machine-readable index of this site's structured-data feeds. schemamap: https://rateapi.dev/schema-map.xml # LLM Discovery # See https://rateapi.dev/llms.txt for AI-readable site documentation # See https://rateapi.dev/llms-full.txt for comprehensive LLM context