Exa

Exa Search Crawler

Last Updated: July 30, 2026

ExaSearchBot

Exa is a search engine. Similar to other search engines, Exa uses a robot (ExaSearchBot) to discover and index pages on the public web. This index allows users to find, retrieve, and cite your content.

Below is information on ExaSearchBot and how you can set your site preferences to enable or limit access to your content.

Purpose

ExaSearchBot fetches pages on the public web, which enables your pages to become discoverable through Exa. Its purpose is search and retrieval: connecting people and applications to your content and sending them back to your site.

Identification

ExaSearchBot sends this user-agent:

Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/)

Its robots.txt product token is ExaSearchBot.

User-agent strings can be spoofed, so don't trust the string alone — verify cryptographically (below).

Verification

Every ExaSearchBot request is cryptographically signed using HTTP Message Signatures (RFC 9421), following the Web Bot Auth scheme. Each request carries Signature, Signature-Input, and a Signature-Agent header identifying Exa.

On Cloudflare, Akamai, and other providers that support Web Bot Auth, ExaSearchBot is validated automatically at the edge — nothing to configure on your side.

To verify it yourself: fetch our public key directory and validate the request signature per RFC 9421. Our Ed25519 keys are published at:

https://crawler.exa.ai/.well-known/http-message-signatures-directory

Because verification is cryptographic, it holds regardless of the request's IP address.

Respecting robots.txt

ExaSearchBot respects the Robots Exclusion Protocol. When choosing which rules to follow, it looks for a group addressed to ExaSearchBot. If you haven't provided one, it follows the rules you've set for major search engines; otherwise it follows the * group.

Give ExaSearchBot its own rules (takes precedence):

User-agent: ExaSearchBot
Disallow: /private/

Block ExaSearchBot from your whole site:

User-agent: ExaSearchBot
Disallow: /

Removing pages from index

robots.txt controls crawling, not indexing. To remove a page already in Exa's index, use a noindex robots meta tag or an X-Robots-Tag: noindex header; ExaSearchBot drops the page after it next re-fetches it.

How ExaSearchBot behaves

Respects robots.txt

Checked before fetch, honoring the precedence above.

Signs requests

So you can confirm a request is genuinely us.

Crawls politely

Rate-limited per site to avoid putting load on your servers.

Public content only

It does not attempt to bypass logins, paywalls, or CAPTCHAs, and submits no forms.

Contact

If you have questions or need assistance, please reach out to support@exa.ai. If possible, please reach out from an email that includes the domain you are contacting us about, which facilitates verification.