WOXABot, our web crawler
WOXABot visits public web pages to build WOXA's own search index. Here's how to recognise it, control it, and get pages updated or removed.
What WOXABot is
WOXABot is the crawler run by WOXA. It fetches public pages so they can appear in search on woxa.com and in the WOXA Search API. We build our own index so our results don't depend on other search engines.
It sends this User-Agent header on every request, including requests for robots.txt:
WOXABot/1.0 (+https://woxa.com/bot; polite-niche-crawler)
The WOXABot product token stays the same between versions. If you filter traffic, match on WOXABot rather than the whole string.
How WOXABot follows robots.txt
WOXABot reads /robots.txt before it fetches anything on a host and follows the Robots Exclusion Protocol (RFC 9309).
- A group for User-agent: WOXABot (matched case-insensitively) applies to us. If there isn't one, we follow the User-agent: * group.
- Allow and Disallow work by longest match, and the * and $ wildcards are supported.
- If robots.txt returns 404 or another 4xx error, we treat the site as open to crawling.
- If robots.txt returns a 5xx error or 429, or can't be reached at all, we don't crawl the site and try again later.
- If a page redirects to another host, that host's robots.txt must allow the target too.
- We cache robots.txt for up to 15 minutes, so changes take effect within about 15 minutes.
Crawl rate and conditional requests
- By default WOXABot waits at least 1 second between requests to the same host.
- If your robots.txt sets a longer Crawl-delay, we wait the full delay. We never cap or shorten it.
- On repeat visits we send If-None-Match and If-Modified-Since using your ETag and Last-Modified headers. A 304 Not Modified response costs you almost nothing.
- Pages that rarely change are revisited less often, as little as once every 30 days. Errors make us back off for longer.
- Each request times out after 8 seconds, and we download at most 8 MB per page.
Block WOXABot
To stop WOXABot crawling your whole site, add this to robots.txt:
User-agent: WOXABot
Disallow: /To keep it out of certain sections only:
User-agent: WOXABot
Disallow: /private/
Disallow: /*.pdf$
Allow: /private/press/Rules for User-agent: * apply to WOXABot when there's no WOXABot group. Pages we've already indexed are removed the next time we try to fetch them and robots.txt says no.
Slow WOXABot down
Set Crawl-delay in seconds (decimals are allowed). This example allows at most one request every 10 seconds:
User-agent: WOXABot
Crawl-delay: 10If WOXABot is causing you a problem that Crawl-delay doesn't fix, email us and we'll lower the rate for your site by hand. bot@woxa.com
Remove a page from WOXA
- Return 404 or 410 for the URL. We remove the page the next time WOXABot visits it.
- Disallow the URL in robots.txt. Indexed copies are removed on the next visit too.
- For a page that moved, use a permanent redirect (301 or 308). We replace the old URL with the new one.
- To bring the next visit forward, submit the URL with IndexNow (see below).
We don't read the noindex robots meta tag or the X-Robots-Tag header yet. Until we do, use robots.txt or a 404/410 status. For anything urgent, such as personal information, email us with the full URL. bot@woxa.com
Tell us about new or changed pages (IndexNow)
WOXA accepts IndexNow submissions, so you can tell us when a page is added, updated or deleted instead of waiting for our next visit.
- Create a key of 8 to 128 characters, using only letters, digits and hyphens.
- Host a UTF-8 text file containing only the key at https://your-site/<key>.txt. A key file elsewhere on the same host also works if you pass keyLocation, but it only covers URLs under that file's folder.
- Submit one URL with a GET request:
https://woxa.com/indexnow?url=https%3A%2F%2Fexample.com%2Fnew-page&key=<key> - Or submit up to 10,000 URLs from the same host in one POST:
POST https://woxa.com/indexnow Content-Type: application/json; charset=utf-8 { "host": "example.com", "key": "<key>", "keyLocation": "https://example.com/<key>.txt", "urlList": [ "https://example.com/new-page", "https://example.com/updated-page" ] }
A 200 response means we've queued the URLs. 403 means we couldn't verify the key file, 422 means a URL isn't on the submitted host or can't be crawled, and 429 means you've sent too many submissions. Each host can send up to 30 a minute. Submitted URLs still go through robots.txt.
What we use crawled pages for
Only for search: to rank and show results on woxa.com, and to answer requests to the WOXA Search API, including passage retrieval for developers. We store the page text, title, headings, links and the caching headers we need for polite recrawling.
We don't use crawled content for ad targeting, we don't build profiles of people from it, and we don't sell it or any personal data in it.
What WOXABot doesn't do
- It doesn't log in, accept or send cookies, or crawl pages that need an account. It skips login, sign-up, cart and admin paths.
- It doesn't try to get past bot protection, CAPTCHAs, JavaScript challenges or paywalls. If your site blocks it, it stays blocked.
- It doesn't fill in forms or follow links to private network addresses.
Contact
Questions, problems with crawl load, or removal requests: email us and include the affected URLs, the time and, if you think someone is pretending to be WOXABot, the IP address the requests came from.