Gov Glance Crawl

About the crawler

GovCrawlBot visited your site

It is building a free, public search index of official government websites so that people can find what their governments publish. This page explains what it does, and how to slow it down or turn it off.

Identification

User-Agent: GovCrawlBot/1.0 (+https://crawl.govglance.org/about)
Operated by: Gov Glance, an independent non-profit publisher of government information. Not affiliated with any government agency.
Contact: crawler@govglance.org
What it indexes: Publicly reachable pages and PDFs on official government domains in the United States, Canada, and the United Kingdom.
Search index: crawl.govglance.org, with a free public API

How it behaves

It obeys robots.txt. Rules for GovCrawlBot are applied first, and User-agent: * otherwise. Crawl-delay is honoured up to a cap.
One request at a time per host, with a delay between them. It will not open parallel connections to the same server.
It prefers your sitemap. Reading a sitemap is cheaper for your server than following every link, so it is always tried first.
It re-reads pages rarely — typically every one to four weeks, adjusted by how often a page actually changes.
It sends conditional requests. If you return ETag or Last-Modified, it will send If-None-Match and take a 304 for an answer.
It backs off on errors and stops entirely after repeated failures.
It does not submit forms, run JavaScript, attempt logins, or collect personal data.

Controlling it

Slow it down

# In your robots.txt
User-agent: GovCrawlBot
Crawl-delay: 10

Keep it out of one area

User-agent: GovCrawlBot
Disallow: /search
Disallow: /calendar

Block it entirely

User-agent: GovCrawlBot
Disallow: /

Changes take effect within a day, which is how long robots.txt is cached. If you need something removed sooner, or a URL dropped from the index, email us and we will handle it by hand.

Please prefer robots.txt to a firewall rule. Blocking at the edge returns a 403 that looks identical to a transient failure, so the crawler keeps retrying on its normal schedule. A robots.txt rule is understood, respected, and costs your server one request a day instead.

Why this exists

Government information is public by law and hard to find in practice. It is spread across tens of thousands of separate websites, most with a search box that only covers that one site, and commercial search engines index them unevenly and rank them against everything else on the web.

This index covers government domains and nothing else, which means a search for a form, an ordinance, or a council agenda returns the government's own page rather than a copy of it. Everything indexed is already public; nothing behind a login is touched.