Gov Glance Crawl

Getting started

Every endpoint is a GET request that returns JSON. Send no headers and no credentials.

Base URL

https://crawl.govglance.org

Your first request

# Search for passport renewal pages
curl "https://crawl.govglance.org/api/v1/search?q=renew+passport&size=3"

Search

The main endpoint. Full-text search across every indexed page, in all three countries unless you narrow it.

Try it

Results will appear here.

Search operators

Operators go inside q and combine freely with the query parameters.

Response shape

Trimmed to one result for readability.

{
  "query": "renew passport",
  "total": 3255,
  "total_is_lower_bound": false,
  "page": 1,
  "size": 10,
  "results": [
    {
      "url": "https://travel.state.gov/…/renew.html",
      "title": "Renew my <mark>Passport</mark>",
      "snippet": "You can renew by mail if your …",
      "domain": "state.gov",
      "organization": "US Department of State",
      "country": "US",
      "level": "federal",
      "state": null,
      "language": "en",
      "content_type": "text/html",
      "published_at": null,
      "last_crawled": "2026-08-05T14:21:56Z",
      "word_count": 807,
      "score": 317.2
    }
  ],
  "facets": {
    "countries": [{ "value": "US", "count": 2911 }],
    "levels":  [{ "value": "federal", "count": 836 }],
    "states":  [{ "value": "DC", "count": 644 }],
    "domains": [{ "value": "uscis.gov", "count": 279 }],
    "types":   [{ "value": "text/html", "count": 3067 }]
  },
  "took_ms": 77.3
}

Rendering titles and snippets

title and snippet are already HTML-escaped and may contain <mark> around matched terms. Insert them as HTML; escaping them a second time renders the escaping itself, so apply/renew comes out as apply/renew. Every other field — url, domain, organization and the rest — is raw text and must be escaped by you before it reaches the DOM. For plain text, strip the <mark> tags and HTML-decode.

When total_is_lower_bound is true, total is a floor rather than an exact count.

Suggestions

Autocomplete drawn from indexed page titles. Useful for search-as-you-type.

curl "https://crawl.govglance.org/api/v1/suggest?q=medicare"

{ "query": "medicare", "suggestions": ["Medicare Costs", "Medicare Enrollment"] }

Countries

The countries in scope and the levels of government each one contributes. Build country and level pickers from this rather than hard-coding the lists, which change as the crawl grows.

Levels are deliberately not forced into a shared vocabulary: a UK council is not a US county, and collapsing both into "local" would discard the distinction you are usually filtering on.

public_body covers the several thousand .gov.uk registrations that the published register lists by name only. Recording them as public bodies is accurate; guessing whether each is a department or a council would not be.

curl "https://crawl.govglance.org/api/v1/countries"

{
  "default": "all countries",
  "countries": [
    { "code": "US", "name": "United States", "aliases": [],
      "levels": ["federal", "state", …], "domains": 13084, "pages_indexed": 2841003 }
  ]
}

Domains

Browse the government domains that make up the index.

Statistics

Index and crawl coverage. Recomputed in the background every five minutes.

indexed_pages and domains_total are exact. The frontier totals under crawl are planner estimates, flagged by counts_are_estimates, because an exact count of a table this size is far too expensive to serve. Also useful: GET /health returns per-dependency status and a 503 when search is unavailable.

Errors & limits

Errors return {"detail": "…"}. The rate limit is applied per IP per minute. Responses are cached briefly, so repeated identical queries are cheap and return X-Cache: HIT. If you need a higher limit for research or public-interest work, get in touch through govglance.org.