Gov Glance Crawl

Getting started

Every endpoint is a GET request that returns JSON. Send no headers and no credentials.

Base URL

https://crawl.govglance.org

Your first request

# Search for passport renewal pages
curl "https://crawl.govglance.org/api/v1/search?q=renew+passport&size=3"

Search

The main endpoint. Full-text search across every indexed page, in all three countries unless you narrow it.

Try it

Results will appear here.

Search operators

Operators go inside q and combine freely with the query parameters.

Response shape

Trimmed to one result for readability.

{
  "query": "renew passport",
  "total": 3255,
  "total_is_lower_bound": false,
  "page": 1,
  "size": 10,
  "results": [
    {
      "url": "https://travel.state.gov/…/renew.html",
      "title": "Renew my <mark>Passport</mark>",
      "snippet": "You can renew by mail if your …",
      "domain": "state.gov",
      "organization": "US Department of State",
      "country": "US",
      "level": "federal",
      "state": null,
      "language": "en",
      "city": null,
      "content_type": "text/html",
      "published_at": null,
      "content_changed_at": "2026-07-30T02:11:04Z",
      "updated_at": "2026-07-30T02:11:04Z",
      "last_crawled": "2026-08-05T14:21:56Z",
      "word_count": 807,
      "score": 317.2
    }
  ],
  "facets": {
    "countries": [{ "value": "US", "count": 2911 }],
    "levels":  [{ "value": "federal", "count": 836 }],
    "states":  [{ "value": "DC", "count": 644 }],
    "cities":  [{ "value": "Washington", "count": 612 }],
    "domains": [{ "value": "uscis.gov", "count": 279 }],
    "types":   [{ "value": "text/html", "count": 3067 }]
  },
  "took_ms": 77.3
}

Rendering titles and snippets

title and snippet are already HTML-escaped and may contain <mark> around matched terms. Insert them as HTML; escaping them a second time renders the escaping itself, so apply/renew comes out as apply/renew. Every other field — url, domain, organization and the rest — is raw text and must be escaped by you before it reaches the DOM. For plain text, strip the <mark> tags and HTML-decode.

When total_is_lower_bound is true, total is a floor rather than an exact count.

Recent feeds

The latest published or updated pages, ordered by date instead of relevance. Every filter from Search applies, so the same endpoint gives you "what has Austin published lately" and "what has the UK published lately".

What "recent" means

Two independent signals say a page is current: the date the page declares, and the date we watched its text change on a recrawl. sort chooses between them.

Pages with no known date are left out. A page we have crawled once that declares no date has no recency we can honestly report, and dating it to the moment we happened to reach it would turn the feed into a report on our own crawler. Dates in the future are dropped for the same reason: a page about next month's council meeting was not published next month.

Examples

# Everything Austin, Texas has published in the last month
curl "https://crawl.govglance.org/api/v1/recent?city=Austin&state=TX&since=30d"

# UK councils only, as an RSS feed you can subscribe to
curl "https://crawl.govglance.org/api/v1/recent.rss?country=uk&level=local_authority"

# Pages about flooding that changed in the past week, anywhere
curl "https://crawl.govglance.org/api/v1/recent?q=flooding&sort=changed&since=7d"

The JSON response is the same shape as Search, plus sort, sort_meaning, window and place echoing what was actually applied. The RSS variant is RSS 2.0 and takes the same parameters except until, page and facets.

Places

The states, provinces and cities you can filter on. Only places with at least one crawled government domain appear, so this describes the index rather than the world. Every city value returned is exactly what the city filter expects.

{
  "states": [
    { "code": "TX", "name": "Texas", "country": "US",
      "type": "state or territory", "cities_indexed": 431 }
  ],
  "cities": [
    { "city": "Austin", "state": "TX",
      "state_name": "Texas", "country": "US", "domains": 47 }
  ],
  "cities_known": 9332
}

What a city value means

The jurisdiction a domain serves, not the address its registrant receives post at. Those differ for about one US city domain in six — addisontx.gov is registered from Dallas — so the served city is reconstructed from the organization name and confirmed against the domain name before it is indexed. Where the evidence is inconclusive the registry's own answer is kept, so a handful of small jurisdictions still carry a neighbouring town's name.

The United Kingdom contributes no entries to states. No published register records which nation or region a UK public body belongs to, and inferring it from a council's name would be wrong often enough to be worse than saying nothing. UK results are filtered by city, level and domain instead.

Suggestions

Autocomplete drawn from indexed page titles. Useful for search-as-you-type.

curl "https://crawl.govglance.org/api/v1/suggest?q=medicare"

{ "query": "medicare", "suggestions": ["Medicare Costs", "Medicare Enrollment"] }

Countries

The countries in scope and the levels of government each one contributes. Build country and level pickers from this rather than hard-coding the lists, which change as the crawl grows.

Levels are deliberately not forced into a shared vocabulary: a UK council is not a US county, and collapsing both into "local" would discard the distinction you are usually filtering on.

public_body covers the several thousand .gov.uk registrations that the published register lists by name only. Recording them as public bodies is accurate; guessing whether each is a department or a council would not be.

curl "https://crawl.govglance.org/api/v1/countries"

{
  "default": "all countries",
  "countries": [
    { "code": "US", "name": "United States", "aliases": [],
      "levels": ["federal", "state", …], "domains": 13084, "pages_indexed": 2841003 }
  ]
}

Domains

Browse the government domains that make up the index.

Statistics

Index and crawl coverage. Recomputed in the background every five minutes.

indexed_pages and domains_total are exact. The frontier totals under crawl are planner estimates, flagged by counts_are_estimates, because an exact count of a table this size is far too expensive to serve. Also useful: GET /health returns per-dependency status and a 503 when search is unavailable.

Errors & limits

Errors return {"detail": "…"}. The rate limit is applied per IP per minute. Responses are cached briefly, so repeated identical queries are cheap and return X-Cache: HIT. If you need a higher limit for research or public-interest work, get in touch through govglance.org.