Sites API

Sites are the crawled websites that back an agent's knowledge. Use this API to inspect crawl state, browse indexed pages, and trigger a re-crawl. Gated on sites.read / sites.write.

List sites

GET /api/v1/rest/sites   ?widget_id=   |   ?kb_id=

Covers both agent-owned crawls and standalone knowledge-base sources (widget_id null, kb_id set). Scope with ?widget_id= or ?kb_id=.

Get one site

GET /api/v1/rest/sites/:id

Returns the site with status, pages_crawled, last_crawled_at, error, plus its schedule (crawl_cadence, next_crawl_at) and kb_id.

Re-crawl a site

POST /api/v1/rest/sites/:id

Queues a fresh crawl to pick up content changes. The agent keeps serving the existing index until the new one is ready.

Set the re-crawl schedule

PATCH /api/v1/rest/sites/:id
{ "crawl_cadence": "weekly" }     // off | daily | weekly | monthly

Sets automatic re-crawling; next_crawl_at is recomputed and the crawler picks up due sites on schedule.

List crawled pages

GET /api/v1/rest/sites/:id/pages   ?include_body=true

Add ?include_body=trueto inline each page's extracted text.

Delete a site

DELETE /api/v1/rest/sites/:id

Removes the site and its indexed pages. For knowledge added by hand rather than crawled, see the Knowledge API.