Reference links

You attach one or more URLs to an agent. A background job fetches each page, converts it to clean markdown, splits it into chunks, embeds them, and files each page as an ordinary knowledge-base document. From there it is the same retrieval as an uploaded PDF: a local search of a few tens of milliseconds, with nothing fetched while the call is running. A link can be a single page, or a whole site: in `site` mode the crawler follows the page's internal links, bounded by the limits below.

Send a list, not one URL at a time. `crawl_mode` is per link, because a documentation root is worth crawling as a `site` while a pricing page is not. The request returns immediately with each link `queued`: a fifty-page crawl with embedding does not fit inside an HTTP request.

curl -X POST "$BASE_URL/agents/$AGENT_ID/reference-links" \
  -H "X-API-Key: $AETHEX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "links": [
      { "url": "https://docs.example.com/", "crawl_mode": "site" },
      { "url": "https://example.com/pricing", "crawl_mode": "page" }
    ]
  }'

A crawl does not put anything in front of your callers. Every page it stores waits in `pending_review` until you approve the link, because a bad scrape that lands silently poisons all future calls. The preview sorts anomalies first rather than by crawl order. A held candidate, then anything awaiting review, and within each group the shortest extraction first, since a page that came back nearly empty is the likeliest bad scrape. Approving takes the whole link, minus the pages you exclude; an excluded page is remembered as rejected and loses its chunks, so the index carries nothing that will never be served.

# What the crawl found, anomalies first.
curl "$BASE_URL/agents/$AGENT_ID/reference-links/$LINK_ID/pages" \
  -H "X-API-Key: $AETHEX_API_KEY"

# Put them into service, minus the ones you do not want.
curl -X POST "$BASE_URL/agents/$AGENT_ID/reference-links/$LINK_ID/approve" \
  -H "X-API-Key: $AETHEX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "exclude": ["doc-uuid-of-a-page-you-reject"] }'

Every link is re-read on a fixed interval so a page that changes does not go stale. There is no per-link setting: the interval is the same for every link, which means a link cannot be pinned to never re-crawl. A pass that finds the page unchanged does nothing at all: it does not re-embed, and it does not ask you to approve anything again.

One case gets neither treatment: a re-crawl that comes back with a fraction of the text it had before. That is what a broken template or a site migration looks like, and replacing good content with it would be worse than doing nothing. So the page keeps serving what it had, and the new extraction is held aside as a candidate. The preview then shows both sides: the size before, the size now, and the candidate's own title, so you can judge which one is the real page.

{
  "id": "doc-uuid",
  "source_url": "https://docs.example.com/pricing",
  "status": "completed",
  "size_bytes": 7178,
  "chunk_count": 4,
  "candidate": {
    "at": "2026-08-31T12:19:22Z",
    "title": "Pricing",
    "size": 501,
    "previous_size": 7178
  }
}

# Accept it, and the shorter version takes over without the page
# ever leaving service.
curl -X POST \
  "$BASE_URL/agents/$AGENT_ID/reference-links/$LINK_ID/pages/$DOC_ID/accept-candidate" \
  -H "X-API-Key: $AETHEX_API_KEY"

`crawl_state` answers one question only: did the crawl work. It is written by a crawl and nothing else, so every value it can hold is settled the moment a pass ends.

A failure on a discovered page does not stop the crawl: it is recorded and the pass continues, and `last_crawl_error_detail` summarises what went wrong. A failure on the URL you attached fails the link, and `last_crawl_error_code` carries one of the following.

Ten links per agent. A `site` crawl reads up to 50 pages, three levels deep, and gives up after ten minutes. A single page is read up to 5 MB. The crawler identifies itself, honours `robots.txt` including `Crawl-delay`, and makes one request at a time per host with a pause between them. It is reading someone else's site, not stress-testing it. These are the same for every account and are not configurable.

Reference links and uploaded files land in the same place. See the knowledge base for what happens to a document once it is in service, and how retrieval uses it.