How it works
One endpoint, four steps, no hidden state.
The entire public behaviour of Polymathy is a single request handler. There is no
background job, no crawler, no persistent index in v0.2. Here is exactly what happens
when a request hits GET /v1/search.
Query SearxNG
The handler issues a GET to SEARXNG_URL with the query q and format=json. SearxNG aggregates upstream engines and returns a ranked results array.
GET {SEARXNG_URL}/search?q={q}&format=jsonTake the top ten URLs
Polymathy reads the first ten entries from results[] and extracts their URLs. It does not re-rank; it trusts SearxNG’s ordering.
urls = results[..10].map(|r| r.url)Fan out to the processor
Each URL is POSTed to PROCESSOR_URL in parallel with a fixed config block. The processor fetches the page, strips boilerplate, chunks the text, and embeds each chunk.
POST {PROCESSOR_URL}
{ "url": "…",
"chunking_size": 100,
"chunking_type": "words",
"embedding_model": "AllMiniLML6V2" }Assemble the chunk map
Responses are collected into a single map keyed by sequential u64 chunk IDs, each value a (source_url, text) pair. That map is the JSON response.
{ "0": ["https://…", "…"],
"1": ["https://…", "…"] }What the vector index does (and doesn't)
A 384-dim, inner-product, F32-quantised USearch index is instantiated per request in
src/index.rs. In v0.2 the index is wired in but not yet used on the public read
path — Polymathy returns the full chunk map rather than a similarity-filtered subset. We
document what the source does, not what it might do later.
Because the index is built and discarded per request, Polymathy is stateless and scales horizontally with no coordination. The trade-off: nothing is cached between queries yet.