Server-side retrieval quietly became the default. OpenAI's File Search does the chunking, embedding and search behind a chat API, and Cloudflare's AutoRAG will index an entire R2 bucket for you. The debate has moved past whether retrieval belongs in the platform - it's about how clever the retrieval agent should be.
All of it rests on one assumption: the model answering the question is theirs. Build on a provider's retrieval tool and you migrate when the provider migrates you - anyone who built on the original Assistants API retrieval knows how that goes. And the corpus you uploaded to make it work sits in the provider's storage, embedded with the provider's embedding model, useful to the provider's models.
The self-hosted alternatives stop at the chat window
The self-hosted side has the same problem pointing the other way. AnythingLLM and Open WebUI both do real document chat, and they run it on your machine, fully offline if you want.
But the retrieval happens inside each app's own chat window. The corpus, the embeddings, the index - all locked to one interface. A chat app is not an API. Your scripts, your agents, your internal tools get nothing.
Server-side retrieval, exposed as an OpenAI-compatible endpoint, against an embedding model and a vector database you run yourself - that combination we couldn't find anywhere. So we built it.
The ingest half runs on your machine
Infersec's RAG has two halves, and the split is the point.
Ingestion happens wherever the files live, via Infersec's Conduit application - the self-hosted agent you already run next to your hardware. The prerequisites are an embeddings model on one of your sources (something like nomic-ai/nomic-embed-text-v1.5-GGUF) and a Qdrant database to hold the resulting vectors - the store the API searches at query time. You run Qdrant yourself, next to your stack, and register it in the Infersec Console under Data - Vector Databases, with the API key stored encrypted. One requirement up front: the API performs the search at request time, so the database has to be reachable from the API. On the cloud platform that means a public address. On self-hosted, anything the API can reach - an internal one is fine.
Create an ingest job in the Console under Data - Ingest Jobs, then run the command its detail page prints, from the machine holding the files:
npx @infersec/conduit ingest files \
--target <ingest-job-id> \
--path ./docs \
--key <your-api-key> \
--api-url <api-url>
Conduit parses the files - PDF, Word, HTML, Markdown, CSV, JSON, plain text - and chunks them with overlap, breaking on sentence boundaries where it can. Scanned PDFs get an OCR fallback when the OCR dependencies are installed in Conduit.
Here's the part that matters: the embeddings come from your hardware. Conduit sends chunk batches up to the API, the API routes them to your embeddings source, and the vectors that come back go straight into a collection in your Qdrant.
What leaves the machine, precisely: chunk text, in transit, to be embedded. On the cloud deployment that text passes through the API in memory and is never stored - the raw files stay put the whole time. On self-hosted, with the API on your own network, nothing crosses a boundary you don't control.
Re-running an ingest is safe: each chunk is stored under a deterministic identifier derived from its file, so unchanged chunks overwrite rather than duplicate. Add --watch and Conduit re-ingests as files change, removing the stored chunks for files that disappear. Point it at the docs folder and forget about it.
The RAG layer your client can't see
The other half runs at request time, inside the API, before your model sees anything.
Attach RAG to an endpoint from its edit page - the RAG Retrieval panel. Pick the vector database and collection, and pick the embeddings source for queries - the same model that built the index. Tune it if you like: top-K, how many passages come back (four by default); an optional score threshold that drops weak matches; whether the context is prepended or appended. There's an optional query extractor too - a chat source that rewrites the user's question into a better search query before it's embedded.
Then every chat completion through that endpoint follows the same sequence:
The top hits go into a numbered context block injected as a system message - above the conversation by default - and inference runs as usual, tool loop included.
The client can't see any of it. No SDK, no file_search tool to enable, no prompt template to maintain, no extra round trip. Anything that speaks the OpenAI chat API - Chatbox, an internal agent, a script somebody wrote in an afternoon - points at the endpoint's base URL and starts answering from your corpus. The clients from our private chat piece would get RAG without touching a single setting.
If retrieval fails - database unreachable, collection empty, source offline - the request proceeds without context and the failure gets logged. A broken index degrades to a plain chat completion rather than an error.
It composes with tools
RAG and tool calling aren't either/or. The context block is injected before the server-side tool loop runs, so an endpoint with MCP tools attached gets retrieval and tools in the same request: the model reads the retrieved chunk, then goes and queries the database about it if that's what the answer needs. Let your AI query your database - safely covers the tool half of that pairing.
Cloud or self-hosted, same RAG
The feature behaves identically in both Infersec deployments, because your hardware does the embedding work either way. On the cloud platform each retrieval-augmented request records a rag_query charge - €0.005, same as a server-side tool interception - plus the usual token charges. Self-hosted has no per-query billing; licence tiers set how many RAG attachments you can run, and pricing has those numbers.
The corpus stays on your side of the wall. The API just knows where to look.
The RAG guide walks the whole setup end to end. Public access opens on the 1st of December - the 14-day self-hosted licence trial and €5 cloud credit start that day, and pricing has the numbers.
