Infersec

Conduit is the self-hostable agent that bridges your local hardware to the Infersec cloud. It downloads model files, manages the LLM engine lifecycle, and proxies inference requests. Conduit runs as a lightweight Node.js process or as a Docker container on your machine.

The quickest way to run Conduit is via npx:

npx @infersec/conduit inference start \
  --engine <engine> \
  --key <your-api-key> \
  --source <source-id>

For Docker use, see Docker below. For pre-built images that bundle Conduit with vLLM, see Engine Docker Images.

Commands

Conduit exposes four top-level commands:

Command Description
inference start Start the inference agent (downloads model, boots engine, connects to source)
tool connect <toolID> Connect a local tool (e.g. local-filesystem) to the API
models list List cached models and their disk usage
models clear Remove cached models (all, or filtered by ID prefix)
benchmark run Run model benchmarks from a benchmarks.json config file

inference start

Starts the inference agent against a single Infersec source.

Flag Required Default Notes
--engine Yes - Engine type, e.g. llama.cpp, vllm, sglang.
--key Yes - Infersec API key.
--source Yes - Inference source ID.
--api-url No https://api.infersec.ai API base URL (use the self-hosted URL for self-hosted deployments).
--engine-port No 9700 Port the LLM engine listens on.
--port No 9600 Port Conduit listens on (for the engine control plane).
--root No $HOME/.cache/infersec/iagent Root directory for model files and engine state.
--start-mode No auto auto boots the engine immediately; idle leaves Conduit waiting.

tool connect <toolID>

Connects a local tool (currently local-filesystem) to Infersec so it can be invoked by server-side tool calling.

Flag / positional Required Default Notes
<toolID> Yes - Tool ID to connect.
--key Yes - Infersec API key.
--path No . (current directory) Filesystem base path. Access is confined within this path.
--read-only No false Expose only read operations.
--api-url No https://api.infersec.ai API base URL.

models list / models clear

Manage cached model files on disk (the models/ directory under --root).

Flag Command Default Notes
--root both - Override root directory (or ROOT_DIRECTORY env).
--force models clear false Skip the interactive confirmation prompt.
--model models clear - Clear a specific model by ID prefix instead of all cached models.

benchmark run

Runs benchmarks from a benchmarks.json config file. Typically used by the Infersec team to populate Recommended Models; most users do not need this.

Flag Required Default Notes
--config Yes - Path to benchmarks.json.
--api-url No env API base URL (or API_URL env).
--api-key No env API key (or API_KEY env).
--account-id No env Account ID (or ACCOUNT_ID env).
--output-dir No config Override output directory.

Environment variables

Every CLI flag has a corresponding environment variable. Flags override env vars when both are provided.

Inference

Variable Required Default Notes
ENGINE Yes - Engine type (matches --engine).
API_KEY Yes - API key (matches --key).
SOURCE Yes - Inference source ID (matches --source).
API_URL No https://api.infersec.ai API base URL (matches --api-url).
ENGINE_PORT No 9700 Engine port (matches --engine-port).
PORT No 9600 Conduit listen port (matches --port).
ROOT_DIRECTORY No $HOME/.cache/infersec/iagent Root directory (matches --root).
START_MODE No auto Startup mode (matches --start-mode).
AUTO_PORTS No false Auto-allocate Conduit and engine ports. Conflicts with PORT/ENGINE_PORT.

Tool connect

Variable Required Default Notes
API_KEY Yes - API key (matches --key).
TOOL_ID Yes - Tool ID (matches <toolID> positional).
TOOL_PATH No . Filesystem base path (matches --path).
READ_ONLY No false Expose only read operations (matches --read-only).
API_URL No https://api.infersec.ai API base URL (matches --api-url).

Docker

When running Conduit in Docker, pass the same environment variables via -e flags (or an env file). The model cache and engine state should be mounted as a volume so they survive container restarts.

docker run -d \
  --name infersec-conduit \
  -p 9600:9600 \
  -v ~/.cache/infersec/iagent:/data/iagent \
  -e ENGINE=llama.cpp \
  -e API_KEY=your-api-key \
  -e SOURCE=your-source-id \
  -e ROOT_DIRECTORY=/data/iagent \
  node:24 \
  npx @infersec/conduit inference start

The relevant env vars in a container context are:

Variable Notes
ENGINE Engine type. For Docker-based engines (vllm), use a pre-built image — see Engine Docker Images.
API_KEY Infersec API key.
SOURCE Inference source ID.
API_URL Override the API base URL (e.g. for self-hosted deployments).
ROOT_DIRECTORY Path inside the container where model files are stored. Mount this as a volume.
ENGINE_PORT Port the LLM engine listens on (only relevant when Conduit launches the engine itself).
PORT Conduit listen port.
START_MODE auto (default) boots the engine; idle waits for an external engine.
HF_HOME HuggingFace cache location (set by the pre-built engine images to /data/hf-cache).

For pre-built images that bundle Conduit with vLLM for specific hardware (AMD ROCm, NVIDIA CUDA, NVIDIA DGX Spark), see Engine Docker Images.