Infersec

Infersec publishes pre-built Docker images that bundle the Conduit agent with vLLM for common GPU hardware. Use these when you want to serve models via vLLM without installing vLLM on the host yourself.

Available images

All images are published to the public Infersec registry and tagged with the release version (e.g. :1.87.0) plus :latest.

Image Hardware target Base image Architectures
rg.nl-ams.scw.cloud/infersec-public/infersec-production/conduit-vllm-nvidia NVIDIA CUDA GPUs vllm/vllm-openai linux/amd64, linux/arm64
rg.nl-ams.scw.cloud/infersec-public/infersec-production/conduit-vllm-amd AMD ROCm GPUs rocm/vllm linux/amd64
rg.nl-ams.scw.cloud/infersec-public/infersec-production/conduit-vllm-spark NVIDIA DGX Spark / ASUS Ascent GX10 (GB10 Grace-Blackwell) ghcr.io/timothystewart6/vllm-gb10 linux/arm64

The Spark image uses timothystewart6/vllm-gb10 which tracks upstream vLLM directly and is compiled natively for the GB10 sm_121a CUDA arch. The official NVIDIA NGC image tends to lag upstream releases.

Pull an image

docker pull rg.nl-ams.scw.cloud/infersec-public/infersec-production/conduit-vllm-nvidia:latest

Or pin to a specific release:

docker pull rg.nl-ams.scw.cloud/infersec-public/infersec-production/conduit-vllm-nvidia:1.87.0

Run Conduit

The image entrypoint is conduit inference start, so you only need to provide the runtime env vars. Pull the image, then start Conduit pointing at your Infersec source:

docker run -d \
  --name infersec-conduit \
  --gpus all \
  --ipc=host \
  --network host \
  -v ~/.cache/infersec/iagent:/data/iagent \
  -v ~/.cache/huggingface:/data/hf-cache \
  -e API_KEY=your-api-key \
  -e SOURCE=your-source-id \
  -e ROOT_DIRECTORY=/data/iagent \
  -e HF_HOME=/data/hf-cache \
  rg.nl-ams.scw.cloud/infersec-public/infersec-production/conduit-vllm-nvidia:latest

The container will:

  1. Download the model files (from HuggingFace) into the mounted HF cache.
  2. Boot vLLM and bind to --engine-port (default 9700).
  3. Connect to the Infersec API over --network host (or the API base URL you configure).

Once the inference source shows Online in the console, the source is ready for traffic. See Getting Started for the end-to-end flow.

Environment variables

The image bakes in ENGINE=vllm and API_URL=https://api.infersec.ai at build time. Override API_URL to point at a self-hosted deployment. Pass the rest at runtime:

Variable Required Default Notes
API_KEY Yes - Infersec API key.
SOURCE Yes - Inference source ID.
API_URL No https://api.infersec.ai API base URL (override for self-hosted).
ROOT_DIRECTORY No /data/iagent Where Conduit stores model files. Mount this as a volume.
HF_HOME No /data/hf-cache HuggingFace cache. Mount this as a volume to persist downloads.
ENGINE_PORT No 9700 Port the vLLM engine listens on.
PORT No 9600 Conduit listen port.
START_MODE No auto auto boots the engine; idle waits for an external engine.

See the Conduit docs for the full CLI and env var reference.

Hardware requirements

Each variant expects its respective GPU stack on the host:

  • NVIDIA: NVIDIA driver and NVIDIA Container Toolkit installed. Pass --gpus all.
  • AMD ROCm: ROCm driver and runtime. ROCm containers run on linux/amd64 only.
  • DGX Spark / GX10: GB10 base system with the NVIDIA stack already installed. See Ubuntu Server on the NVIDIA DGX Spark for a clean baseline. The image is linux/arm64 only.

For inference without Docker (e.g. macOS, or a Linux box with vLLM installed natively), install Conduit directly via npx — see the Conduit docs.