LLM Providers Guide

LibreFang ships with a comprehensive model catalog covering 6 native LLM drivers, 32 providers, 90+ builtin models, and 23 aliases. Every provider uses one of six battle-tested drivers: the native Anthropic driver, the native Gemini driver, the ChatGPT session driver, the GitHub Copilot OAuth driver, the Claude Code subprocess driver, or the universal OpenAI-compatible driver. This guide is the single source of truth for configuring, selecting, and managing LLM providers in LibreFang.

Table of Contents

1. Quick Setup
2. Provider Reference
3. Model Catalog
4. Model Aliases
5. Per-Agent Model Override
6. Model Routing
7. Cost Tracking
8. Fallback Providers
9. API Endpoints
10. Channel Commands

Quick Setup

The fastest path from zero to running:

# Pick ONE provider — set its env var — done.
export GEMINI_API_KEY="your-key"        # Free tier available
# OR
export GROQ_API_KEY="your-key"          # Free tier available
# OR
export ANTHROPIC_API_KEY="your-key"
# OR
export OPENAI_API_KEY="your-key"

LibreFang auto-detects which providers have API keys configured at boot. Any model whose provider is authenticated becomes immediately available. Local providers (Ollama, vLLM, LM Studio) require no key at all.

For Gemini specifically, either GEMINI_API_KEY or GOOGLE_API_KEY will work.

Provider Reference

1. Anthropic

Available Models:

• claude-opus-4-20250514 (Frontier)
• claude-sonnet-4-20250514 (Smart)
• claude-haiku-4-5-20251001 (Fast)

Setup:

1. Sign up at console.anthropic.com
2. Create an API key under Settings > API Keys
3. export ANTHROPIC_API_KEY="sk-ant-..."

2. OpenAI

Available Models:

• gpt-4.1 (Frontier)
• gpt-4o (Smart)
• o3-mini (Smart)
• gpt-4.1-mini (Balanced)
• gpt-4o-mini (Fast)
• gpt-4.1-nano (Fast)

Setup:

1. Sign up at platform.openai.com
2. Create an API key under API Keys
3. export OPENAI_API_KEY="sk-..."

3. Google Gemini

Available Models:

• gemini-2.5-pro (Frontier)
• gemini-2.5-flash (Smart)
• gemini-2.0-flash (Fast)

Setup:

1. Go to aistudio.google.com
2. Get an API key (free tier included)
3. export GEMINI_API_KEY="AIza..." or export GOOGLE_API_KEY="AIza..."

Notes: The Gemini driver is a fully native implementation. It is not OpenAI-compatible. Model goes in the URL path, system prompt via systemInstruction, tools via functionDeclarations, streaming via streamGenerateContent?alt=sse.

4. DeepSeek

Available Models:

• deepseek-chat (Smart) -- DeepSeek V3
• deepseek-reasoner (Smart) -- DeepSeek R1, no tool support

Setup:

1. Sign up at platform.deepseek.com
2. Create an API key
3. export DEEPSEEK_API_KEY="sk-..."

5. Groq

Available Models:

• llama-3.3-70b-versatile (Balanced)
• mixtral-8x7b-32768 (Balanced)
• llama-3.1-8b-instant (Fast)
• gemma2-9b-it (Fast)

Setup:

1. Sign up at console.groq.com
2. Create an API key
3. export GROQ_API_KEY="gsk_..."

Notes: Groq runs open-source models on custom LPU hardware. Extremely fast inference. Free tier has rate limits but is very usable.

6. OpenRouter

Available Models:

• openrouter/google/gemini-2.5-flash (Smart) -- cheap, fast, 1M context (default)
• openrouter/anthropic/claude-sonnet-4 (Smart) -- strong reasoning + tools
• openrouter/openai/gpt-4o (Smart) -- GPT-4o via OpenRouter
• openrouter/deepseek/deepseek-chat (Smart) -- DeepSeek V3
• openrouter/meta-llama/llama-3.3-70b-instruct (Balanced) -- Llama 3.3 70B
• openrouter/qwen/qwen-2.5-72b-instruct (Balanced) -- Qwen 2.5 72B
• openrouter/google/gemini-2.5-pro (Frontier) -- Gemini 2.5 Pro
• openrouter/mistralai/mistral-large-latest (Smart) -- Mistral Large
• openrouter/google/gemma-2-9b-it (Fast) -- Gemma 2 9B, free
• openrouter/deepseek/deepseek-r1 (Frontier) -- DeepSeek R1 reasoning

Setup:

1. Sign up at openrouter.ai
2. Create an API key under Keys
3. export OPENROUTER_API_KEY="sk-or-..."

Notes: OpenRouter is a unified gateway to 200+ models from many providers. Model IDs use the upstream format (e.g. google/gemini-2.5-flash). You can use any model from OpenRouter's catalog by specifying the full model path with the openrouter/ prefix.

7. Mistral AI

Available Models:

• mistral-large-latest (Smart)
• codestral-latest (Smart)
• mistral-small-latest (Fast)

Setup:

1. Sign up at console.mistral.ai
2. Create an API key
3. export MISTRAL_API_KEY="..."

8. Together AI

Available Models:

• meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo (Frontier)
• Qwen/Qwen2.5-72B-Instruct-Turbo (Smart)
• mistralai/Mixtral-8x22B-Instruct-v0.1 (Balanced)

Setup:

1. Sign up at api.together.ai
2. Create an API key
3. export TOGETHER_API_KEY="..."

9. Fireworks AI

Available Models:

• accounts/fireworks/models/llama-v3p1-405b-instruct (Frontier)
• accounts/fireworks/models/mixtral-8x22b-instruct (Balanced)

Setup:

1. Sign up at fireworks.ai
2. Create an API key
3. export FIREWORKS_API_KEY="..."

10. Ollama

Available Models (builtin):

• llama3.2 (Local)
• mistral:latest (Local)
• phi3 (Local)

Setup:

1. Install Ollama from ollama.com
2. Pull a model: ollama pull llama3.2
3. Start the server: ollama serve
4. No env var needed -- Ollama is always available

Notes: LibreFang auto-discovers models from a running Ollama instance and merges them into the catalog with Local tier and zero cost. Any model you pull becomes usable immediately.

11. vLLM

Available Models (builtin):

• vllm-local (Local)

Setup:

1. Install vLLM: pip install vllm
2. Start the server: python -m vllm.entrypoints.openai.api_server --model <model-name>
3. No env var needed

12. LM Studio

Available Models (builtin):

• lmstudio-local (Local)

Setup:

1. Download LM Studio from lmstudio.ai
2. Download a model from the built-in model browser
3. Start the local server from the "Local Server" tab
4. No env var needed

13. Perplexity AI

Available Models:

• sonar-pro (Smart) -- online search-augmented
• sonar (Balanced) -- online search-augmented

Setup:

1. Sign up at perplexity.ai
2. Go to API settings and generate a key
3. export PERPLEXITY_API_KEY="pplx-..."

Notes: Perplexity models have built-in web search. They do not support tool use.

14. Cohere

Available Models:

• command-r-plus (Smart)
• command-r (Balanced)

Setup:

1. Sign up at dashboard.cohere.com
2. Create an API key
3. export COHERE_API_KEY="..."

15. AI21 Labs

Available Models:

• jamba-1.5-large (Smart)

Setup:

1. Sign up at studio.ai21.com
2. Create an API key
3. export AI21_API_KEY="..."

16. Cerebras

Available Models:

• cerebras/llama3.3-70b (Balanced)
• cerebras/llama3.1-8b (Fast)

Setup:

1. Sign up at cloud.cerebras.ai
2. Create an API key
3. export CEREBRAS_API_KEY="..."

Notes: Cerebras runs inference on wafer-scale chips. Ultra-fast and ultra-cheap ($0.06/M tokens for both input and output on the 70B model).

17. SambaNova

Available Models:

• sambanova/llama-3.3-70b (Balanced)

Setup:

1. Sign up at cloud.sambanova.ai
2. Create an API key
3. export SAMBANOVA_API_KEY="..."

18. Hugging Face

Available Models:

• hf/meta-llama/Llama-3.3-70B-Instruct (Balanced)

Setup:

1. Sign up at huggingface.co
2. Create a token under Settings > Access Tokens
3. export HF_API_KEY="hf_..."

19. xAI

Available Models:

• grok-2 (Smart) -- supports vision
• grok-2-mini (Fast)

Setup:

1. Sign up at console.x.ai
2. Create an API key
3. export XAI_API_KEY="xai-..."

20. Replicate

Available Models:

• replicate/meta-llama-3.3-70b-instruct (Balanced)

Setup:

1. Sign up at replicate.com
2. Go to Account > API Tokens
3. export REPLICATE_API_TOKEN="r8_..."

21. ChatGPT (Session Auth)

Available Models:

• gpt-5.4-codex (Frontier)
• gpt-5.3-codex (Frontier)
• gpt-5.2-codex (Smart)
• gpt-5.1-codex (Smart)
• gpt-5.1-codex-mini (Balanced)

Setup:

1. Run librefang auth chatgpt to authenticate via browser session
2. The session token is captured automatically
3. Alternatively: export CHATGPT_SESSION_TOKEN="your-session-token"

Notes: This provider uses ChatGPT's internal backend API with session-based authentication. It leverages your existing ChatGPT subscription (Plus/Team/Enterprise). Models appear at zero API cost since they use your subscription. This is not the official OpenAI API -- it's the ChatGPT web interface backend.

22. GitHub Copilot

Available Models:

• copilot/gpt-4 (Frontier)
• copilot/gpt-4o (Smart)

Setup:

1. Ensure you have an active GitHub Copilot subscription
2. Run librefang auth copilot to authenticate via GitHub device flow
3. Token exchange is handled automatically -- no manual key management needed

Notes: The Copilot driver implements GitHub's OAuth device flow and automatic token exchange. Your GitHub token is exchanged for a Copilot API token transparently. Models appear at zero API cost since they use your Copilot subscription.

23. Moonshot (Kimi)

Available Models:

• kimi-k2.5 (Frontier) -- latest Kimi model with vision
• kimi-k2 (Frontier) -- strong reasoning with vision
• moonshot-v1-128k (Smart) -- 128K context
• moonshot-v1-32k (Balanced) -- 32K context
• moonshot-v1-8k (Fast) -- 8K context, cheapest

Setup:

1. Sign up at platform.moonshot.cn
2. Create an API key
3. export MOONSHOT_API_KEY="sk-..."

Example config:

[default_model]
provider = "moonshot"
model = "kimi-k2"

24. Qwen (Alibaba DashScope)

Available Models:

• qwen3-235b-a22b (Frontier) -- Qwen3 MoE 235B
• qwen-max (Frontier) -- strongest Qwen model
• qwen-vl-max (Frontier) -- vision flagship
• qwen-plus (Smart) -- good balance of cost and capability
• qwen-vl-plus (Smart) -- vision
• qwen-coder-plus (Smart) -- code-specialized
• qwen-coder-plus-latest (Smart) -- latest coder snapshot
• qwen-long (Balanced) -- 1M context window
• qwen2.5-coder-32b-instruct (Balanced) -- open-source coder
• qwen3-30b-a3b (Fast) -- small MoE, very cheap
• qwen-turbo (Fast) -- fastest Qwen model

Setup:

1. Sign up at dashscope.console.aliyun.com
2. Create an API key under DashScope console
3. export DASHSCOPE_API_KEY="sk-..."

Example config:

[default_model]
provider = "qwen"
model = "qwen-plus"

Notes: Qwen uses Alibaba Cloud's DashScope platform. The API is OpenAI-compatible. The qwen-long model supports up to 1M token context window.

25. Zhipu AI (GLM)

Available Models:

• glm-5-20250605 (Frontier) -- GLM-5, latest with vision
• glm-4-plus (Smart) -- strong reasoning, 128K context
• glm-4.7 (Smart) -- enhanced with vision
• glm-4v-plus (Smart) -- vision-specialized
• glm-4-long (Balanced) -- 1M context window
• glm-4-flash (Fast) -- free tier, 128K context

Setup:

1. Sign up at open.bigmodel.cn
2. Create an API key
3. export ZHIPU_API_KEY="..."

Example config:

[default_model]
provider = "zhipu"
model = "glm-4-plus"

Notes: GLM-4 Flash is completely free with no rate limits, making it excellent for development and testing.

26. MiniMax (International)

Available Models:

• MiniMax-M2.5 (Frontier) -- vision, 1M context
• MiniMax-M2.5-highspeed (Smart) -- fast variant with vision
• MiniMax-M2.1 (Smart) -- 1M context
• minimax-text-01 (Smart) -- 1M context
• abab7-chat (Smart) -- vision, 512K context
• abab6.5-chat (Balanced) -- 245K context

Setup:

1. Sign up at platform.minimax.chat
2. Create an API key
3. export MINIMAX_API_KEY="..."

Example config:

[default_model]
provider = "minimax"
model = "MiniMax-M2.5"

27. MiniMax (China)

Available Models: Same models as MiniMax International, served from China endpoints.

Setup:

1. Sign up at platform.minimaxi.com (China)
2. Create an API key
3. export MINIMAX_CN_API_KEY="..."

Notes: This is the China-domestic version of MiniMax with servers in mainland China. Use this if you need lower latency from China or compliance with data residency requirements.

28. Baidu Qianfan (ERNIE)

Available Models:

• ernie-4.5-8k (Smart) -- Baidu's flagship model
• ernie-4.0-turbo-8k (Balanced) -- fast variant
• ernie-speed-128k (Fast) -- free, 128K context

Setup:

1. Sign up at qianfan.cloud.baidu.com
2. Create an API key
3. export QIANFAN_API_KEY="..."

Example config:

[default_model]
provider = "qianfan"
model = "ernie-4.5-8k"

Notes: ERNIE Speed 128K is free with no usage limits, making it a great option for development and high-volume tasks.

29. Volcano Engine (Doubao)

Available Models:

• doubao-seed-1-6-251015 (Smart) -- Doubao flagship, 256K context
• doubao-seed-code (Smart) -- code-specialized
• doubao-seed-2-0-lite (Balanced) -- cost-effective
• doubao-seed-2-0-mini (Fast) -- cheapest option

Setup:

1. Sign up at console.volcengine.com
2. Enable the Ark (model service) product
3. Create an API key
4. export VOLCENGINE_API_KEY="..."

Example config:

[default_model]
provider = "volcengine"
model = "doubao-seed-1-6-251015"

Notes: Volcano Engine is ByteDance's cloud platform. Doubao models are ByteDance's in-house LLMs.

30. Venice.ai

Available Models:

• qwen3-235b-a22b-instruct-2507 (Smart) -- Qwen3 235B
• llama-3.3-70b (Balanced) -- Llama 3.3 70B
• venice-uncensored (Fast) -- uncensored model

Setup:

1. Sign up at venice.ai
2. Create an API key from dashboard
3. export VENICE_API_KEY="..."

Example config:

[default_model]
provider = "venice"
model = "llama-3.3-70b"

Notes: Venice.ai is an OpenAI-compatible inference platform focused on privacy and uncensored models.

31. Chutes.ai

Available Models:

• chutes/deepseek-ai/DeepSeek-V3 (Smart) -- DeepSeek V3
• chutes/deepseek-ai/DeepSeek-R1 (Smart) -- DeepSeek R1 reasoning
• chutes/Qwen/Qwen3-235B-A22B (Smart) -- Qwen3 235B
• chutes/meta-llama/Llama-4-Maverick-17B-128E-Instruct (Balanced) -- Llama 4 Maverick
• chutes/meta-llama/Llama-3.3-70B-Instruct (Balanced) -- Llama 3.3 70B

Setup:

1. Sign up at chutes.ai
2. Create an API key
3. export CHUTES_API_KEY="..."

Example config:

[default_model]
provider = "chutes"
model = "chutes/deepseek-ai/DeepSeek-V3"

Notes: Chutes.ai provides serverless inference for popular open-source models at competitive pricing.

32. Z.AI

Setup:

1. Sign up at z.ai
2. Uses the same API key as Zhipu AI
3. export ZHIPU_API_KEY="..."

Example config:

[default_model]
provider = "zai"
model = "glm-4-plus"

Notes: Z.AI is an international endpoint for Zhipu AI models. It shares the same API key (ZHIPU_API_KEY) and model catalog as Zhipu.

Model Catalog

The complete catalog of all 90+ builtin models, sorted by provider. Pricing is per million tokens.

Model Tiers:

Notes:

• Local providers (Ollama, vLLM, LM Studio) auto-discover models at runtime. Any model you download and serve will be merged into the catalog with Local tier and zero cost.
• The 46 entries above are the builtin models. The total of 51 referenced in the catalog includes runtime auto-discovered models that vary per installation.

Model Aliases

All 23 aliases resolve to canonical model IDs. Aliases are case-insensitive.

You can use aliases anywhere a model ID is accepted: in config files, REST API calls, chat commands, and the model routing configuration.

Per-Agent Model Override

Each agent in your config.toml can specify its own model, overriding the global default:

# Global default model
[agents.defaults]
model = "claude-sonnet-4-20250514"

# Per-agent override: use an alias or full model ID
[[agents]]
name = "orchestrator"
model = "opus"                      # alias for claude-opus-4-20250514

[[agents]]
name = "ops"
model = "llama-3.3-70b-versatile"   # cheap Groq model for simple ops

[[agents]]
name = "coder"
model = "gemini-2.5-flash"          # fast + cheap + 1M context

[[agents]]
name = "researcher"
model = "sonar-pro"                 # Perplexity with built-in web search

# You can also pin a model in the agent manifest TOML
[[agents]]
name = "production-bot"
pinned_model = "claude-sonnet-4-20250514"  # never auto-routed

When pinned_model is set on an agent manifest, that agent always uses the specified model regardless of routing configuration. This is used in Stabilisation mode (KernelMode::Stable) where the model is frozen for production reliability.

Model Routing

LibreFang can automatically select the cheapest model capable of handling each query. This is configured per-agent via ModelRoutingConfig.

How It Works

1. The ModelRouter scores each incoming CompletionRequest based on heuristics
2. The score maps to a TaskComplexity tier: Simple, Medium, or Complex
3. Each tier has a pre-configured model

Scoring Heuristics

Thresholds

Configuration

# In agent manifest or config.toml
[routing]
simple_model = "claude-haiku-4-5-20251001"
medium_model = "gemini-2.5-flash"
complex_model = "claude-sonnet-4-20250514"
simple_threshold = 100
complex_threshold = 500

The router also integrates with the model catalog:

• validate_models() checks that all configured model IDs exist in the catalog
• resolve_aliases() expands aliases to canonical IDs (e.g., "sonnet" becomes "claude-sonnet-4-20250514")

Cost Tracking

LibreFang tracks the cost of every LLM call and can enforce per-agent spending quotas.

Per-Response Cost Estimation

After each LLM call, cost is calculated as:

cost = (input_tokens / 1,000,000) * input_rate + (output_tokens / 1,000,000) * output_rate

The MeteringEngine first checks the model catalog for exact pricing. If the model is not found, it falls back to a pattern-matching heuristic.

Cost Rates (per million tokens)

Quota Enforcement

Quotas are checked on every LLM call. If the agent exceeds its hourly limit, the call is rejected with a QuotaExceeded error.

# Per-agent quota in config.toml
[[agents]]
name = "chatbot"
[agents.resources]
max_cost_per_hour_usd = 5.00   # cap at $5/hour

The usage footer (when enabled) appends cost information to each response:

> Cost: $0.0042 | Tokens: 1,200 in / 340 out | Model: claude-sonnet-4-20250514

Fallback Providers

The FallbackDriver wraps multiple LLM drivers in a chain. If the primary driver fails, the next driver in the chain is tried automatically.

Behavior

• On success: returns immediately
• On rate limit / overload errors (429, 529): bubbles up for retry logic (does NOT failover, because the primary should be retried after backoff)
• On all other errors: logs a warning and tries the next driver in the chain
• If all drivers fail: returns the last error

Configuration

Fallback chains are configured in your agent manifest or config.toml. The FallbackDriver is used automatically when an agent is in Stabilisation mode (KernelMode::Stable) or when multiple providers are configured for reliability.

# Example: primary Anthropic, fallback to Gemini, then Groq
[[agents]]
name = "production-bot"
model = "claude-sonnet-4-20250514"
fallback_models = ["gemini-2.5-flash", "llama-3.3-70b-versatile"]

The fallback driver creates a chain: AnthropicDriver -> GeminiDriver -> OpenAIDriver(Groq).

API Endpoints

List All Models

GET /api/models

Returns the complete model catalog with metadata, pricing, and feature flags.

Response:

[
  {
    "id": "claude-sonnet-4-20250514",
    "display_name": "Claude Sonnet 4",
    "provider": "anthropic",
    "tier": "Smart",
    "context_window": 200000,
    "max_output_tokens": 64000,
    "input_cost_per_m": 3.0,
    "output_cost_per_m": 15.0,
    "supports_tools": true,
    "supports_vision": true,
    "supports_streaming": true,
    "aliases": ["sonnet", "claude-sonnet"]
  }
]

Get Specific Model

GET /api/models/{id}

Returns a single model entry. Supports both canonical IDs and aliases.

GET /api/models/sonnet
GET /api/models/claude-sonnet-4-20250514

List Aliases

GET /api/models/aliases

Returns a map of all alias-to-canonical-ID mappings.

Response:

{
  "sonnet": "claude-sonnet-4-20250514",
  "haiku": "claude-haiku-4-5-20251001",
  "flash": "gemini-2.5-flash",
  "grok": "grok-2"
}

List Providers

GET /api/providers

Returns all 32 providers with auth status and model counts.

Response:

[
  {
    "id": "anthropic",
    "display_name": "Anthropic",
    "api_key_env": "ANTHROPIC_API_KEY",
    "base_url": "https://api.anthropic.com",
    "key_required": true,
    "auth_status": "Configured",
    "model_count": 3
  },
  {
    "id": "ollama",
    "display_name": "Ollama",
    "api_key_env": "OLLAMA_API_KEY",
    "base_url": "http://localhost:11434/v1",
    "key_required": false,
    "auth_status": "NotRequired",
    "model_count": 5
  }
]

Auth status values: Configured, Missing, NotRequired.

Set Provider API Key

POST /api/providers/{name}/key
Content-Type: application/json

{ "api_key": "sk-..." }

Configures an API key for a provider at runtime (stored as a Zeroizing<String>, wiped from memory on drop).

Remove Provider API Key

DELETE /api/providers/{name}/key

Removes the configured API key for a provider.

Test Provider Connection

POST /api/providers/{name}/test

Sends a minimal test request to verify the provider is reachable and the API key is valid.

Channel Commands

Two chat commands are available in any channel for inspecting models and providers:

/models

Lists all available models with their tier, provider, and context window. Only shows models from providers that have authentication configured (or do not require it).

/models

Example output:

Available models (12):

Frontier:
  claude-opus-4-20250514 (Anthropic) — 200K ctx
  gemini-2.5-pro (Google Gemini) — 1M ctx

Smart:
  claude-sonnet-4-20250514 (Anthropic) — 200K ctx
  gemini-2.5-flash (Google Gemini) — 1M ctx
  deepseek-chat (DeepSeek) — 64K ctx

Balanced:
  llama-3.3-70b-versatile (Groq) — 128K ctx

Fast:
  claude-haiku-4-5-20251001 (Anthropic) — 200K ctx
  gemini-2.0-flash (Google Gemini) — 1M ctx

Local:
  llama3.2 (Ollama) — 128K ctx

/providers

Lists all 32 providers with their authentication status.

/providers

Example output:

LLM Providers (32):

  Anthropic          ANTHROPIC_API_KEY       Configured    3 models
  OpenAI             OPENAI_API_KEY          Missing       6 models
  Google Gemini      GEMINI_API_KEY          Configured    3 models
  DeepSeek           DEEPSEEK_API_KEY        Missing       2 models
  Groq               GROQ_API_KEY            Configured    4 models
  Ollama             (no key needed)         Ready         3 models
  vLLM               (no key needed)         Ready         1 model
  LM Studio          (no key needed)         Ready         1 model
  ...

Environment Variables Summary

Quick reference for all provider environment variables:

Security Notes

• All API keys are stored as Zeroizing<String> -- the key material is automatically overwritten with zeros when the value is dropped from memory.
• Auth detection (detect_auth()) only checks std::env::var() for presence -- it never reads or logs the actual secret value.
• Provider API keys set via the REST API (POST /api/providers/{name}/key) follow the same zeroization policy.
• The health endpoint (/api/health) never exposes provider auth status or API keys. Detailed info is behind /api/health/detail which requires authentication.
• All DriverConfig and KernelConfig structs implement Debug with secret redaction -- API keys are printed as "***" in logs.