LLM orchestration with multi-provider failover

How Arka orchestrates 24 LLM providers with automatic failover, API key rotation, per-task profiles, and per-skill model overrides.

Arka’s LlmFallbackEngine manages all LLM calls across 25 providers with automatic failover, API key rotation, and per-task or per-skill model overrides.

Supported providers

Anthropic, OpenAI, Gemini, Groq, xAI, DeepSeek, Moonshot, Z.AI, MiniMax, Venice, Mistral, Cohere, Together, Fireworks, Perplexity, Hugging Face, OpenRouter, Sakana Fugu, Bedrock, Azure, LiteLLM, Ollama, LM Studio, vLLM, and vLLM Cloud.

How failover works

The orchestrator builds a candidate list in order:

1. AI_PREFERRED_PROVIDER + AI_PREFERRED_MODEL (if set)
2. Live model lists from each provider API (when keys allow)
3. Built-in chain: Gemini 2.5/2.0 Flash → Groq Llama 3.3/3.1 → Ollama cloud/local

On 429, 401, or timeout, Arka tries the next model. API key rotation (API_KEY_ROTATION=1) cycles backup keys before switching providers.

AI_PREFERRED_PROVIDER=gemini
AI_PREFERRED_MODEL=gemini-2.0-flash
LLM_AUTO_FALLBACK=1
API_KEY_ROTATION=1

Per-task profiles

Task	Used for
route	Skill routing decisions
summarize	Email and document summaries
chat	Conversational replies
research	Deep web research
agent	Goal agent steps
pdf	PDF question answering
predictions	Stock and forecast analysis
compose_video	Video script generation
compose_slides	Presentation slide script generation

Override per task:

LLM_FALLBACK_ROUTE=gemini/gemini-2.0-flash,groq/llama-3.3-70b-versatile
LLM_FALLBACK_SUMMARIZE=groq/llama-3.3-70b-versatile

Per-skill models

arka ai-skill-model web_answer groq/llama-3.3-70b-versatile
arka ai-skill-model profiles
arka ai-models

Or edit ~/.config/arka/llm-skill-models.json.

Benchmark orchestration

If you want model selection to follow live benchmark winners, run a benchmark suite first and then enable orchestration:

arka benchmark run
arka benchmark show
ARKA_BENCHMARK_ORCHESTRATE=1 arka orchestrate --benchmark "how to get free ai credits"

The benchmark profile follows the routed task, so a request that routes to web_answer uses the chat benchmark results, while agent-style requests use agent.

Shared exhaustion cache

One session cache — exhausted models are skipped across skills until reset:

arka ai reset-exhaustion

Enable stderr notification on failover: LLM_FALLBACK_NOTIFY=1. Debug attempts: LLM_VERBOSE=1.

Local LLM servers

OLLAMA_HOST=127.0.0.1:11434
LLM_AUTO_START_SERVERS=1
LLM_AUTO_STOP_SERVERS=1

Why skills stay concise

Arka does not ask the LLM to memorize 70 skill manuals. Skills are local programs — the model only picks the right name. On LLM route, Arka passes:

• A compact skill catalog (registered names only)
• Curated routing rules (high-signal patterns)
• Shell aliases via ROUTE_ALIASES for your machine

The route task uses low temperature and a dedicated chain so a fast/cheap model handles classification; heavy lifting runs inside the skill locally.

Inspect at runtime

arka ai-models
arka ai models --task summarize
arka ai models --skill web_answer
arka ai active-model
arka ai reset-exhaustion

Set SHOW_MODEL=1 (default) to display Model: provider/name under answers.

Related