Model Router Sidecar

Cost-optimized LLM routing for multi-agent orchestration planes.

Architecture

┌─────────────────────────────────────────┐
│     Orchestrator Container              │
│  ┌──────────┐  ┌──────────┐            │
│  │ Triage   │  │ Analysis │            │
│  │ Agent    │  │ Agent    │            │
│  └────┬─────┘  └────┬─────┘            │
│       │             │                   │
│       └─────────────┴──────────┐        │
│                                │        │
│       ┌────────────────────────▼─────┐  │
│       │    ModelFactory              │  │
│       │  call_optimized_model()      │  │
│       └────────────┬─────────────────┘  │
│                    │                    │
└────────────────────┼────────────────────┘
                     │ HTTP
                     ▼
        ┌────────────────────────┐
        │  Model Router Sidecar  │
        │  (Port 8080)           │
        │                        │
        │  • Route by constraints│
        │  • Cost/speed/accuracy │
        │  • Provider config     │
        └───────────┬────────────┘
                    │
        ┌───────────┴────────────┐
        │                        │
        ▼                        ▼
   Azure OpenAI            Anthropic

Features

• Lazy client initialization - Only create LLM clients when needed
• Multi-constraint routing - Optimize for cost, speed, or accuracy
• Provider abstraction - Unified interface across OpenAI/Anthropic
• Sidecar pattern - One router per orchestration plane
• Configuration-driven - All model specs in YAML

Quick Start

# 1. Configure environment
cp .env.example .env
# Edit .env with your API keys

# 2. Start services
docker-compose up -d

# 3. Test routing
curl -X POST http://localhost:8080/route \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Classify this incident",
    "constraints": {
      "max_cost_per_1m_tokens": 0.50,
      "max_latency_ms": 2000,
      "min_accuracy_score": 0.80
    },
    "optimize_for": "cost"
  }'

# 4. Run example agents
docker-compose exec orchestrator python examples/itsm_demo.py

Project Structure

model-router-sidecar/
├── router/                 # Model router sidecar service
│   ├── server.py          # FastAPI routing service
│   ├── engine.py          # Routing decision logic
│   ├── models.yaml        # Model specifications
│   └── Dockerfile
├── orchestrator/          # Agent orchestration plane
│   ├── factory.py         # ModelFactory implementation
│   ├── agents/
│   │   ├── triage.py     # L1 triage agent
│   │   └── analysis.py   # L2 analysis agent
│   ├── examples/
│   │   └── itsm_demo.py  # ITSM automation demo
│   └── Dockerfile
├── docker-compose.yml
├── .env.example
└── README.md

Usage Examples

Basic Agent

from factory import ModelFactory

ModelFactory.configure('http://model-router:8080')

result = ModelFactory.call_optimized_model(
    prompt="Classify incident severity: App timeout after login",
    constraints={
        'max_cost_per_1m_tokens': 0.50,
        'max_latency_ms': 2000,
        'min_accuracy_score': 0.80
    },
    optimize_for='cost'
)

Multi-tier ITSM Flow

from agents.triage import TriageAgent
from agents.analysis import AnalysisAgent

# L1: Fast triage with cheap model
triage = TriageAgent()
severity = triage.classify(incident)

# L2: Deep analysis with better model (if needed)
if 'critical' in severity.lower():
    analysis = AnalysisAgent()
    root_cause = analysis.analyze(incident)

Configuration

Router Models (router/models.yaml)

models:
  gpt-4o-mini:
    provider: azure-openai
    endpoint: ${AZURE_OPENAI_ENDPOINT}
    cost_per_1m: 0.15
    avg_latency_ms: 1200
    accuracy_scores:
      classification: 0.87
      extraction: 0.85

Environment Variables (.env)

AZURE_OPENAI_KEY=your-key
AZURE_OPENAI_ENDPOINT=https://your-instance.openai.azure.com
ANTHROPIC_KEY=your-key
ROUTER_URL=http://model-router:8080

Cost Optimization

The router automatically selects the most cost-effective model that meets your constraints:

Tier	Constraints	Typical Model	Cost/1M Tokens
L1 Triage	Low cost, fast	gpt-4o-mini	$0.15
L2 Analysis	Balanced	gpt-4o	$2.50
L3 Research	High accuracy	claude-sonnet-4	$3.00

Example savings: For 10K incidents/month with 77% L1 resolution:

• Without routing: 10K × $2.50 = $25,000
• With routing: 7.7K × $0.15 + 2.3K × $2.50 = $6,905
• Savings: 72%

Development

# Run router standalone
cd router
pip install -r requirements.txt
python server.py

# Run tests
pytest tests/

# Add new model
# Edit router/models.yaml, restart router

License

MIT