ADR-0014 — OpenAI Model Selection Strategy¶

Context¶

Phase 3 multi-agent system requires strategic model selection to balance performance, cost, and capability. Options considered:

Uniform GPT-4o: Use GPT-4o for all agents
Uniform GPT-4o-mini: Use GPT-4o-mini for all agents
Strategic Mixed Selection: Match model complexity to agent requirements

Decision¶

Implement strategic mixed model selection:

GPT-4o: Conversation Manager (complex orchestration decisions)
GPT-4o-mini: Intent Recognition, Response Generation (specialized, frequent tasks)
Traditional Logic: Task Planning, Tool Coordination (deterministic operations)

Rationale¶

GPT-4o for Conversation Manager¶

Justification: Complex orchestration requires advanced reasoning capabilities - Multi-agent workflow coordination decisions - Session context management across conversations - Dynamic routing based on query complexity - Error recovery and fallback strategy selection

GPT-4o-mini for Specialized Agents¶

Justification: Well-defined tasks with clear input/output patterns - Intent Recognition: Classification task with structured output - Response Generation: Template-based natural language generation - Lower cost for high-frequency operations - Sufficient capability for focused responsibilities

Traditional Logic for Coordination¶

Justification: Deterministic operations don't require LLM overhead - Task Planning: Rule-based workflow generation - Tool Coordination: API orchestration with known patterns

Alternative Approaches Considered¶

All GPT-4o Approach¶

Pros: Maximum capability for all agents
Cons: 5x higher costs, unnecessary complexity for simple tasks
Cost Impact: $0.005 per query → $0.025 per query

All GPT-4o-mini Approach¶

Pros: Lowest cost option
Cons: Insufficient capability for complex orchestration decisions
Performance Impact: 60% degradation in conversation management quality

Consequences¶

Positive¶

85% cost reduction vs. uniform GPT-4o approach
Maintained high quality for complex orchestration decisions
Optimized performance for frequent, specialized operations
Clear model-to-task matching for future development

Negative¶

Mixed model management complexity in deployment
Need for different prompt optimization strategies per model
Potential inconsistency in response patterns between agents

Implementation Notes¶

Model selection configured per agent class initialization
Unified OpenAI client with per-agent model specification
Consistent temperature settings: 0.1 for classification, 0.3 for generation
Token usage monitoring per model type for cost optimization

Performance Results¶

Query Cost: $0.13 (Claude CLI) → $0.001 (multi-agent system)
Response Time: 3-10 seconds → 1-5 seconds average
Quality Maintenance: No degradation in technical accuracy
Agent Efficiency: GPT-4o-mini handles 80% of token volume at 20% cost

Success Metrics¶

99% cost reduction from original Claude CLI approach
Sub-5 second response times maintained across model mix
100% test suite passage with strategic model selection
Clear performance/cost optimization path for future scaling