lemony-ai/cascadeflow

---

The in-process intelligence layer for AI agents. Optimize cost, latency, quality, budget, compliance, and energy — inside the execution loop, not at the HTTP boundary.

cascadeflow works where external proxies can't: per-step model decisions based on agent state, per-tool-call budget gating, runtime stop/continue/escalate actions, and business KPI injection during agent loops. It accumulates insight from every model call, tool result, and quality score — the agent gets smarter the more it runs. Sub-5ms overhead. Works with LangChain, OpenAI Agents SDK, CrewAI, PydanticAI, Google ADK, n8n, Vercel AI SDK, and Hermes Agent.

Update

### Hermes Agent delegation cascading

> CascadeFlow now provides a Hermes Agent integration for per-skill model cascading, task-complexity cascading, topic-aware subagent cascading, observe-mode rollout, and auditable decisions without taking over provider credentials, base URLs, fallback chains, or API modes.

pip install cascadeflow

npm install @cascadeflow/core

---

Why cascadeflow?

Proxy vs In-Process Harness

| Dimension | External Proxy | cascadeflow Harness | |---|---|---| | Scope | HTTP request boundary | Inside agent execution loop | | Dimensions | Cost only | Cost + quality + latency + budget + compliance + energy | | Latency overhead | 10-50ms network RTT | <5ms in-process | | Business logic | None | KPI weights and targets | | Enforcement | None (observe only) | stop, deny_tool, switch_model | | Auditability | Request logs | Per-step decision traces |

cascadeflow is a library and agent harness — an intelligent AI model cascading package that dynamically selects the optimal model for each query or tool call through speculative execution. It's based on the research that 40-70% of queries don't require slow, expensive flagship models, and domain-specific smaller models often outperform large general-purpose models on specialized tasks. For the remaining queries that need advanced reasoning, cascadeflow automatically escalates to flagship models if needed.

---

How cascadeflow Works

cascadeflow uses speculative execution with quality validation:

1. Speculatively executes small, fast models first - optimistic execution ($0.15-0.30/1M tokens) 2. Validates quality of responses using configurable thresholds (completeness, confidence, correctness) 3. Dynamically escalates to larger models only when quality validation fails ($1.25-3.00/1M tokens) 4. Learns patterns to optimize future cascading decisions and domain specific routing

Zero configuration. Works with YOUR existing models (>17 providers currently supported).

In practice, 60-70% of queries are handled by small, efficient models (8-20x cost difference) without requiring escalation

Result: 40-85% cost reduction, 2-10x faster responses, zero quality loss.

┌─────────────────────────────────────────────────────────────┐
│                      cascadeflow Stack                      │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  Cascade Agent                                        │  │
│  │                                                       │  │
│  │  Orchestrates the entire cascade execution            │  │
│  │  • Query routing & model selection                    │  │
│  │  • Drafter -> Verifier coordination                   │  │
│  │  • Cost tracking & telemetry                          │  │
│  └───────────────────────────────────────────────────────┘  │
│                          ↓                                  │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  Domain Pipeline                                      │  │
│  │                                                       │  │
│  │  Automatic domain classification                      │  │
│  │  • Rule-based detection (CODE, MATH, DATA, etc.)      │  │
│  │  • Optional ML semantic classification                │  │
│  │  • Domain-optimized pipelines & model selection       │  │
│  └───────────────────────────────────────────────────────┘  │
│                          ↓                                  │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  Quality Validation Engine                            │  │
│  │                                                       │  │
│  │  Multi-dimensional quality checks                     │  │
│  │  • Length validation (too short/verbose)              │  │
│  │  • Confidence scoring (logprobs analysis)             │  │
│  │  • Format validation (JSON, structured output)        │  │
│  │  • Semantic alignment (intent matching)               │  │
│  └───────────────────────────────────────────────────────┘  │
│                          ↓                                  │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  Cascading Engine (<2ms overhead)                     │  │
│  │                                                       │  │
│  │  Smart model escalation strategy                      │  │
│  │  • Try cheap models first (speculative execution)     │  │
│  │  • Validate quality instantly                         │  │
│  │  • Escalate only when needed                          │  │
│  │  • Automatic retry & fallback                         │  │
│  └───────────────────────────────────────────────────────┘  │
│                          ↓                                  │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  Provider Abstraction Layer                           │  │
│  │                                                       │  │
│  │  Unified interface for >17 providers                   │  │
│  │  • OpenAI • Anthropic • Groq • Ollama                 │  │
│  │  • Together • vLLM • HuggingFace • LiteLLM            │  │
│  │  • Vercel AI SDK (17+ additional providers)            │  │
│  └───────────────────────────────────────────────────────┘  │
│                                                             │
└─────────────────────────────────────────────────────────────┘

---

Harness API

Three tiers of integration — zero-change observability to full policy control:

Tier 1: Zero-change observability

import cascadeflow
cascadeflow.init(mode="observe")
All OpenAI/Anthropic SDK calls are now tracked. No code changes needed.

Tier 2: Scoped runs with budget

with cascadeflow.run(budget=0.50, max_tool_calls=10) as session:
    result = await agent.run("Analyze this dataset")
    print(session.summary())  # cost, latency, energy, steps, tool calls
    print(session.trace())    # full decision audit trail

Tier 3: Decorated agents with policy

@cascadeflow.agent(budget=0.20, compliance="gdpr", kpi_weights={"quality": 0.6, "cost": 0.3, "latency": 0.1})
async def my_agent(query: str):
    return await llm.complete(query)

---

Quick Start

Python

pip install cascadeflow[all]

from cascadeflow import CascadeAgent, ModelConfig
Define your cascade - try cheap model first, escalate if needed
agent = CascadeAgent(models=[
    ModelConfig(name="nous/hermes-flash", provider="openai", cost=0.000375),  # Draft model (~$0.375/1M tokens)
    ModelConfig(name="gpt-5", provider="openai", cost=0.00562),         # Verifier model (~$5.62/1M tokens)
])
Run query - automatically routes to optimal model
result = await agent.run("What's the capital of France?")
print(f"Answer: {result.content}")
print(f"Model used: {result.model_used}")
print(f"Cost: ${result.total_cost:.6f}")

pip install cascadeflow[semantic]  # Adds semantic similarity via FastEmbed (~80MB model)

from cascadeflow.quality.semantic import SemanticQualityChecker
Initialize semantic checker (downloads model on first use)
checker = SemanticQualityChecker(
    similarity_threshold=0.5,  # Minimum similarity score (0-1)
    toxicity_threshold=0.7     # Maximum toxicity score (0-1)
)
Validate query-response alignment
query = "Explain Python decorators"
response = "Decorators are a way to modify functions using @syntax..."
result = checker.validate(query, response, check_toxicity=True)
print(f"Similarity: {result.similarity:.2%}")
print(f"Passed: {result.passed}")
print(f"Toxic: {result.is_toxic}")

⚠️ GPT-5 Note: GPT-5 streaming requires organization verification. Non-streaming works for all users. Verify here if needed (~15 min). Basic cascadeflow examples work without - GPT-5 is only called when needed (typically 20-30% of requests).

📖 Learn more: Python Documentation | Quickstart Guide | Providers Guide

TypeScript

npm install @cascadeflow/core

import { CascadeAgent, ModelConfig } from '@cascadeflow/core';
// Same API as Python!
const agent = new CascadeAgent({
  models: [
    { name: 'nous/hermes-flash', provider: 'openai', cost: 0.000375 },
    { name: 'gpt-4o', provider: 'openai', cost: 0.00625 },
  ],
});
const result = await agent.run('What is TypeScript?');
console.log(Model: ${result.modelUsed});
console.log(Cost: $${result.totalCost});
console.log(Saved: ${result.savingsPercentage}%);

npm install @cascadeflow/ml @huggingface/transformers

import { CascadeAgent, SemanticQualityChecker } from '@cascadeflow/core';
const agent = new CascadeAgent({
  models: [
    { name: 'nous/hermes-flash', provider: 'openai', cost: 0.000375 },
    { name: 'gpt-4o', provider: 'openai', cost: 0.00625 },
  ],
  quality: {
    threshold: 0.40,                    // Traditional confidence threshold
    requireMinimumTokens: 5,            // Minimum response length
    useSemanticValidation: true,        // Enable ML validation
    semanticThreshold: 0.5,             // 50% minimum similarity
  },
});
// Responses now validated for semantic alignment
const result = await agent.run('Explain TypeScript generics');

import { SemanticQualityChecker } from '@cascadeflow/core';
const checker = new SemanticQualityChecker();
if (await checker.isAvailable()) {
  const result = await checker.checkSimilarity(
    'What is TypeScript?',
    'TypeScript is a typed superset of JavaScript.'
  );
console.log(Similarity: ${(result.similarity * 100).toFixed(1)}%);
  console.log(Passed: ${result.passed});
}

📖 Learn more: TypeScript Documentation | Quickstart Guide | Node.js Examples

🔄 Migration Example

Migrate in 5min from direct Provider implementation to cost savings and full cost control and transparency.

Before (Standard Approach)

Cost: $0.000113, Latency: 850ms

# Using expensive model for everything
result = openai.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What's 2+2?"}]
)

After (With cascadeflow)

Cost: $0.000007, Latency: 234ms

agent = CascadeAgent(models=[
    ModelConfig(name="nous/hermes-flash", provider="openai", cost=0.000375),
    ModelConfig(name="gpt-4o", provider="openai", cost=0.00625),
])
result = await agent.run("What's 2+2?")

🔥 Saved: $0.000106 (94% reduction), 3.6x faster

📊 Learn more: Cost Tracking Guide | Production Best Practices | Performance Optimization

python -m cascadeflow.server --mode auto --port 8084

---

n8n Integration

Use cascadeflow in n8n workflows for no-code AI automation with automatic cost optimization!

Installation

1. Open n8n 2. Go to Settings → Community Nodes 3. Search for: @cascadeflow/n8n-nodes-cascadeflow 4. Click Install

Two Nodes

| Node | Type | Use case | |------|------|----------| | CascadeFlow (Model) | Language Model sub-node | Drop-in for any Chain/LLM node | | CascadeFlow Agent | Standalone agent (main in/out) | Tool calling, memory, multi-step reasoning |

Quick Start (Model): 1. Add two AI Chat Model nodes (cheap drafter + powerful verifier) 2. Add CascadeFlow (Model) and connect both models 3. Connect to Basic LLM Chain or Chain node 4. Check Logs tab on the Chain node to see cascade decisions

Quick Start (Agent): 1. Add a Chat Trigger node 2. Add CascadeFlow Agent and connect it to the trigger 3. Connect Drafter, Verifier, optional Memory and Tools 4. Check the Agent Output tab for cascade metadata and trace

Result: 40-85% cost savings in your n8n workflows!

Features:

• Works with any AI Chat Model node (OpenAI, Anthropic, Ollama, Azure, etc.)
• Mix providers (e.g., Ollama drafter + GPT-4o verifier)
• Agent node: tool calling, memory, per-tool routing, tool call validation
• 16-domain cascading for specialized model routing
• Real-time flow visualization in Logs/Output tabs

🔌 Learn more: n8n Integration Guide | n8n Package

---