Swarm：纯Rust实现的多Agent编排与LLM网关框架

While experimenting with multi-agent systems, I kept ending up with two separate pieces of infrastructure: an orchestration layer for agents and tools, and a gateway layer for routing LLM requests.

我在实验多 Agent 系统时，反复遇到同一个问题：需要维护两套独立的基础设施——一套负责 Agent 和工具的编排，另一套负责 LLM 请求的路由。

I wanted both to share the same runtime, provider abstractions, state management, and protocol contracts.

我希望两者能够共享同一个运行时、相同的 provider 抽象层、状态管理机制以及协议契约。

So I built Swarm, an open-source AI orchestration framework and model gateway written in Rust.

于是我构建了 Swarm，一个纯 Rust 编写的开源 AI 编排框架和模型网关。

The Dual-Mode Architecture

双模架构

Many AI stacks end up separating these concerns: a dedicated proxy for lightweight routing and a separate orchestrator for more complex reasoning. Swarm unifies both patterns around a single high-performance Tokio runtime.

多数 AI 技术栈最终都将这些职责拆分开：一套专用代理负责轻量级路由，另一套独立的编排器处理复杂推理。Swarm 围绕一个高性能的 Tokio 运行时统一了两种模式。

+--------------------------------------------------------------------------------------------------+
|                                        SWARM MODES                                               |
+--------------------------------------------------------------------------------------------------+
|                                                                                                  |
|   MODE 1: MULTI-AGENT & MCP ORCHESTRATION               MODE 2: MODEL GATEWAY SERVER             |
|   (kickstart/multi_agent_orchestration_kickstart/)      (kickstart/gateway_kickstart/)           |
|                                                                                                  |
|   • Planner Agent (Dynamic plan generation)             • POST /v1/chat/completions (OpenAI)     |
|   • Executor Agent (Workflow DAG execution)             • POST /v1/responses (Open Responses)    |
|   • Domain Specialists with MCP Tool integration        • Stateful multi-turn chaining          |
|   • Discovery & Memory services                         • Multi-provider (Groq, Gemini, OpenAI,  |
|   • Evaluation & Judge Service                            Ollama / vLLM / local endpoints)       |
|   • Resilient OAuth2 / JWT authentication               • High-throughput lock-free cache        |
|                                                                                                  |
+--------------------------------------------------------------------------------------------------+

The key idea: Swarm can run as a full agent orchestration stack or as a standalone LLM gateway without requiring two unrelated frameworks.

核心设计理念：Swarm 既可以作为完整的 Agent 编排技术栈运行，也可以作为独立的 LLM 网关运行，无需引入两套互不相关的框架。

Mode 1: Deterministic Multi-Agent Orchestration with MCP

模式一：基于 MCP 的确定性多 Agent 编排

Coordinating multiple agents becomes much easier when service boundaries and message contracts are explicit.

当服务边界和消息契约明确之后，多 Agent 协作会变得简单许多。

Mode 1 splits responsibilities across decoupled, specialized services:

模式一将职责拆分到多个解耦的专门服务中：

Planner Agent：Analyzes incoming user requests and dynamically constructs execution DAGs.

Planner Agent：分析用户的传入请求，动态构建执行 DAG。

Executor Agent：Resolves task dependencies and controls step execution.

Executor Agent：解析任务依赖，控制步骤执行。

Domain Specialists：Execute live tools via a native Model Context Protocol (MCP) runtime (supporting SSE and streaming tool calls).

Domain Specialists：通过原生 MCP 运行时执行实时工具（支持 SSE 和流式工具调用）。

Discovery & Memory：Maintain service registries and conversational state.

Discovery & Memory：维护服务注册表和对话状态。

Evaluation Service：Built-in LLM-as-a-Judge validation loop for output verification and self-correction.

Evaluation Service：内置的 LLM-as-a-Judge 验证循环，用于输出验证和自我修正。

Inter-agent communication relies on type-safe agent-to-agent (A2A) message contracts, catching many contract and integration errors during development and compilation.

Agent 间通信依赖类型安全的 A2A 消息契约，在开发阶段和编译期就能捕获大量契约和集成错误。

User Request
    ↓
Planner
    ↓
Execution DAG
    ↓
Executor
    ↓
Weather Agent
    ↓
MCP Weather Tool
    ↓
Evaluation
    ↓
Final Response

Mode 2: OpenAI-Compatible Model Gateway

模式二：OpenAI 兼容的模型网关

Mode 2 exposes an OpenAI-compatible gateway for client applications, developer tools, and automated pipelines.

模式二对外暴露一个 OpenAI 兼容的网关，供客户端应用、开发者工具和自动化流水线使用。

OpenAI Compatibility (POST /v1/chat/completions)：Works with standard OpenAI SDKs, Cursor, and developer extensions.

OpenAI 兼容性 (POST /v1/chat/completions)：兼容标准 OpenAI SDK、Cursor 以及各类开发者扩展。

Stateful Responses (POST /v1/responses)：Supports multi-turn conversation chaining using explicit previous_response_id references.

有状态的 Responses (POST /v1/responses)：支持通过显式的 previous_response_id 引用进行多轮对话链式编排。

Unified Multi-Provider Routing：Route requests across Groq, Google Gemini, OpenAI, or local backends such as Ollama, vLLM, and llama.cpp through TOML configuration.

统一多 Provider 路由：通过 TOML 配置将请求路由至 Groq、Google Gemini、OpenAI 或 Ollama、vLLM、llama.cpp 等本地后端。

[server]
bind_address = "0.0.0.0:8080"
log_level = "info"

[models]
default_model = "openai/gpt-oss-20b"

[providers.groq]
api_url = "https://api.groq.com/openai/v1/chat/completions"

[providers.google]
api_url = "https://generativelanguage.googleapis.com/v1beta/models"

[providers.custom]
# Local inference (Ollama / vLLM / llama.cpp / LocalAI)
api_url = "http://localhost:11434/v1/chat/completions"
recommended_models = ["llama3.2:latest", "mistral:latest", "deepseek-r1:8b"]

Rust gives Swarm a few useful properties for orchestration and gateway workloads:

Rust 为 Swarm 的编排和网关工作负载赋予了若干实用特性：

⚡ Low-Overhead Request Handling：Built on Tokio and Hyper for asynchronous, high-concurrency gateway workloads.

⚡ 低开销请求处理：基于 Tokio 和 Hyper 构建，适用于异步高并发网关工作负载。

🔒 Concurrent Session Management：DashMap and Arc-based stores allow shared state across concurrent requests without a global application lock.

🔒 并发会话管理：DashMap 和基于 Arc 的存储支持跨并发请求共享状态，无需全局应用锁。

🪶 Small Runtime Footprint：No garbage collector and predictable memory ownership make the gateway suitable for lightweight deployments.

🪶 小运行时占用：无垃圾回收器且内存所有权可预测，使网关非常适合轻量级部署。

🛡️ Strongly Typed Protocols：Internal MCP and A2A message contracts reduce schema mismatches and integration errors.

🛡️ 强类型协议：内部 MCP 和 A2A 消息契约减少了模式不匹配和集成错误。

You can test either mode locally in minutes:

你可以在几分钟内在本地测试任一模式：

git clone https://github.com/fcn06/swarm.git
cd swarm

# Configure your provider keys
cp .env.example .env

Option A: Launch the Model Gateway (Mode 2)

选项 A：启动模型网关（模式二）

./kickstart/gateway_kickstart/01_launch_gateway.sh

# Test OpenAI-compatible completions
curl -X POST http://localhost:8080/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "openai/gpt-oss-20b",
    "messages": [{"role": "user", "content": "Explain Swarm architecture in 2 sentences."}]
  }'

Option B: Launch Multi-Agent MCP Suite (Mode 1)

选项 B：启动多 Agent MCP 套件（模式一）

# Launch agents, MCP server, and infrastructure services
./kickstart/multi_agent_orchestration_kickstart/01_launch_all.sh

# Run a live MCP tool test query
./kickstart/multi_agent_orchestration_kickstart/02_test_weather_query.sh "What is the current weather in Boston ?"

Open Source & Community

开源与社区

Swarm is fully open-source under the Apache-2.0 license. We rely on and contribute back to the emerging Rust AI ecosystem, including the official MCP Rust SDK and A2A Protocol.

Swarm 基于 Apache-2.0 许可证完全开源。我们依赖并回馈新兴的 Rust AI 生态，包括官方的 MCP Rust SDK 和 A2A Protocol。

I'm especially interested in feedback from people running agent systems or LLM gateways in production:

我特别期待来自在生产环境中运行 Agent 系统或 LLM 网关的朋友们的反馈：

• Would you rather deploy the orchestration and gateway as one runtime, or keep them completely separate?

你更倾向于将编排器和网关作为同一个运行时部署，还是完全分开？

• If you try Swarm, I'd also love feedback on the MCP runtime, gateway compatibility, and APIs.

如果你尝试了 Swarm，我也非常欢迎关于 MCP 运行时、网关兼容性以及 API 的反馈。