生产级AI系统的评估与护栏实战指南

构建生产级 AI 系统需要两种互补机制：Evaluation（评估）用于衡量质量与检测问题，Guardrails（护栏）用于控制、阻止或纠正违规行为。将二者混为一谈是常见错误，会导致系统不可靠。

两者区别至关重要：评估告诉你哪里出了问题，护栏防止问题发生。

根本区别

评估用于测量和检测问题：

• 识别 AI 响应中的质量问题
• 衡量相关性、准确性、事实性
• 检测幻觉和合规违规
• 提供系统行为的可观测性
• 用于监控和持续改进

护栏用于控制和阻止行为：

• 防止违规内容进入或离开系统
• 在工具执行前验证参数
• 强制执行业务规则和策略
• 主动纠正或阻止不合规行为
• 用于安全和合规强制执行

最佳实践 #1：定义明确的评估标准

在实现系统之前，建立明确的评估标准：

public class EvaluationCriteria {
    // Relevance: Is response on-topic and appropriate?
    public double evaluateRelevance(String response, String query) {
        // Your relevance scoring logic
    }

    // Factuality: Are claims verifiable and accurate?
    public double evaluateFactuality(String response) {
        // Fact-checking against knowledge base
    }

    // Quality: Does response meet quality standards?
    public double evaluateQuality(String response) {
        // Grammar, coherence, completeness
    }

    // Compliance: Does it follow business rules?
    public boolean evaluateCompliance(String response) {
        // Check against policies
    }
}

最佳实践 #2：使用可信上下文进行评估

尤其是对于 RAG（检索增强生成）解决方案，根据可靠来源评估响应：

public class TrustedContextEvaluator {

    public EvaluationResult evaluateAgainstContext(String response, String trustedContext) {
        EvaluationResult result = new EvaluationResult();

        // Only validate information that has a trusted reference source
        if (!hasReliableSource(response, trustedContext)) {
            result.isFactual = false;
            result.reason = "Response lacks verification in trusted sources";
            return result;
        }

        result.isFactual = true;
        return result;
    }

    private boolean hasReliableSource(String response, String trustedContext) {
        // Check if claims in response can be traced to trusted sources
        return response.matches(".*verified.*source.*");
    }
}

最佳实践 #3：将评估与护栏分离

这是流水线中不同的操作：

// Measures problems
EvaluationResult eval = evaluator.evaluate(response);
if (!eval.meetsQualityThreshold()) {
    logger.warn("Response quality below threshold: " + eval.score);
    // Log for monitoring, but don't necessarily block
}

// Controls behavior
GuardrailResult guard = guardrail.validate(response);
if (!guard.isAllowed()) {
    // BLOCK the response
    return "This response violates our policies.";
}

最佳实践 #4：先应用输入护栏

在发送到模型之前验证格式、内容、意图和约束：

// Validate user input before LLM processing
InputGuardrail.ValidationResult inputCheck = inputGuardrail.validate(userQuery);
if (!inputCheck.isValid) {
    return "Invalid request: " + inputCheck.errorMessage;
}

// Safe to proceed
String response = chatClient.prompt().user(userQuery).call().content();

最佳实践 #5：在执行前应用工具护栏

在 LLM 执行任何外部操作之前验证它：

// LLM decided to call a tool
ToolCall toolCall = extractToolCall(response);

// Guardrail validates BEFORE execution
ToolGuardrail.ValidationResult toolCheck = toolGuardrail.validate(toolCall);
if (!toolCheck.isAllowed) {
    return "Tool execution blocked: " + toolCheck.reason;
}

// Safe to execute
Object result = executeToolSafely(toolCall);

最佳实践 #6：在交付前应用输出护栏

在向用户展示响应之前过滤它：

// LLM generated response
String aiResponse = chatClient.prompt().user(query).call().content();

// Output guardrail filters sensitive content
OutputGuardrail.ValidationResult outputCheck = outputGuardrail.validate(aiResponse);
if (!outputCheck.isSafe) {
    // Return sanitized or filtered version
    return outputCheck.sanitizedContent;
}

// Safe to deliver to user
return aiResponse;

最佳实践 #7：不要只依赖 Prompt

仅靠 Prompt 工程无法强制执行关键规则。你需要代码级控制：

// WRONG: Relying only on prompt
String systemPrompt = "Never reveal passwords or API keys";
// A prompt injection could override this

// RIGHT: Code-level guardrail
private boolean containsSecrets(String content) {
    return content.matches(".*(?i)(password|api_key|secret).*");
}

if (containsSecrets(response)) {
    return "Response blocked: Contains sensitive information";
}

最佳实践 #8：记录和监控评估与护栏结果

维护已评估和已阻止内容的审计跟踪：

private void logEvaluationResult(String query, EvaluationResult result) {
    logger.info("Query: {}; Relevance: {}; Factuality: {}; Quality: {}",
        query, result.relevance, result.factuality, result.quality);
}

private void logGuardrailViolation(String response, String violation) {
    logger.warn("Guardrail violation: {}; Response: {}",
        violation, response);
    // Alert monitoring system
}

最佳实践 #9：定义失败策略

当护栏阻止了某些内容或评估标记了问题时，要有明确的程序：

public String handleGuardrailFailure(String response, String reason) {
    // Strategy 1: Block (default)
    if (reason.contains("sensitive_data")) {
        return "Cannot complete request due to sensitivity";
    }

    // Strategy 2: Correct
    if (reason.contains("format")) {
        return formatCorrectly(response);
    }

    // Strategy 3: Regenerate
    if (reason.contains("quality")) {
        return regenerateResponse();
    }

    // Strategy 4: Escalate
    if (reason.contains("critical")) {
        escalateToHuman();
        return "This requires human review";
    }
}

最佳实践 #10：避免不必要的评估

评估有延迟成本。根据风险策略性地应用它：

// Quick response for low-risk queries
if (isLowRiskQuery(query)) {
    return generateResponse(query);
}

// Full evaluation suite for high-risk domains
if (isHighRiskDomain(query)) {
    EvaluationResult eval = comprehensiveEvaluation(response);
    if (!eval.passes()) {
        return handleFailure(eval);
    }
}

// Mid-tier evaluation for medium-risk
EvaluationResult eval = quickEvaluation(response);

架构模式：三层防御

User Input
    ↓
[INPUT GUARDRAIL] → Validate format, content, intent
    ↓
LLM Processing
    ↓
[TOOL GUARDRAIL] → Validate external operations
    ↓
Output Generation
    ↓
[OUTPUT GUARDRAIL] → Validate before delivery
    ↓
[EVALUATION] → Monitor quality and detect issues
    ↓
User Response (+ Monitoring Data)

生产级 AI 系统需要评估和护栏协同工作。评估让你对质量有感知。护栏让你对安全有控制。

在每个阶段——输入、处理和输出——都实现它们。记录一切。定义明确的失败策略。永远不要假设仅靠 Prompt 就能阻止不想要的行为。

有了评估和护栏，你的 AI 系统就变得值得信赖、可预测、可以投入生产。