可解释LLM实战：用链式推理构建安全审查Agent

今天我们要构建一个自解释的安全审查 Agent，它暴露自身的思维链（chain-of-thought）推理过程和 token 级别的置信度。这为开发者提供了一个实用的窗口，让他们能够理解 LLM 做出某个具体决策的原因——在将任何自动化系统投入生产环境之前，这一点至关重要。

通过 pip install openai 安装 OpenAI SDK

一个来自 https://portal.oxlo.ai 的 Oxlo.ai API key

我在 Oxlo.ai 上运行这个项目，因为它的按请求计费模式即使在我们传递长代码片段和详细推理追踪时也能保持费用稳定。关于当前的套餐详情，请参阅 https://oxlo.ai/pricing。

步骤 1：配置 Oxlo.ai 客户端

首先导入 SDK 并将其指向 Oxlo.ai。这是一个字面意义上的直接替代品，适用于任何 OpenAI 兼容脚本。

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY"),
)

步骤 2：定义推理 Agent 提示词

为了让模型的推理过程可被审查，我强制它使用严格的 XML 标签将内部独白与最终裁决分开。

REASONING_SYSTEM_PROMPT = """You are a security review assistant. Your job is to analyze a user-supplied code snippet and decide if it is Safe, Suspicious, or Vulnerable.

Follow these rules exactly:
1. Think step by step about the code inside <reasoning> tags.
2. State your final verdict inside <verdict> tags using exactly one word: Safe, Suspicious, or Vulnerable.
3. Be concise but thorough.
"""

步骤 3：通过 logprobs 生成决策

我们通过 Oxlo.ai 调用 Kimi K2.6，并启用 logprobs。logprobs 内容将让我们能够检查裁决 token 周围的概率质量。

def get_reasoning_and_verdict(user_code: str):
    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": REASONING_SYSTEM_PROMPT},
            {"role": "user", "content": f"Analyze this code:\n\n\n```python\n{user_code}\n```\n\n"},
        ],
        logprobs=True,
        top_logprobs=5,
        temperature=0.2,
        max_tokens=800,
    )
    return response.choices[0].message.content, response.choices[0].logprobs

code_snippet = "user_input = input('Enter command: ')\neval(user_input)"
raw_output, raw_logprobs = get_reasoning_and_verdict(code_snippet)
print(raw_output)

步骤 4：解析追踪并提取置信度

接下来我们提取推理块，并定位分配给裁决 token 的对数概率。这个数字是核心的可解释性信号。

import re

def parse_explanation(content: str, logprobs_data):
    reasoning_match = re.search(r"<reasoning>(.*?)</reasoning>", content, re.DOTALL)
    verdict_match = re.search(r"<verdict>(.*?)</verdict>", content, re.DOTALL)

    reasoning = reasoning_match.group(1).strip() if reasoning_match else "No reasoning found"
    verdict = verdict_match.group(1).strip() if verdict_match else "Unknown"

    verdict_token = None
    token_prob = None
    if logprobs_data and logprobs_data.content and verdict:
        for token_info in logprobs_data.content:
            if verdict in token_info.token or token_info.token.strip() == verdict.split()[0]:
                verdict_token = token_info.token
                token_prob = token_info.logprob
                break

    return reasoning, verdict, verdict_token, token_prob

reasoning, verdict, v_token, v_prob = parse_explanation(raw_output, raw_logprobs)
print(f"Verdict: {verdict}")
print(f"Token: {v_token}, logprob: {v_prob}")

步骤 5：审计生成人类可读的报告

最后，我们将原始推理过程和置信度分数输入到 Llama 3.3 70B。这第二次调用作为一个可解释性层，将思维链翻译成通俗英语，并标记任何逻辑漏洞。

EXPLAINER_PROMPT = """You are an interpretability analyst. Given a model's internal reasoning chain, its final verdict, and the token-level log probability for that verdict, produce a short report that explains:

1. Why the model likely chose this verdict.
2. How confident the model was (convert the log probability to a percentage).
3. Any assumptions or jumps in logic that a human should double-check.

Keep the report under 150 words.
"""

def explain_decision(reasoning: str, verdict: str, logprob: float):
    prob_pct = round(100 * (2.718 ** logprob), 2) if logprob is not None else "unknown"

    audit_input = f"""Reasoning chain:
{reasoning}

Final verdict: {verdict}
Verdict token log probability: {logprob} (~{prob_pct}% confidence)
"""

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": EXPLAINER_PROMPT},
            {"role": "user", "content": audit_input},
        ],
        temperature=0.3,
        max_tokens=300,
    )
    return response.choices[0].message.content

report = explain_decision(reasoning, verdict, v_prob)
print(report)

下面是完整的脚本。将其保存为 explain_agent.py，设置你的 OXLO_API_KEY 环境变量，然后运行它。

from openai import OpenAI
import os
import re

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY"),
)

REASONING_SYSTEM_PROMPT = """You are a security review assistant. Your job is to analyze a user-supplied code snippet and decide if it is Safe, Suspicious, or Vulnerable.

Follow these rules exactly:
1. Think step by step about the code inside <reasoning> tags.
2. State your final verdict inside <verdict> tags using exactly one word: Safe, Suspicious, or Vulnerable.
3. Be concise but thorough.
"""

EXPLAINER_PROMPT = """You are an interpretability analyst. Given a model's internal reasoning chain, its final verdict, and the token-level log probability for that verdict, produce a short report that explains:

1. Why the model likely chose this verdict.
2. How confident the model was (convert the log probability to a percentage).
3. Any assumptions or jumps in logic that a human should double-check.

Keep the report under 150 words.
"""

def get_reasoning_and_verdict(user_code: str):
    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": REASONING_SYSTEM_PROMPT},
            {"role": "user", "content": f"Analyze this code:\n\n\n```python\n{user_code}\n```\n\n"},
        ],
        logprobs=True,
        top_logprobs=5,
        temperature=0.2,
        max_tokens=800,
    )
    return response.choices[0].message.content, response.choices[0].logprobs

def parse_explanation(content: str, logprobs_data):
    reasoning_match = re.search(r"<reasoning>(.*?)</reasoning>", content, re.DOTALL)
    verdict_match = re.search(r"<verdict>(.*?)</verdict>", content, re.DOTALL)
    reasoning = reasoning_match.group(1).strip() if reasoning_match else "No reasoning found"
    verdict = verdict_match.group(1).strip() if verdict_match else "Unknown"

    verdict_token = None
    token_prob = None
    if logprobs_data and logprobs_data.content and verdict:
        for token_info in logprobs_data.content:
            if verdict in token_info.token or token_info.token.strip() == verdict.split()[0]:
                verdict_token = token_info.token
                token_prob = token_info.logprob
                break
    return reasoning, verdict, verdict_token, token_prob

def explain_decision(reasoning: str, verdict: str, logprob: float):
    prob_pct = round(100 * (2.718 ** logprob), 2) if logprob is not None else "unknown"
    audit_input = f"""Reasoning chain:
{reasoning}

Final verdict: {verdict}
Verdict token log probability: {logprob} (~{prob_pct}% confidence)
"""
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": EXPLAINER_PROMPT},
            {"role": "user", "content": audit_input},
        ],
        temperature=0.3,
        max_tokens=300,
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    code_snippet = "user_input = input('Enter command: ')\neval(user_input)"
    raw_output, raw_logprobs = get_reasoning_and_verdict(code_snippet)
    reasoning, verdict, v_token, v_prob = parse_explanation(raw_output, raw_logprobs)
    report = explain_decision(reasoning, verdict, v_prob)

    print("=== RAW OUTPUT ===")
    print(raw_output)
    print("\n=== PARSED VERDICT ===")
    print(f"{verdict} (token: {v_token}, logprob: {v_prob})")
    print("\n=== INTERPRETABILITY REPORT ===")
    print(report)

=== RAW OUTPUT ===

The code reads untrusted user input via input() and passes it directly to eval(). This allows arbitrary code execution, which is a critical security vulnerability.

<verdict>Vulnerable</verdict>

=== PARSED VERDICT ===
Vulnerable (token: Vulnerable, logprob: -0.0423)

=== INTERPRETABILITY REPORT ===
The model flagged the direct use of eval() on raw user input as a critical vulnerability. Its confidence is high at approximately 95.8%. The reasoning is sound, though it assumes the input source is truly untrusted. If this runs in a sandboxed environment, a human reviewer might downgrade the severity.

将推理步骤中的模型换成 deepseek-v3.2 或 qwen-3-32b，可以比较不同模型如何为同一个裁决提供依据。你也可以将其扩展为一个小型的 Web 服务，在推理文本上以热力图形式渲染 token 概率。