分析LLM输出不确定性的开源工具

生成式 AI 工具包：自动化可解释性、错误缓解与多模态支持
📊 07/03 更新：使用 entropy + judge LLM 进行 hallucination 检测的基准测试，准确率超过 80%！
🖼️ 12/02 更新：支持集成 VLM 与视觉注意力监控
🐳 08/02 更新：支持分析推理模型的 CoT entropy，并改进 RL 数据集

Klarity 是一套用于检查和调试 AI 决策过程的工具包。它将不确定性分析、推理洞察与视觉注意力模式结合起来，帮助你理解模型是如何思考的，并在问题进入生产环境之前将其修复。

• 双重 Entropy 分析：通过原始 entropy 和语义相似度指标衡量模型的置信度
• 推理分析：提取并评估模型输出中逐步展开的思考模式
• 视觉注意力分析：可视化并分析视觉语言模型如何关注图像
• 语义聚类：将相似的预测归为一组，揭示决策路径
• 结构化洞察：获取关于不确定性模式与推理步骤的详细 JSON 分析
• AI 驱动的报告：利用能力出色的模型解读生成模式，并提供人类可读的洞察

VLM 分析示例——深入了解模型关注的位置，并检查相关 token 的不确定性。

推理分析示例——理解模型逐步展开的思考过程。

Entropy 分析示例——分析 token 级别的不确定性模式。

我们最新的基准测试展示了 Klarity 的 entropy + judge LLM 指标与标准方法相比，如何显著提升 hallucination 检测效果。结果表明，它在不同规模、不同类型的模型上都展现出了更出色的性能。你可以在 examples 文件夹中找到我们的测试 notebook。

Hallucination 检测基准测试——在 hallucination 检测数据集上测试基于 entropy + judge LLM 的指标。

🚀 快速开始：Hugging Face

直接从 GitHub 安装：

pip install git+https://github.com/klara-research/klarity.git

🖼️ VLM 分析使用示例

如果想了解模型正在关注哪些位置，并分析相关 token 的不确定性，可以使用 VLMAnalyzer：

from transformers import AutoProcessor, LlavaOnevisionForConditionalGeneration, LogitsProcessorList
from PIL import Image
import torch
from klarity import UncertaintyEstimator
from klarity.core.analyzer import EnhancedVLMAnalyzer
import os
import json

os.environ["TOKENIZERS_PARALLELISM"] = "false"

# Initialize VLM model
model_id = "llava-hf/llava-onevision-qwen2-0.5b-ov-hf"
model = LlavaOnevisionForConditionalGeneration.from_pretrained(
    model_id,
    output_attentions=True,
    low_cpu_mem_usage=True
)

processor = AutoProcessor.from_pretrained(model_id)

# Create estimator with EnhancedVLMAnalyzer
estimator = UncertaintyEstimator(
    top_k=100,
    analyzer=EnhancedVLMAnalyzer(
        min_token_prob=0.01,
        insight_model="together:meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo",
        insight_api_key="your_api_key",
        vision_config=model.config.vision_config,
        use_cls_token=True
    ),
)

uncertainty_processor = estimator.get_logits_processor()

# Set up generation for the example
image_path = "examples/images/plane.jpg"
question = "How many engines does the plane have?"
image = Image.open(image_path)

# Prepare input with image and text
conversation = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": question},
            {"type": "image"}
        ]
    }
]
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)

# Process inputs
inputs = processor(
    images=image,
    text=prompt,
    return_tensors='pt'
)

try:
    # Generate with uncertainty and attention analysis
    generation_output = model.generate(
        **inputs,
        max_new_tokens=200,
        temperature=0.7,
        top_p=0.9,
        do_sample=True,
        logits_processor=LogitsProcessorList([uncertainty_processor]),
        return_dict_in_generate=True,
        output_scores=True,
        output_attentions=True,
        use_cache=True
    )

    # Analyze the generation - now includes both images and enhanced analysis
    result = estimator.analyze_generation(
        generation_output=generation_output,
        model=model,
        tokenizer=processor,
        processor=uncertainty_processor,
        prompt=question,
        image=image  # Image is required for enhanced analysis
    )

    # Get generated text
    input_length = inputs.input_ids.shape[1]
    generated_sequence = generation_output.sequences[0][input_length:]
    generated_text = processor.decode(generated_sequence, skip_special_tokens=True)

    print(f"\nQuestion: {question}")
    print(f"Generated answer: {generated_text}")

    # Token Analysis
    print("\nDetailed Token Analysis:")
    for idx, metrics in enumerate(result.token_metrics):
        print(f"\nStep {idx}:")
        print(f"Raw entropy: {metrics.raw_entropy:.4f}")
        print(f"Semantic entropy: {metrics.semantic_entropy:.4f}")
        print("Top 3 predictions:")
        for i, pred in enumerate(metrics.token_predictions[:3], 1):
            print(f"  {i}. {pred.token} (prob: {pred.probability:.4f})")

    # Show comprehensive insight
    print("\nComprehensive Analysis:")
    print(json.dumps(result.overall_insight, indent=2))

except Exception as e:
    print(f"Error during generation: {str(e)}")
    import traceback
    traceback.print_exc()

📝 推理 LLM 使用示例

如果想获得关于模型推理模式的洞察与不确定性分析，可以使用 ReasoningAnalyzer：

from transformers import AutoModelForCausalLM, AutoTokenizer, LogitsProcessorList
from klarity import UncertaintyEstimator
from klarity.core.analyzer import ReasoningAnalyzer
import torch

# Initialize model with GPU support
device = "cuda" if torch.cuda.is_available() else "cpu"

model_name = "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"
model = AutoModelForCausalLM.from_pretrained(model_name)
model = model.to(device)  # Move model to GPU
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Create estimator with reasoning analyzer and togetherai hosted model
estimator = UncertaintyEstimator(
    top_k=100,
    analyzer=ReasoningAnalyzer(
        min_token_prob=0.01,
        insight_model="together:meta-llama/Llama-3.3-70B-Instruct-Turbo",
        insight_api_key="your_api_key",
        reasoning_start_token="<think>", 
        reasoning_end_token="</think>"
    )
)
uncertainty_processor = estimator.get_logits_processor()

# Set up generation
prompt = "Your prompt <think>\n"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

# Generate with uncertainty analysis
generation_output = model.generate(
    **inputs,
    max_new_tokens=200,  # Increased for reasoning steps
    temperature=0.6,
    logits_processor=LogitsProcessorList([uncertainty_processor]),
    return_dict_in_generate=True,
    output_scores=True,
)

# Analyze the generation
result = estimator.analyze_generation(
    generation_output,
    tokenizer,
    uncertainty_processor,
    prompt  # Include prompt for better reasoning analysis
)

# Get generated text
generated_text = tokenizer.decode(generation_output.sequences[0], skip_special_tokens=True)
print(f"\nPrompt: {prompt}")
print(f"Generated text: {generated_text}")

# Show reasoning analysis
print("\nReasoning Analysis:")
if result.overall_insight and "reasoning_analysis" in result.overall_insight:
    analysis = result.overall_insight["reasoning_analysis"]
    
    # Print each reasoning step
    for idx, step in enumerate(analysis["steps"], 1):
        print(f"\nStep {idx}:")  # Use simple counter instead of accessing step_number
        print(f"Content: {step['step_info']['content']}")
        
        # Print step analysis
        if 'analysis' in step and 'training_insights' in step['analysis']:
            step_analysis = step['analysis']['training_insights']
            print("\nQuality Metrics:")
            for metric, score in step_analysis['step_quality'].items():
                print(f"  {metric}: {score}")
            
            print("\nImprovement Targets:")
            for target in step_analysis['improvement_targets']:
                print(f"  Aspect: {target['aspect']}")
                print(f"  Importance: {target['importance']}")
                print(f"  Issue: {target['current_issue']}")
                print(f"  Suggestion: {target['training_suggestion']}")

📝 标准 LLM 使用示例

为了规避大多数常见的不确定性场景，并将请求路由到更合适的模型，你可以使用我们的 EntropyAnalyzer：

from transformers import AutoModelForCausalLM, AutoTokenizer, LogitsProcessorList
from klarity import UncertaintyEstimator
from klarity.core.analyzer import EntropyAnalyzer

# Initialize your model
model_name = "Qwen/Qwen2.5-7B-Instruct"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Create estimator
estimator = UncertaintyEstimator(
    top_k=100,
    analyzer=EntropyAnalyzer(
        min_token_prob=0.01,
        insight_model=model,
        insight_tokenizer=tokenizer
    )
)

uncertainty_processor = estimator.get_logits_processor()

# Set up generation
prompt = "Your prompt"
inputs = tokenizer(prompt, return_tensors="pt")

# Generate with uncertainty analysis
generation_output = model.generate(
    **inputs,
    max_new_tokens=20,
    temperature=0.7,
    top_p=0.9,
    logits_processor=LogitsProcessorList([uncertainty_processor]),
    return_dict_in_generate=True,
    output_scores=True,
)

# Analyze the generation
result = estimator.analyze_generation(
    generation_output,
    tokenizer,
    uncertainty_processor
)

generated_text = tokenizer.decode(generation_output.sequences[0], skip_special_tokens=True)

# Inspect results
print(f"\nPrompt: {prompt}")
print(f"Generated text: {generated_text}")

print("\nDetailed Token Analysis:")
for idx, metrics in enumerate(result.token_metrics):
    print(f"\nStep {idx}:")
    print(f"Raw entropy: {metrics.raw_entropy:.4f}")
    print(f"Semantic entropy: {metrics.semantic_entropy:.4f}")
    print("Top 3 predictions:")
    for i, pred in enumerate(metrics.token_predictions[:3], 1):
        print(f"  {i}. {pred.token} (prob: {pred.probability:.4f})")

# Show comprehensive insight
print("\nComprehensive Analysis:")
print(result.overall_insight)

Klarity 提供三类分析输出：

模型关注位置及相关 token 不确定性的注意力洞察：

{
    "scores": {
        "overall_uncertainty": "<0-1>",
        "visual_grounding": "<0-1>",  
        "confidence": "<0-1>"         
    },
    "visual_analysis": {
        "attention_quality": {     
            "score": "<0-1>",
            "key_regions": ["<main area 1>", "<main area 2>"],
            "missed_regions": ["<ignored area 1>", "<ignored area 2>"]
        },
        "token_attention_alignment": [  
            {
                "word": "<token>",
                "focused_spot": "<region>",
                "relevance": "<0-1>",   
                "uncertainty": "<0-1>"  
            }
        ]
    }},
    "uncertainty_analysis": {
        "problem_spots": [  
            {
                "text": "<text part>",
                "reason": "<why uncertain>",
                "looked_at": "<image area>",
                "connection": "<focus vs doubt link>"
            }
        ],
        "improvement_tips": [ 
            {
                "area": "<what to fix>",
                "tip": "<how to fix>"
            }
        ]
    }
}

你将获得关于模型推理过程的详细洞察：

{
    "reasoning_analysis": {
        "steps": [
            {
                "step_number": 1,
                "step_info": {
                    "content": "",
                    "type": ""
                },
                "analysis": {
                    "training_insights": {
                        "step_quality": {
                            "coherence": "<0-1>",
                            "relevance": "<0-1>",
                            "confidence": "<0-1>"
                        },
                        "improvement_targets": [
                            {
                                "aspect": "",
                                "importance": "<0-1>",
                                "current_issue": "",
                                "training_suggestion": ""
                            }
                        ]
                    }
                }
            }
        ]
    }
}

对于标准语言模型，你将获得一份综合性的不确定性报告：

{
    "scores": {
        "overall_uncertainty": "<0-1>",  
        "confidence_score": "<0-1>",     
        "hallucination_risk": "<0-1>"    
    },
    "uncertainty_analysis": {
        "high_uncertainty_parts": [
            {
                "text": "",
                "why": ""
            }
        ],
        "main_issues": [
            {
                "issue": "",
                "evidence": ""
            }
        ],
        "key_suggestions": [
            {
                "what": "",
                "how": ""
            }
        ]
    }
}

🤖 支持的 Framework 与模型

✅ Hugging Face Transformers → 完整的不确定性分析，包括原始 entropy、语义 entropy 指标与视觉注意力监控

✅ Hugging Face Transformers → 完整的不确定性分析，包括原始 entropy、语义 entropy 指标与视觉注意力监控

✅ vLLM → 完整的不确定性分析，包括原始 entropy 和语义 entropy 指标，每个 token 最多支持 20 个 logprobs

✅ vLLM → 完整的不确定性分析，包括原始 entropy 和语义 entropy 指标，每个 token 最多支持 20 个 logprobs

✅ Together AI → 使用原始 log prob. 指标进行不确定性分析

✅ Together AI → 使用原始 log prob. 指标进行不确定性分析

洞察所用 Analysis Model 支持的 Framework

✅ Hugging Face Transformers

JSON 输出可靠性指南

✅ 高：能够稳定输出有效 JSON（超过 80% 的响应）

⚠️ 中：通常能够输出有效 JSON（50%～80% 的响应）

⚡ 低：JSON 输出不稳定（低于 50% 的响应）

自定义分析配置

你可以自定义分析参数：

analyzer = EntropyAnalyzer(
    min_token_prob=0.01,  # Minimum probability threshold
    semantic_similarity_threshold=0.8  # Threshold for semantic grouping
)

欢迎贡献！我们希望改进的方向包括：

• 支持更多 Framework
• 增强语义分析
• 增加更多分析指标
• 完善文档和示例

详情请参阅我们的贡献指南。

采用 Apache 2.0 License。更多信息请参阅 LICENSE。

📫 社区与支持

如需报告 bug 或提出功能建议，请使用 GitHub Issues。