通过 Prompt 压缩加速 LLM 推理

LLMLingua 系列｜通过 prompt 压缩，将信息高效传递给 LLM

项目主页	LLMLingua	LongLLMLingua	LLMLingua-2	LLMLingua Demo	LLMLingua-2 Demo

🍩 [24/12/13] 我们很高兴宣布发布以 KV cache 为核心的分析工作 SCBench。它从 KV cache 的视角评估长上下文方法。

👘 [24/09/16] 我们很高兴宣布发布 KV cache 卸载工作 RetrievalAttention。它通过向量检索，加速长上下文 LLM 的推理。

🌀 [24/07/03] 我们很高兴宣布发布 MInference，用于加速长上下文 LLM 的推理。在 A100 上进行预填充时，它最多可将推理延迟降低 10 倍，同时在包含 100 万 token 的 prompt 中保持准确率！如需了解更多信息，请查看我们的论文并访问项目主页。

🧩 LLMLingua 已集成至 Prompt flow，这是一个面向 LLM AI 应用的精简工具框架。

🦚 我们很高兴宣布发布 LLMLingua-2，其速度比 LLMLingua 提升了 3～6 倍！如需了解更多信息，请查看我们的论文、访问项目主页并体验 Demo。

👾 LLMLingua 已集成至 LangChain 和 LlamaIndex，这两个框架都是广泛使用的 RAG 框架。

🤳 2024 年 1 月 AI Time 演讲的幻灯片现已发布。

🖥 EMNLP'23 幻灯片现已发布，可在 Session 5 和 BoF-6 中查看。

📚 欢迎阅读我们的新博客文章，其中讨论了如何通过 prompt 压缩获得 RAG 收益并节省成本。脚本示例可在此处查看。

🎈 欢迎访问我们的项目主页，了解 RAG、在线会议、CoT 和代码等真实场景中的案例研究。

👨‍🦯 欢迎浏览我们的 ./examples 目录，查看 LLMLingua-2、RAG、在线会议、CoT、代码以及使用 LlamaIndex 实现 RAG 等实际应用。

LLMLingua 使用一个紧凑且训练充分的语言模型（例如 GPT2-small、LLaMA-7B），识别并移除 prompt 中非必要的 token。这种方法能够提升大型语言模型（LLM）的推理效率，在性能损失极小的情况下实现最高 20 倍的压缩率。

LLMLingua：通过压缩 Prompt 加速大型语言模型推理（EMNLP 2023）
Huiqiang Jiang、Qianhui Wu、Chin-Yew Lin、Yuqing Yang、Lili Qiu

LongLLMLingua 缓解了 LLM 的“中间信息丢失”（lost in the middle）问题，增强了对长上下文信息的处理能力。它通过 prompt 压缩降低成本、提升效率，仅使用四分之一的 token，就能将 RAG 性能最多提升 21.4%。

LongLLMLingua：通过 Prompt 压缩加速并增强长上下文场景中的 LLM（ACL 2024、ICLR ME-FoMo 2024）
Huiqiang Jiang、Qianhui Wu、Xufang Luo、Dongsheng Li、Chin-Yew Lin、Yuqing Yang、Lili Qiu

LLMLingua-2 是一种体积小巧但功能强大的 prompt 压缩方法。它通过 GPT-4 进行数据蒸馏，使用 BERT 级别的 encoder 完成 token 分类，并且擅长与具体任务无关的压缩。处理领域外数据时，它的表现超过 LLMLingua，速度也快了 3～6 倍。

LLMLingua-2：通过数据蒸馏实现高效、忠实且与任务无关的 Prompt 压缩（ACL 2024 Findings）
Zhuoshi Pan、Qianhui Wu、Huiqiang Jiang、Menglin Xia、Xufang Luo、Jue Zhang、Qingwei Lin、Victor Ruhle、Yuqing Yang、Chin-Yew Lin、H. Vicky Zhao、Lili Qiu、Dongmei Zhang

SecurityLingua 是一个安全护栏模型。它使用具备安全意识的 prompt 压缩，揭示 jailbreak 攻击背后的恶意意图，使 LLM 能够检测攻击并生成安全响应。得益于效率极高的 prompt 压缩，这套防御机制的额外开销几乎可以忽略不计；与当前最先进的 LLM 护栏方案相比，其 token 成本降低了 100 倍。

SecurityLingua：通过具备安全意识的 Prompt 压缩，高效防御 LLM Jailbreak 攻击（CoLM 2025）
Yucheng Li、Surin Ahn、Huiqiang Jiang、Amir H. Abdi、Yuqing Yang、Lili Qiu

你是否曾在让 ChatGPT 总结长篇文本时遇到 token 上限？

你是否曾因进行了大量 fine-tuning 后，ChatGPT 忘记之前的指令而感到沮丧？

你是否曾在实验中使用 GPT3.5/4 API，虽然结果出色，却不得不承担高昂成本？

尽管 ChatGPT 和 GPT-4 等大型语言模型在泛化与推理方面表现优异，但它们经常面临 prompt 长度限制和按 prompt 计价等问题。

现在，你可以使用 LLMLingua、LongLLMLingua 和 LLMLingua-2！

这些工具提供了一种高效方案，最多可将 prompt 压缩 20 倍，从而进一步提升 LLM 的实用性。

💰 节省成本：以极低的额外开销，同时缩短 prompt 和生成内容的长度。

📝 支持更长上下文：增强对长上下文的支持，缓解“中间信息丢失”问题，并提升整体性能。

⚖️ 鲁棒性：无需对 LLM 进行额外训练。

🕵️ 知识保留：保留原始 prompt 中的信息，例如 ICL 和推理过程。

📜 KV-Cache 压缩：加速推理过程。

🪃 全面恢复：GPT-4 可以从压缩后的 prompt 中恢复所有关键信息。

PS：此 Demo 基于 alt-gpt 项目。特别感谢 @Livshitz 所作出的宝贵贡献。

如果这个 repo 对你有所帮助，请引用以下论文：

@inproceedings{jiang-etal-2023-llmlingua,
    title = "{LLML}ingua: Compressing Prompts for Accelerated Inference of Large Language Models",
    author = "Huiqiang Jiang and Qianhui Wu and Chin-Yew Lin and Yuqing Yang and Lili Qiu",
    editor = "Bouamor, Houda  and
      Pino, Juan  and
      Bali, Kalika",
    booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.emnlp-main.825",
    doi = "10.18653/v1/2023.emnlp-main.825",
    pages = "13358--13376",
}

@inproceedings{jiang-etal-2024-longllmlingua,
    title = "{L}ong{LLML}ingua: Accelerating and Enhancing {LLM}s in Long Context Scenarios via Prompt Compression",
    author = "Huiqiang Jiang and Qianhui Wu and and Xufang Luo and Dongsheng Li and Chin-Yew Lin and Yuqing Yang and Lili Qiu",
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.acl-long.91",
    pages = "1658--1677",
}

@inproceedings{pan-etal-2024-llmlingua,
    title = "{LLML}ingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression",
    author = "Zhuoshi Pan and Qianhui Wu and Huiqiang Jiang and Menglin Xia and Xufang Luo and Jue Zhang and Qingwei Lin and Victor Ruhle and Yuqing Yang and Chin-Yew Lin and H. Vicky Zhao and Lili Qiu and Dongmei Zhang",
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Findings of the Association for Computational Linguistics ACL 2024",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand and virtual meeting",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-acl.57",
    pages = "963--981",
}

@inproceedings{li2025securitylingua,
  title={{S}ecurity{L}ingua: Efficient Defense of {LLM} Jailbreak Attacks via Security-Aware Prompt Compression},
  author={Yucheng Li and Surin Ahn and Huiqiang Jiang and Amir H. Abdi and Yuqing Yang and Lili Qiu},
  booktitle={Second Conference on Language Modeling},
  year={2025},
  url={https://openreview.net/forum?id=tybbSo6wba}
}

要开始使用 LLMLingua，只需通过 pip 安装：

pip install llmlingua

借助 LLMLingua，你可以轻松压缩 prompt。具体用法如下：

from llmlingua import PromptCompressor

llm_lingua = PromptCompressor()
compressed_prompt = llm_lingua.compress_prompt(prompt, instruction="", question="", target_token=200)

# > {'compressed_prompt': 'Question: Sam bought a dozen boxes, each with 30 highlighter pens inside, for $10 each box. He reanged five of boxes into packages of sixlters each and sold them $3 per. He sold the rest theters separately at the of three pens $2. How much did make in total, dollars?\nLets think step step\nSam bought 1 boxes x00 oflters.\nHe bought 12 * 300ters in total\nSam then took 5 boxes 6ters0ters.\nHe sold these boxes for 5 *5\nAfterelling these  boxes there were 3030 highlighters remaining.\nThese form 330 / 3 = 110 groups of three pens.\nHe sold each of these groups for $2 each, so made 110 * 2 = $220 from them.\nIn total, then, he earned $220 + $15 = $235.\nSince his original cost was $120, he earned $235 - $120 = $115 in profit.\nThe answer is 115',
#  'origin_tokens': 2365,
#  'compressed_tokens': 211,
#  'ratio': '11.2x',
#  'saving': ', Saving $0.1 in GPT-4.'}

## Or use the phi-2 model,
llm_lingua = PromptCompressor("microsoft/phi-2")

## Or use the quantation model, like TheBloke/Llama-2-7b-Chat-GPTQ, only need <8GB GPU memory.
## Before that, you need to pip install optimum auto-gptq
llm_lingua = PromptCompressor("TheBloke/Llama-2-7b-Chat-GPTQ", model_config={"revision": "main"})

若要在你的场景中尝试 LongLLMLingua，可以使用：

from llmlingua import PromptCompressor

llm_lingua = PromptCompressor()
compressed_prompt = llm_lingua.compress_prompt(
    prompt_list,
    question=question,
    rate=0.55,
    # Set the special parameter for LongLLMLingua
    condition_in_question="after_condition",
    reorder_context="sort",
    dynamic_context_compression_ratio=0.3, # or 0.4
    condition_compare=True,
    context_budget="+100",
    rank_method="longllmlingua",
)

若要在你的场景中尝试 LLMLingua-2，可以使用：

from llmlingua import PromptCompressor

llm_lingua = PromptCompressor(
    model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
    use_llmlingua2=True, # Whether to use llmlingua-2
)
compressed_prompt = llm_lingua.compress_prompt(prompt, rate=0.33, force_tokens = ['\n', '?'])

## Or use LLMLingua-2-small model
llm_lingua = PromptCompressor(
    model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank",
    use_llmlingua2=True, # Whether to use llmlingua-2
)

若要在你的场景中尝试 SecurityLingua，可以使用：

from llmlingua import PromptCompressor

securitylingua = PromptCompressor(
    model_name="SecurityLingua/securitylingua-xlm-s2s",
    use_slingua=True
)
intention = securitylingua.compress_prompt(malicious_prompt)

有关 SecurityLingua 的更多详细信息，请参阅 SecurityLingua 的 README。

你可以将文本拆分为多个区段，并决定是否压缩以及采用何种压缩率。使用 <llmlingua></llmlingua> 标签划分上下文，还可以选择性设置 rate 和 compress 参数。

structured_prompt = """<llmlingua, compress=False>Speaker 4:</llmlingua><llmlingua, rate=0.4> Thank you. And can we do the functions for content? Items I believe are 11, three, 14, 16 and 28, I believe.</llmlingua><llmlingua, compress=False>
Speaker 0:</llmlingua><llmlingua, rate=0.4> Item 11 is a communication from Council on Price recommendation to increase appropriation in the general fund group in the City Manager Department by $200 to provide a contribution to the Friends of the Long Beach Public Library. Item 12 is communication from Councilman Super Now. Recommendation to increase appropriation in the special advertising and promotion fund group and the city manager's department by $10,000 to provide support for the end of summer celebration. Item 13 is a communication from Councilman Austin. Recommendation to increase appropriation in the general fund group in the city manager department by $500 to provide a donation to the Jazz Angels . Item 14 is a communication from Councilman Austin. Recommendation to increase appropriation in the general fund group in the City Manager department by $300 to provide a donation to the Little Lion Foundation. Item 16 is a communication from Councilman Allen recommendation to increase appropriation in the general fund group in the city manager department by $1,020 to provide contribution to Casa Korero, Sew Feria Business Association, Friends of Long Beach Public Library and Dave Van Patten. Item 28 is a communication. Communication from Vice Mayor Richardson and Council Member Muranga. Recommendation to increase appropriation in the general fund group in the City Manager Department by $1,000 to provide a donation to Ron Palmer Summit. Basketball and Academic Camp.</llmlingua><llmlingua, compress=False>
Speaker 4:</llmlingua><llmlingua, rate=0.6> We have a promotion and a second time as councilman served Councilman Ringa and customers and they have any comments.</llmlingua>"""
compressed_prompt = llm_lingua.structured_compress_prompt(structured_prompt, instruction="", question="", rate=0.5)
print(compressed_prompt['compressed_prompt'])

# > Speaker 4:. And can we do the functions for content? Items I believe are11,,116 28,.
# Speaker 0: a from Council on Price to increase the fund group the Manager0 provide a the the1 is Councilman Super Now. the special group the provide the summerman a the Jazzels a communication from Councilman Austin. Recommendation to increase appropriation in the general fund group in the City Manager department by $300 to provide a donation to the Little Lion Foundation. Item 16 is a communication from Councilman Allen recommendation to increase appropriation in the general fund group in the city manager department by $1,020 to provide contribution to Casa Korero, Sew Feria Business Association, Friends of Long Beach Public Library and Dave Van Patten. Item 28 is a communication. Communication from Vice Mayor Richardson and Council Member Muranga. Recommendation to increase appropriation in the general fund group in the City Manager Department by $1,000 to provide a donation to Ron Palmer Summit. Basketball and Academic Camp.
# Speaker 4: We have a promotion and a second time as councilman served Councilman Ringa and customers and they have any comments.

若想了解如何在 RAG、在线会议、CoT 和代码等真实场景中应用 LLMLingua 与 LongLLMLingua，请参阅我们的示例。有关详细指导，文档提供了大量建议，帮助你有效使用 LLMLingua。

若要使用自己的定制数据训练 compressor，请参阅我们的 data_collection 和 model_training。

常见问题

如需获得更多见解和答案，请访问 FAQ 部分。

本项目欢迎贡献和建议。大多数贡献都要求你同意《贡献者许可协议》（Contributor License Agreement，CLA），声明你有权并且确实授予我们使用你的贡献内容所需的权利。详情请访问 https://cla.opensource.microsoft.com。

提交 pull request 时，CLA bot 会自动判断你是否需要提供 CLA，并相应地标记 PR，例如添加状态检查或评论。你只需按照 bot 提供的指示操作即可。在所有使用我们 CLA 的 repo 中，你只需要完成一次这项操作。

本项目采用了 Microsoft Open Source Code of Conduct。如需了解更多信息，请参阅 Code of Conduct FAQ；若还有其他问题或意见，请联系 opencode@microsoft.com。

本项目可能包含其他项目、产品或服务的商标或徽标。对 Microsoft 商标或徽标的授权使用，必须遵守 Microsoft Trademark & Brand Guidelines。在本项目的修改版本中使用 Microsoft 商标或徽标时，不得引起混淆，也不得暗示该版本由 Microsoft 赞助。使用任何第三方商标或徽标时，均须遵守相应第三方的政策。