树莓派边缘部署 Gemma 模型：LiteRT 实战指南

Imagine building a fully autonomous robot that can see, hear, and react to its environment in real time, completely offline on a single compact device like Raspberry Pi. Edge AI unlocks this exact autonomy. It enables developers to build highly secure and self-contained systems like intelligent robotics and local AI agents with zero cloud dependencies, ultra-low latency, and total data privacy.

We've made running edge AI on the Raspberry Pi a breeze with Google AI Edge's LiteRT, high-performance production-proven on-device inference runtime. LiteRT allows you to seamlessly deploy everything from classical ML models to state-of-the-art LLMs right out of the box across platforms. By delivering optimized execution and hyper-efficient memory usage on both CPU and GPU, LiteRT maximizes your Raspberry Pi's full computing potential.

This hardware-software synergy shines brightest when paired with Gemma, Google's family of lightweight open models. To show you what's possible, we'll demonstrate how Gemma and LiteRT on a Raspberry Pi 5 can power the Reachy Mini robot to perceive and react to its environment entirely locally in real time. Read on to get started with your own deployment.

Discover the Agentic Capability of Gemma

Gemma models are well suited for building autonomous agents, smart cameras, and social robotics that can reason and execute complex, multi-step workflows directly on your Raspberry Pi. To accommodate different hardware constraints, the Gemma family of models provides several highly efficient options:

Gemma 3 270M: A hyper-efficient and compact base model designed for task-specific post fine-tuning, enabling high-speed, low-latency features like sentiment analysis or entity extraction in resource-constrained environments.

EmbeddingGemma 300M: A text embedding model that produces high-quality embeddings on-device, great for Retrieval Augmented Generation (RAG), semantic search, and classification.

Gemma 3 1B: A lightweight and multilingual text-only model that balances compact size with strong generative capabilities, making it ideal for a wide range of on-device tasks, such as summarization and content creation.

Gemma 4 E2B: Tailored specifically for mobile and tight edge environments, it features memory-mapped per-layer embeddings, and is ideal for continuous monitoring, fast text/image/audio inference, and edge-based speech processing where saving RAM is absolutely critical.

Gemma 4 E4B: The sweet spot for performance and size. This model delivers noticeably stronger reasoning capabilities and frontier-level edge performance while remaining compact. It is the perfect choice for complex multi-step planning without overwhelming the Pi's resources.

Gemma Performance on Raspberry Pi CPU

Through LiteRT-LM, a specialized orchestration layer on top of LiteRT, developers can seamlessly deploy Gemma right out of the box. Under the hood, sophisticated CPU acceleration via LiteRT and XNNPACK ensures the Gemma family of models is highly optimized for resource efficiency and low-latency execution directly on the Raspberry Pi.

On a Raspberry Pi 5, LiteRT-LM delivers a robust performance for Gemma 4 E2B, achieving 99 tokens/sec for prefill and 9 tokens/sec for decode, all while maintaining a remarkably low peak memory footprint of just 1432 MB. This brings Gemma's highly responsive, general-purpose intelligence to Raspberry Pi.

Thanks to Gemma 4 E2B's highly efficient tokenizer, which packs more text into fewer tokens (averaging ~4.2 characters per token), LiteRT-LM achieves an impressive end-to-end generation speed of ~27.3 characters per sec, roughly 300 words per minute (wpm), in the Reachy Mini voice demo. This throughput makes Gemma 4 E2B excellent for real-time speech and translation tasks, delivering text at twice the speed of normal human speech (~150 wpm).

Explore more ready-to-use open models to run on Raspberry Pi from the LiteRT Hugging Face Community.

Execute on Raspberry Pi GPU with LiteRT

On the Raspberry Pi 5, the quad-core ARM Cortex-A76 CPU is a raw computing powerhouse, delivering ~153.6 GFLOPS (FP32) and up to ~2.0 TOPS (INT8). In comparison, the integrated Broadcom VideoCore VII GPU is clocked at 800 MHz and offers a peak of ~76.8 GFLOPS (FP32) and ~0.24 TOPS (INT8).

While the CPU possesses a massive capacity advantage, the GPU introduces heterogeneous parallel execution, a paradigm critical for real-time edge applications. Rather than saturating the CPU, developers can delegate tasks across both processors to optimize overall system and thermal efficiency. For example, by offloading continuous vision or audio models to the VideoCore VII GPU, it preserves high-priority CPU cycles for overall system monitoring, pipeline orchestration, or computationally demanding LLM inference.

As such, we have enabled GPU inference on the Raspberry Pi with LiteRT's WebGPU (Vulkan) backend via ML Drift. This integration allows you to run a wide range of computer vision, audio, and embedding models directly from the LiteRT Hugging Face Community, including seamless support for popular MediaPipe models, Ultralytics YOLO models, Moonshine, and much more.

The table below demonstrates the CPU and GPU latency of running classic computer vision and audio models via LiteRT:

Deep Dive: Reachy Mini Pipeline Powered by LiteRT

The Reachy Mini pipeline is a powerful showcase of low-latency, real-time edge AI inference running entirely on the Raspberry Pi 5. By leveraging LiteRT, the system splits intensive vision and language workloads into a concurrent, dual-processing architecture across CPU and GPU.

Here is how the parallel architecture works under the hood to ensure seamless interactions:

Object Detection (Ultralytics YOLO on GPU): Camera frames are streamed to the Pi, where a Ultralytics YOLO detection runs continuously on the GPU, avoiding resource contention and frees up the CPU.

Speech Recognition (Moonshine on CPU): When the user speaks, the ASR component transcribes the audio into text directly on the CPU.

Reasoning & Action (Gemma 4 E2B on CPU): The Gemma 4 E2B model processes the resulting transcript alongside the latest visual metadata to generate low-latency, streaming responses, such as speech replies and physical robotic gestures.

Text-to-Speech (TTS on CPU): The TTS component synthesizes the generated text into audio in streaming. The system streams the synthesized voice back to the Reachy Mini robot.

See the full source code of the Reachy demo in the LiteRT Samples Github repo.

Agentic Coding with LiteRT on Raspberry Pi

LiteRT provides a comprehensive suite of tools that covers the full development cycle: conversion, quantization, benchmark, and inference. For a fast, frictionless setup, the most straightforward approach is using the LiteRT CLI tool. Rather than requiring developers or coding agents to manually manage multiple independent libraries, the LiteRT CLI aggregates core edge workflows into a single, unified command set.

You can now supercharge your development cycle by adding the LiteRT CLI skill and other advanced LiteRT skills into your AI coding agent, such as Google Antigravity. This empowers agents to autonomously orchestrate and execute complex, multi-stage machine learning workflows on your behalf. For example, you can easily build your own voice translator completely offline on a Raspberry Pi, like the Gemma Translator shown below.

Explore the complete implementation details in the Gemma Translator GitHub repo.

An Ultra-Lean Binary Footprint for IoT Devices

For resource-constrained IoT devices, minimizing storage and memory overhead is critical. Without special optimization, generic AI runtimes often bundle heavy desktop or server dependencies. In contrast, LiteRT is engineered specifically for on-device deployment, maintaining an exceptionally lean and modular distribution.

The table below compares the download footprint required to run LLM inference on a Raspberry Pi (ARM64 Linux).

Running Your First Model

You can install the LiteRT CLI and run your first model on a Raspberry Pi 5 with just a few simple commands.

To get started, install the LiteRT CLI via pip (ideally within a virtual environment):

pip install litert-cli

Download and run any compatible model directly from the LiteRT Hugging Face Community. The code snippet below demonstrates how to execute Gemma 4 E2B (e.g. gemma-4-E2B-it-litert-lm) on the Raspberry Pi 5.

Run the model by providing your Hugging Face authentication token:

export HUGGING_FACE_HUB_TOKEN=<your_hugging_face_token_here>
litert lm run \
  --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
  gemma-4-E2B-it.litertlm \
  --attachment=image.jpg \
  --prompt="You are Reachy Mini. Identify the main object in front of you, "\
    "state its location (Left/Right/Center), and suggest head action in "\
    "10 words or less."

We are excited to share that LiteRT integration and Gemma models are coming soon to Hailo AI accelerators! This update will allow you to seamlessly offload model inference to the Raspberry Pi AI HAT+ and AI HAT+ 2, delivering massive hardware acceleration benefits through the exact same, familiar LiteRT workflows you use today.

Explore our resources and start your journey with LiteRT:

• Official Documentation: Access installation guides, API references, and quick-start tutorials on the LiteRT Developer Site.
• GitHub repos: Find the latest source code, implementation details, and updates on the LiteRT and LiteRT-LM GitHub repos.
• Samples & Templates: Check out the LiteRT-Samples GitHub repo for reference code. Kickstart your own application using the AI Edge Gallery app.
• Ready-to-use Models: Download optimized, open-weights models like the lightweight and powerful Gemma 4 E2B directly from the LiteRT Hugging Face Community.

We value your input. Please share your thoughts, feedback, or feature requests by opening an issue on our GitHub Issue Tracker. Share your cool Raspberry Pi + LiteRT + Gemma projects with @googlegemma. We can't wait to see what you build!

Google: Changming Sun, Chintan Parikh, Cormac Brick, Dillon Sharlet, Daisuke Majima, Erin Walsh, Frank Barchard, Glenn Cameron, Ian Ballantyne, Jingjiang Li, Jun Jiang, Kimish Patel, Lu Wang, Matthias Grundmann, Rodney Witcher, Sachin Kotwani, Sasha Denisov, Scott Loftin, Shuangfeng Li, Somdatta Banerjee, Terry (Woncheol) Heo, Volodymyr Kysenko, Weiyi Wang, Yi-Chun Kuo, Yu-hui Chen, and gtech team

Raspberry Pi & Hailo: Ashley Whittaker, Eldad Rubinstein, José María Casanova (Igalia), Naushir Patuck, and Sarah Cunningham

Ultralytics: Francesco Mattioli, Onuralp Sezer, Lakshantha Dissanayake

Moonshine AI: Pete Warden