企业 RAG 实战：500GB 文档下的架构踩坑与解决思路

Every company that decides it wants "an AI assistant for our internal documents" hits the same wall within about a week.

The documents exist. Contracts, employee handbooks, product specs, meeting notes, support conversations, policies, half a decade of technical documentation. Somewhere around a few hundred gigabytes of it. And there is no version of pasting that into a chat window that works. Sending the whole repository to a model on every question isn't just expensive, it's architecturally wrong.

The pattern that does work is Retrieval-Augmented Generation. And the reason it's worth understanding properly, rather than as a buzzword, is that most of the engineering effort has nothing to do with the model you pick.

The two obvious approaches, and why both fail

The first instinct is to build a search engine. Index titles, keywords, contents, metadata, rank the results, done. This works until the corpus grows, and then every query means scanning an enormous amount of data. You've moved the cost, not removed it.

The second instinct is to pre-process everything into summaries. Cheaper to search, much smaller footprint. The problem is that summarization is lossy by definition, and in enterprise documents the thing you lose is usually the thing that mattered. A contract can turn on a single clause. A summary drops that clause without telling you it did. You end up with a system that confidently answers questions from a version of the truth you deliberately damaged.

What you actually need is efficient retrieval combined with semantic understanding. That's the gap RAG fills.

Three stages, and the one people skip

The architecture splits into retrieval, augmentation, and generation.

Retrieval means converting documents into a form that supports meaning-based search. Text gets turned into embeddings, vectors that encode aspects of what the text means rather than which words it contains. "Dogs are allowed" and "Pets are permitted" land close together in that vector space despite sharing no keywords. The user's question goes through the same embedding model, and the system compares vectors instead of matching strings.

This is the real difference from traditional search. Ask "Are pets permitted in the office?" and a keyword engine hunts for pets, permitted, office. A semantic search can surface "Employees may bring dogs into the workplace" and rank it highly, because meaning is what's being compared.

Augmentation is the part people skip when they explain RAG, and it's the part that carries most of the value. The retrieved chunks get injected directly into the prompt, alongside the user's question, with an instruction to answer from those documents. That's it. That's the mechanism. Your company's current contracts, this quarter's pricing, yesterday's policy change, all of it becomes runtime knowledge for a model that was frozen long before any of it existed. Update the knowledge base, and the assistant's answers update. No retraining, no fine-tuning.

Generation is the model doing what it's good at: reading the supplied context and writing a grounded answer.

Which leads to the distinction that matters architecturally: the model never learns your documents. Nothing is stored in it. Every single request runs a fresh retrieval, and the model reads what it's handed. RAG is a runtime information retrieval architecture wearing an AI costume.

Chunking is where most RAG projects quietly fail

You don't embed a hundred-page document as one vector. You split it into chunks, embed each one, and retrieve the pieces that matter. A typical starting point is something like 500-token chunks with 100 tokens of overlap, and the overlap exists for a specific reason.

Imagine a sentence split across a boundary. Chunk A ends with "The contract states that the company must provide..." and chunk B begins with "...support for a period of 12 months." Retrieve either one alone and you have a fragment that reads as complete but says nothing useful. Overlap means some context survives in both.

Here's the part that isn't in the tutorials: there is no correct chunk size. It's a property of your data, not of RAG.

Legal documents are built out of long structured paragraphs, defined terms, numbered clauses, and references back to earlier sections. Shred them into small pieces and you destroy the structure that carried the meaning. They want structure-aware chunking that respects section boundaries.

Support conversations are the opposite. Short turns, heavy dependence on surrounding context, meaning distributed across a back-and-forth. They want smaller chunks and generous overlap.

Same framework, opposite parameters. If you're evaluating RAG quality and only tuning the embedding model, you're tuning the wrong dial.

Retrieval quality is the ceiling on answer quality

This is the lesson I'd put on the wall.

If the retriever returns the wrong documents, the model doesn't fail loudly. It fails beautifully. You get a well-structured, confident, fluent answer built on irrelevant context, and nothing in the output signals that anything went wrong. Bad retrieval produces bad context produces a bad answer that reads exactly like a good one.

So a serious retrieval layer needs more than nearest-neighbour search:

• A similarity threshold so that results below a certain score are rejected outright. Ask a system "what's our quantum computing strategy?" when no such document exists, and without a threshold it will retrieve the five least-irrelevant chunks it owns and let the model improvise. With a threshold, it says it couldn't find anything. That's not a complete answer to hallucination, but it removes an entire category of it.
• Top-K limits, because five strong chunks beat a hundred mediocre ones competing for the model's attention.
• Metadata filtering, which in enterprise settings is often more valuable than the vector search itself. Store department, document type, year, counterparty, document ID alongside every chunk. Then "last year's service agreement with CodeCloud" becomes a semantic search constrained to document_type = contract, year = 2025, company = CodeCloud. You've collapsed the search space before similarity even runs.

The tutorial version and the production version

A working demo is genuinely simple: parse, chunk, embed, store in a vector database, search, prompt, answer. You can build it in an afternoon with Python, ChromaDB, a sentence-transformer model, and a small Flask front end. That's enough to understand the idea, and it's a good use of an afternoon.

Production looks different. Parsing and cleaning, structure extraction, metadata enrichment, query rewriting, hybrid retrieval combining keyword and semantic signals, reranking, deliberate context assembly, citation and validation on the way out. Every one of those is a real engineering problem with real trade-offs, and none of them are solved by upgrading to a better LLM.

That's the whole point. A capable model with a weak retrieval pipeline produces a polished, wrong assistant. A strong retrieval pipeline with an ordinary model produces something people trust and keep using.

RAG doesn't make a model learn your documents. It hands the model the right pieces of them at the moment it needs them — and the engineering lives entirely in the word right.

---

For those of you running RAG on real internal data: what's given you the biggest quality jump so far — better chunking, metadata filtering, or reranking?