Markdown比HTML在LLM上下文中价值高10倍

Originally posted on the Scrapio blog — sharing here too.

When you're building a RAG pipeline or an LLM agent that reads web pages, the format you put into the context window matters more than most people realize. Raw HTML and clean markdown can represent the same information, but one costs 10× more in tokens and adds substantial noise that degrades model performance.

The problem with HTML

A typical product page might have 3,000 words of actual content. As raw HTML, that page is often 50,000–80,000 characters — mostly tags, class names, JavaScript, tracking pixels, and navigation menus.

LLMs tokenize all of it. You pay for every <div class="product-carousel__item--highlighted"> even though it carries no information your model needs.

We ran 100 pages through both approaches and measured:

The accuracy difference is real: LLMs perform better when they're not trying to reason through a wall of HTML noise.

What clean markdown looks like

<div class="pdp-price-container" data-testid="price-wrapper">
  <span class="price--sale" aria-label="Sale price">
    <span class="price__currency">$</span>
    <span class="price__amount">279</span>
    <span class="price__cents">.99</span>
  </span>
</div>

Same content as markdown:

**Price:** $279.99

The markdown version is 22 tokens. The HTML version is 89 tokens — and that's a small excerpt. An entire page compounds this by 50–100×.

Getting clean markdown from any URL

curl -X POST https://api.scrapio.dev/v1/fetch \
  -H "Authorization: Bearer sk-..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/article", "output": ["markdown"]}'

{
  "request_id": "req_abc123",
  "mode": "inline",
  "status": "completed",
  "outputs": {
    "markdown": "# Article Title\n\nContent here..."
  },
  "usage": { "credits": 1 }
}

The markdown output strips navigation, ads, scripts, and layout elements. What remains is the readable content — headings, paragraphs, lists, tables, and code blocks.

JavaScript-rendered pages

SPAs and dynamic pages need a render step:

{
  "url": "https://example.com/dashboard",
  "output": ["markdown"],
  "render_js": true
}

Without render_js, you'd get the pre-hydration HTML — often an empty shell.

When crawling large sites for a vector database, markdown is especially important because:

• Chunking is cleaner — markdown structure (headings, paragraphs) provides natural split points
• Embeddings are better — semantic models trained on clean text produce better vectors when the input isn't polluted with HTML syntax
• Storage is cheaper — 10× smaller documents mean lower storage and fewer API calls to retrieve them

For further actions, you may consider blocking this person and/or reporting abuse