共享日志去重失效：Jaccard 相似度误判的根本原因

Every night, an autonomous agent I run scans Hacker News, pulls out two takeaways, and appends them to its own memory as "lessons." Before writing a new one, it checks: have I said this before? If yes, skip it — nobody needs "memory prices are rising" logged for the fifth time in a week.

The dedup check looked solid. It compared the new lesson's words against the last 30 prior scan entries using a Jaccard similarity score. Simple, cheap, no LLM call needed. And yet the same takeaways — "Copilot autofix risk," "memory prices rising" — kept reappearing every few days, exactly the thing the check was built to prevent.

The agent's memory isn't scoped per-strategy. It's one append-only log, shared across everything: the nightly web scan, a nightly reflection pass, a weekly "big think" summarization step, and a self-improvement routine that also writes lessons about its own code changes. Every one of those writes into the same file, interleaved by timestamp, no partitioning.

The store's recent() method is exactly what you'd expect from a shared log:

def recent(self, count: int = 15) -> list[Lesson]:
    return self.all()[-count:]

Last N entries, full stop. It doesn't know or care what category they belong to.

The dedup check then filtered that slice down to just the web-scan entries:

def _is_duplicate(self, text: str) -> bool:
    recent_lessons = self._lessons.recent(_DEDUP_LOOKBACK)  # last 30, ANY category
    web_scan_lessons = [l for l in recent_lessons if "[web-scan]" in l.text]
    # ... compare new text against web_scan_lessons

That reads correctly on a quiet day. It reads correctly in every unit test, because the tests never seed enough volume to expose the problem. It is wrong on any day where other writers are busy.

Here's the failure mode: reflection appends 0–3 lessons a night, big_think appends several more once a week, and the self-improvement routine appends its own entries whenever it changes code. On a day when those three write, say, 20 lessons combined, the "last 30 lessons" window might contain only 10 web-scan entries — or, on a busy week, zero. The comparison set the dedup check was supposed to hold constant instead shrinks and grows with how chatty unrelated parts of the system happen to be that day.

A takeaway logged eight days ago falls out of a 30-entry window in two or three days if enough other categories are writing. The dedup check doesn't error, doesn't warn, doesn't even know it's degraded — it just quietly starts comparing against fewer and fewer real prior entries, and duplicates walk right back in.

The bug isn't the Jaccard similarity math. It's the order of two operations that look interchangeable and aren't: slice first, then filter versus filter first, then slice. Slicing first fixes the window size in terms of the wrong population — everything, not the thing you actually care about. Any category with a lower write rate than its neighbors gets systematically under-represented, and the effect gets worse as the noisy categories get noisier.

Why this isn't a one-off

The same shape shows up anywhere you keep a fixed-size trailing window over a log that multiple producers write into: a rate limiter built on "last N requests" instead of "last N requests from this client," a moving-average metric computed over "last N samples" from a multiplexed stream, or a dedup key check in an incremental sync that looks at "the last N records synced" instead of "the last N records synced for this stream." If you've built a CDC connector or an incremental extractor that dedupes against a rolling window of previously-seen primary keys, and that window is shared across multiple source tables or event types, you've probably got the same trap sitting in the code, waiting for one table to get noisier than the others.

The fix, once you see it, is boring:

def _is_duplicate(self, text: str) -> bool:
    new_words = set(_WORD_RE.findall(text.lower()))
    checked = 0
    for lesson in reversed(self._lessons.all()):
        if "[web-scan]" not in lesson.text:
            continue
        checked += 1
        if checked > _DEDUP_LOOKBACK:
            break
        existing_words = set(_WORD_RE.findall(lesson.text.lower()))
        overlap = len(new_words & existing_words) / len(new_words | existing_words)
        if overlap >= _DEDUP_SIMILARITY:
            return True
    return False

Walk the full log newest-first, and count only matching-category entries toward the cap. The window size is now defined in terms of the population that actually matters, independent of how much unrelated traffic happened to land between them. It costs one extra pass over an already-in-memory list — no new I/O, no new dependency, no change to the public method signature.

The part that actually bit me

The existing test suite didn't catch this, and it's worth being honest about why: none of the tests seeded enough lesson history to make the two approaches diverge. recent(30) and "the last 30 web-scan entries" produce identical results when nothing else is writing to the log — which is exactly the condition every test fixture happened to create. The bug was invisible in isolation and only showed up in production, over days, as a slow degradation with no error message attached to it.

The fix for the test gap is the same shape as the fix for the bug: stop testing the window in a vacuum. A boundary test needs to seed interleaved noise — lessons from other categories mixed in with the ones you're deduping against — and assert that the window still holds exactly N matching entries regardless of how much noise sits between them. If your fixture never interleaves, you'll never see this class of bug, no matter how good your assertions are.

If you're building or reviewing dedup logic, rate limiting, or a rolling checkpoint window against any log or stream that isn't exclusively yours, ask two questions before trusting it:

Is the window size defined over the population I actually care about, or over everything that happens to share the log?

Do my tests interleave unrelated writes, or does every fixture leave the log quiet except for the thing under test?

If the answer to either is "I'm not sure," that's the same fifteen-minute check that would have caught this before it shipped. Filter, then slice — not the other way around.