Measuring the Search of My Knowledge Base

Five local embedding models against 26 test questions: why the search now runs on BGE-M3 instead of multilingual-e5-small, with a new minimum score.

Article Created Updated Also in German: Die Suche meiner Wissensdatenbank vermessen

Reviewed by adrian on

The search of my knowledge base ran on multilingual-e5-small from the start, the model the Astro prototype already used. Nobody had checked whether it finds the right node. In November 2026 I measured it against four other models that run locally, and switched to BGE-M3. How the search works as a whole is in How an LLM Wiki Works.

The setup

  • 44 searchable nodes in 148 sections (published and private, no drafts), 37 in English and 7 in German.
  • 26 test questions, 15 in German and 11 in English, each with the node that answers it. They use other words than the nodes, and most German questions point to nodes that only exist in English.
  • Each model as quantized ONNX (8 bit), run with ONNX Runtime like the kb tool. A node counts with its best section.
  • Measured: how often the right node came first or among the first three, and the mean reciprocal rank (MRR). Also the gap between the right node and the best wrong one, and the time per section and per question.

Results

Model First place Top 3 MRR Gap ms per section / question Size
multilingual-e5-small (before) 19/26 20/26 0.78 0.015 8 / 2 129 MB
multilingual-e5-base 17/26 22/26 0.77 0.014 17 / 4 290 MB
EmbeddingGemma-300m 23/26 25/26 0.93 0.124 55 / 38 326 MB
BGE-M3 26/26 26/26 1.00 0.073 50 / 9 561 MB
Qwen3-Embedding-0.6B 18/26 21/26 0.76 0.027 77 / 24 604 MB
  • e5-small sorted by language more than by meaning. Six of its seven misses were German questions that landed on one of the seven German nodes. "Landkarten, deren Aussehen erst im Browser berechnet wird" found the Safari bookmark sorter first and Vector Tiles only in tenth place.
  • BGE-M3 put the right node first for every question, German questions about English nodes included.
  • EmbeddingGemma separated right and wrong most clearly, but missed three questions narrowly.
  • e5-base was no better than e5-small. Qwen3 scored lower than expected. The 8-bit version or my pooling may be the reason, so I don't take that as a final verdict.

The minimum score

The search drops sections below a minimum similarity. It was 0.75, set for e5-small. A second test with 14 unrelated questions (recipes, weather, travel) showed that it filtered nothing: with e5-small even those reached 0.81.

With BGE-M3, questions about the knowledge base reached at least 0.51 and unrelated ones at most 0.44. The minimum is now 0.48, between the two. All 26 questions still get their answer, none of the 14 unrelated ones does.

The switch

  • kbtool/embed.py uses Xenova/bge-m3 without the "query:" and "passage:" prefixes that e5 needs, and takes the vector of the first token instead of the mean.
  • The index remembers its model. The search refuses an index built with another model until kb index has embedded everything again, about 18 seconds for 148 sections.
  • The model files in the cache grow from about 120 to about 600 MB.

Open points

  • 26 questions I wrote myself are a small sample: one question is four percentage points. The gap between 19 and 26 is clear, the order of the others less so.
  • The minimum score belongs to the model. With the next model it has to be measured again, with the same questions. They are in the saved chat.

Based on

  • Measuring embedding models for the knowledge base search (raw/2026/2026-11-16/measuring-embedding-models-for-the-knowledge-base-search.md) [chat]

More on this

Topic

  • AI / Language Models What I use language models for: agents, local models, RAG and small programs in which a model handles one task.
  • Retrieval-Augmented Generation Answering questions with an LLM by first retrieving relevant passages from a knowledge store and giving them to the model as context.

Mentioned

  • Vector Tiles Map tiles that contain geometries instead of finished images. The map's appearance is only created in the browser.

See also

  • How an LLM Wiki Works How an LLM wiki works, shown on this knowledge base: Markdown in its own Git repository, a Python tool with MCP for agents, and a website that only displays it.

Show in the graph view