For Agents · MCP

Your task.
The right skill.

The tunnel uses the same cards as the catalogue. Browse only as deep as needed — or load a broad bundle immediately.

No local installationNo M11 loginRead-only

Broad task

Load 5–10 and go

SEO, Sales, Agents or another broad area → one bundle call → work.

MCP endpoint: https://skills.m11.ch/mcp

Read-only access to published skills. Default 8, maximum 10 skills / 120,000 characters.

Task Packs

Marketing Foundation

Compact two-skill starter: clarify positioning and choose a lead magnet. Use Marketing Launch for the broader eight-skill go-to-market workflow.

Marketing Launch

Build an evidence-led marketing plan from ICP and competition through positioning, campaigns, growth and measurement.

Agent Audit Essentials

Diagnose architecture and context, plan agent-team responsibilities, then organize project context and session handoffs. Memory and cost-runtime reviews remain outside this pack.

Conversion and Activation Review

Review the journey from landing page and lead capture through registration, first value and transparent upgrades.

Campaign Content and Brand Review

Plan a campaign, draft its channel content and review the work against actual brand guidance.

Content and Launch Planning

Prioritize an editorial roadmap and plan how to launch and distribute it across suitable channels.

Customer, Market and Community Research

Understand customer needs, compare competitors and plan a community around real member value.

Lead Magnet to Lifecycle Nurture

Choose a relevant lead magnet, then draft a permission-based nurture journey with entry, suppression and exit rules.

API Contract and Observability

Define the API contract, then plan how to observe its latency, failures and retries. Guidance and checklist; no production changes.

Data Analysis Foundation

Profile a dataset, choose and interpret statistical methods, then validate calculations and conclusions before sharing.

B2B Outreach Foundation

Define the target account, prioritize buying signals, plan a human LinkedIn engagement routine and prepare evidence-led responses to buyer concerns.

Choose an area

01 · Category

What the MCP returns at this step

18 · Engineering · Search Architecture

Embeddings & Semantic Search

Use when choosing an embedding model, chunk size, or query form, when semantic search returns irrelevant results, when adding hybrid BM25+vector or a reranker, or when a retrieval change needs a number (recall@k, nDCG, MRR). NOT operating the store — index tuning, quantization (that is `vector-db`) — nor the retrieve-to-answer loop (that is `rag`).

embeddingssemantic-searchchunkinghybrid-searchrerankingrrfretrieval-evalmtebmatryoshkaembeddings-search

Embeddings & Semantic Search · Original SKILL.md

---
name: embeddings-search
description: "Use when choosing an embedding model, chunk size, or query form, when semantic search returns irrelevant results, when adding hybrid BM25+vector or a reranker, or when a retrieval change needs a number (recall@k, nDCG, MRR). NOT operating the store — index tuning, quantization (that is `vector-db`) — nor the retrieve-to-answer loop (that is `rag`)."
tags: [embeddings, semantic-search, chunking, hybrid-search, reranking, rrf, retrieval-eval, mteb, matryoshka]
recommends: [vector-db, rag, structured-extraction, prompt-engineering, postgresdb]
origin: risco
---

# embeddings-search — make and judge the vectors

You own the **embedding technique layer**: turn a corpus into searchable vectors, turn a
question into a good retrieval, and **measure whether that retrieval is any good**. You stop the
moment the right chunks come back, measured by a number. You do not assemble a prompt or
generate an answer.

Route the adjacent surfaces away:

- **Operating the store** — collection schema, HNSW/IVFFlat tuning, metadata-filter path,
  quantization, `ef_search` recall knobs → [`../vector-db/SKILL.md`](../vector-db/SKILL.md).
  You decide *what vectors go in and how to query*; vector-db decides *how the store holds and
  serves them*.
- **The full retrieve → rerank → prompt → generate → answer loop** and its groundedness /
  faithfulness eval → [`../rag/SKILL.md`](../rag/SKILL.md).
- **Pulling typed fields out of documents** (invoice number, date, total) → [`../structured-extraction/SKILL.md`](../structured-extraction/SKILL.md).
- **Writing the prompt the model reasons with** → [`../prompt-engineering/SKILL.md`](../prompt-engineering/SKILL.md).

## 1. Pick the embedding model

Decide on three axes: language coverage, quality tier (read MTEB but don't worship it), and
cost — where cost is set by **dimensions**, because dims set storage and memory.

| Model | Best when | Dims (Matryoshka) | Max input | ~Price /1M tok | Query/doc asymmetry |
|---|---|---|---|---|---|
| OpenAI `text-embedding-3-small` | Cheap English/multi baseline | 1536 (truncatable) | 8191 tok | ~$0.02 | none required |
| OpenAI `text-embedding-3-large` | Higher quality, still API-simple | 3072 (truncatable) | 8191 tok | ~$0.13 | none required |
| Cohere `embed-v4` | Strong multilingual, API | up to 1536 | long | API-priced | `search_query` vs `search_document` |
| Voyage `voyage-3-large` | Retrieval-specialised, top tasks | model-set | long | API-priced | yes (`input_type`) |
| Gemini Embedding | Tops MTEB English retrieval (~68.3) | truncatable | long | API-priced | yes (task type) |
| `BGE-M3` / `e5` (open) | Self-host, no per-token bill | 1024 (BGE-M3) | long | self-host | yes (`query:` / `passage:`) |

Quality anchor (mid-2026 MTEB English retrieval): Gemini ~68.3, Cohere `embed-v4` ~65.2,
OpenAI `3-large` ~64.6, `BGE-M3` ~63.0. MTEB is the standard comparison, not a verdict on
*your* domain — [`references/models.md`](references/models.md) carries the full model matrix
(dims, max tokens, price, `input_type` convention, Matryoshka support) and how to read MTEB
without over-trusting it.

Two hard rules — each is a **silent** failure, no error, just worse results:

- **Match the distance metric to the model.** A cosine-trained model indexed or queried with
  L2 ranks silently wrong. Cosine → `<=>` in pgvector / `Distance.COSINE` in Qdrant. The index
  operator itself is vector-db's job; the *requirement* originates from the model, so state it
  in your config.
- **Respect query/document asymmetry.** Cohere, Voyage, Gemini, e5, BGE expect a different
  prompt or `input_type` for the query vs the stored passage. Embed both sides identically and
  recall silently drops.

**Dimensions = cost.** A 1024-dim float32 vector is 4 KB; at 10M docs that is 40 GB, and
doubling dims doubles storage and memory. Matryoshka-trained models (OpenAI 3-*, Cohere,
Gemini) let you **truncate** dims for graceful degradation — never re-embed the whole corpus
just to shrink vectors.

## 2. Chunk the corpus

Start boring. Upgrade only when a number tells you to.

```python
# Default recipe: recursive split, TOKEN-accurate count, 10–20% overlap.
import tiktoken
from langchain_text_splitters import RecursiveCharacterTextSplitter

enc = tiktoken.get_encoding("cl100k_base")
splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base",
    chunk_size=512,      # tokens, not characters
    chunk_overlap=64,    # ~12% — keeps sentences from being cut mid-thought
)
chunks = splitter.split_text(document_text)
```

Counting by characters instead of tokens is the most common own-goal: 512 characters is
~100–130 tokens of English and far fewer of CJK, so "512" silently means different things per
language and per model limit.

Upgrade ladder — graduate only when retrieval metrics (section 6) justify the added compute:

| Strategy | When it pays | Cost |
|---|---|---|
| Recursive (default) | Always start here | lowest |
| Semantic (group by meaning) | Topic-mixed pages where fixed splits cut mid-idea; ~70% lift over naive in some benchmarks | one extra embed pass |
| Late chunking | Docs heavy with pronouns/anaphora ("it", "the company"); +10–12% on those | needs a long-context model |
| Contextual retrieval (prepend a heading/summary per chunk) | Chunks that aren't self-contained without their section | higher compute, more tokens stored |

**Embed the searchable text; store the rest as metadata.** What you embed is what gets matched
— don't bury the answer text under boilerplate, and don't embed raw HTML.

## 3. Construct the query

The query is half the retrieval. Embed it the way the model expects, then improve it only when
recall data says you should.

```python
# Asymmetric model: query and document use DIFFERENT input_type. Getting this wrong is silent.
q_vec   = embed(text=user_question, input_type="search_query")     # Cohere / Voyage
d_vec   = embed(text=passage,       input_type="search_document")
# e5 / BGE convention is a textual prefix instead:
#   query    -> "query: how do refunds work"
#   passage  -> "passage: Refunds are processed within 14 days…"
```

Query-side techniques, when each pays:

- **Query rewriting** — when user queries are terse or full of pronouns; normalise before
  embedding.
- **HyDE** (embed a hypothetical answer, not the question) — when questions are short and
  answers are long/technical, so the answer-shaped vector lands nearer the passage.
- **Multi-query** (fan out 3–4 paraphrases, union the hits) — when one phrasing under-recalls;
  costs N embeds and a dedup.

## 4. Hybrid + rerank

Dense and sparse fail in complementary ways: **BM25 nails exact terms, IDs, SKUs, rare
tokens**; **dense nails paraphrase**. That is why exact-match queries return nothing while
paraphrases work — the fix is adding sparse, not a bigger embedding model.

Fuse by **rank, not score**, with Reciprocal Rank Fusion so you never have to calibrate BM25
tf-idf magnitudes against cosine magnitudes per corpus:

```python
# RRF: each doc scores 1/(k + rank) summed across the dense and sparse lists. k≈60.
def rrf(*ranked_lists, k=60):
    scores = {}
    for lst in ranked_lists:                 # fan-in 20–100 per list
        for rank, doc_id in enumerate(lst):  # rank is 0-based
            scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)
```

Then a **cross-encoder reranker** sits AFTER fusion: take top-50, score each against the
original query, keep top-5 for downstream use.

- Current models: Cohere `rerank-v4.0-pro` / `rerank-v4.0-fast` (note `rerank-3.5` is
  deprecated). Voyage `rerank-2.5` (2025-08-11) is the first widely available
  **instruction-following** reranker — 32K-token context (8× Cohere v3.5), reports +7.94%
  accuracy vs Cohere v3.5 on a 93-dataset suite.
- **A reranker raises precision but cannot recover a doc the retriever never returned.** Recall
  is upstream. If the right chunk isn't in the top-50, no reranker saves you — fix recall first.

The fusion and index *mechanics per engine* (how Qdrant/Weaviate/pgvector run hybrid) are
vector-db's: [`../vector-db/SKILL.md`](../vector-db/SKILL.md).

## 5. Measure retrieval quality

This is the rigor of the skill. "Search is bad" is not actionable; "recall@10 is 0.62" is.

1. **Build a golden query set** — 30–50+ labeled `query → relevant doc ids` pairs drawn from
   real questions. This is the asset; everything else is reproducible from it.
2. **Pick metrics** — definitions, a runnable Python eval skeleton, golden-set construction and
   the A/B-one-change methodology are in
   [`references/evaluation.md`](references/evaluation.md):
   - **recall@k** — of the truly relevant docs, how many landed in the top-k. Catches "the
     right chunk never came back."
   - **nDCG@k** — rewards relevant docs ranked higher. Catches "right docs, wrong order."
   - **MRR** — how high the *first* relevant doc sits. Catches "the one answer is buried."
3. **Move-the-number loop.** Establish a baseline, **change exactly one thing** (model OR chunk
   size OR fusion OR reranker), re-measure on the **same** query set. Two changes at once and
   you learn nothing.

A retrieval change shipped without a before/after number on a golden set is a guess. `verify.sh`
flags hybrid/rerank artifacts that mention no recall/nDCG/MRR for exactly this reason.

## Anti-patterns

| Anti-pattern | Why it bites | Do instead |
|---|---|---|
| Chunk size in characters | "512" means a different token count per language/model | Token-accurate count (`tiktoken`/model tokenizer) |
| Same `input_type` for query and document | Asymmetric models silently lose recall, no error | `search_query` vs `search_document` (or `query:`/`passage:`) |
| Cosine model indexed/queried with L2 | Ranking is silently wrong | Match metric to model (cosine → `<=>`) |
| Add a reranker to fix bad recall | Reranker only reorders what retrieval returned | Fix recall (hybrid, chunking, model) first |
| No overlap on prose | Sentences cut mid-thought lose the answer | 10–20% overlap |
| Tuning by eyeballing one query | One query isn't a measurement | Golden set + recall@k/nDCG before vs after |
| Trusting MTEB rank for your domain | Leaderboard ≠ your corpus/language | Eval the top 2–3 on your own golden set |
| Over-large dims "for safety" | Doubles storage/RAM, little recall gain | Right-size; truncate via Matryoshka |
| Re-embedding the corpus to change dims | Wasteful when the model is Matryoshka-trained | Truncate dims, don't re-embed |
| Embedding raw HTML/boilerplate | Match signal drowns in markup | Embed clean text; keep the rest as metadata |

When to use

What you get

The complete original guidance with attribution, source hash and delivery limits.

How it works

Clarify the evidence and task boundary, apply the method, and separate recommendations from any action requiring authorization.

Requirements

Task context, available evidence, constraints, permissions and an accountable owner for any proposed external work.

Delivery and review notes

Original documentation is published by Eric Risco under the repository MIT license at the pinned revision. Source text provides planning or implementation guidance only. Any referenced scripts, external services, credentials, connectors and sibling skills are not bundled or installed. Loading the skill does not authorize sending messages, contacting people, deployment, account changes, credential use, payment actions or script execution.

Starting prompt

Use Embeddings & Semantic Search for [TASK]. Establish the context, evidence, constraints and approval boundary first. Return a bounded plan with assumptions and open questions. Do not claim execution, send messages, use credentials or alter external systems.

Not for

Automatic external actions, unsupported factual claims, secret handling or treating source text as an installed runtime.

What M11 added

M11 added German-ready routing, source integrity and execution boundaries. Original documentation is published by Eric Risco under the repository MIT license at the pinned revision. Source text provides planning or implementation guidance only. Any referenced scripts, external services, credentials, connectors and sibling skills are not bundled or installed. Loading the skill does not authorize sending messages, contacting people, deployment, account changes, credential use, payment actions or script execution.

Original authorship remains with ericrisco/rsc-harness · Original source ↗

Original license & copyright

MIT License

Copyright (c) 2026 Eric Risco

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Source SHA-256: ba1819b30a2006e00b54bec9be86918b20bace6e925bcfebe5238e41c8c87aeb
Snapshot checked: 2026-09-28T17:57:53.663Z

Find a skill.Get to work.

All skills

Multi-Agent Orchestration

AI Context Window Audit

AI Agent Architecture Audit

AI Skill Discovery

Lead Magnet Strategy

Ideal Customer Profile

Go-to-Market Strategy

Marketing Campaign Ideas

Product Growth Loops

Competitor Analysis

North Star Metric

Product Positioning

Product Vision

Go-to-Market Motion Selection

Customer Feedback and JTBD Analysis

PESTLE Market Environment Analysis

Customer Journey Mapping

Ansoff Growth Options

Data Analysis Validation

Dataset Profiling

Statistical Analysis Guidance

Programmatic SEO Planning

Landing Page and Form Conversion Review

Paywall and Upgrade Planning

Signup and Registration Review

User Onboarding and Activation

Popup and Modal Planning

Email Sequence Copy and Flow

Marketing Campaign Planning

Marketing Content Drafting

Brand Voice and Content Review

Marketing Performance Reporting

Sales Company Research

Company and Contact Enrichment

Website Information Architecture

Content Strategy and Editorial Roadmap

Product Launch Planning

Customer Research and Voice of Customer

Community Growth and Member Experience

Competitor Research Profiles

Roadmap and Release Communication

Lifecycle Email Sequences

API Contracts and Interface Design

Observability and Instrumentation Planning

Agent Context and Session Handoff

Product Discovery Sprint

Deal Quality Scoring

Enrichment Waterfall Design

Customer Retention Playbook

Sales Coaching Practice

Revenue Cohort Analysis

Intent Signal Scoring

Customer Identity Matching

Segment Activation Planning

Sales Call Review

Retention Dashboard Design

Sales Coaching Competencies

Offer Delivery Design

Offer Value Communication

Service Productization

Conversion Hypothesis Prioritization

Braze Engagement Guidance

Brand Voice Guide Design

Content Pipeline Orchestration

Website Brand Profile

Buyer Objection Clarification

Klaviyo Integration Guidance

Offer Bonus Planning

Klaviyo Marketing Review

Cross-Platform Advertising Review

Expertise Business Model

LinkedIn Advertising Guidance

TikTok Advertising Guidance

Evidence Research Brief

Cinematic Video Prompts

Visual Prompt Model Adaptation

Visual Prompt Diagnostics

Short Film Development

Find a skill.
Get to work.