Best Local AI Models for Reranking for RAG Pipelines
Reranking retrieved chunks to improve RAG precision.
Run BGE Reranker v2 M3 unless you're English-only and latency-obsessed — then run Jina Tiny. Either way, adding any reranker to a top-50 retrieval typically improves answer quality more than upgrading your LLM one size.
Rerankers are the cheapest quality upgrade you can bolt onto a RAG pipeline. Your embedding model retrieves the top-50 chunks fast but fuzzily; a cross-encoder reranker then re-scores those 50 against the query with full attention and reorders them so the truly relevant ones reach the LLM. Because it only scores ~50 candidates per query (not your whole corpus), even a small reranker runs comfortably on CPU — this is the rare AI workload where you genuinely don't need a GPU.
Top picks
- #1
BGE Reranker v2 M30.568B · mit · min 1.6GB
The default choice — multilingual, strong accuracy, still CPU-friendly
BAAI's 568M-parameter cross-encoder is the standard open reranker for a reason: it handles 100+ languages, tops open-reranker quality benchmarks, and at FP16 fits in ~1.1 GB. On a modern CPU it scores 50 chunks in well under a second; any GPU with 2 GB VRAM makes it interactive. If you run one reranker, run this.
- #2
Jina Reranker Tiny EN0.033B · apache-2.0 · min 0.1GB
33M params — for high-QPS or edge deployments
Jina's tiny English-only reranker trades some accuracy for a 20x smaller footprint. It reranks in tens of milliseconds on CPU, which matters when you're scoring every query in a high-traffic service or running on a VPS with 1 GB of RAM. English-only corpora only — for anything multilingual, use BGE v2 M3.
- #3
Nomic Embed Text v1.50.137B · apache-2.0 · min 0.3GB
The retrieval half — pair a good embedder with your reranker
A reranker can only reorder what retrieval found, so pair it with a solid embedding model. Nomic Embed Text v1.5 (137M params) is a strong open choice with a clean license and Matryoshka dimensions — run it for the initial top-50 retrieval, then let the reranker fix the ordering.
Hardware guidance
This is the one RAG component you can run on anything. CPU-only: both rerankers work; expect <1s for 50 chunks with BGE, <100ms with Jina Tiny. 2-4 GB VRAM (GTX 1650, RTX 3050): BGE v2 M3 at FP16 becomes effectively instant. There is no benefit to more VRAM unless you're batch-reranking thousands of documents offline.
When to skip local
If your RAG stack already lives on a hosted vector DB with built-in reranking (Cohere Rerank via API, Pinecone's hosted rerankers), the API route costs pennies at low volume and saves you a service to operate. Go local when you have compliance constraints, high query volume, or latency budgets under ~200ms.
Need a guide for a different use case? See all 50 buyer's guides →