Cohere vs Mistral vs OpenAI: Real Enterprise RAG Cost per 1 Million Queries in 2026
The short answer: in a retrieval-augmented generation pipeline, the generation model is where the money goes, embeddings are close to free, and reranking becomes the second-largest line once the model is cheap. With every vendor given the same context and output lengths, an enterprise knowledge-search query (6,000 input tokens, 400 output tokens, one rerank) costs about $21,600 per million queries on Cohere's Command A, $18,550 on OpenAI's GPT-6 Sol and $6,150 on Mistral Large 3, at 1 million queries a month with an external vector database. Embeddings are under 0.1% of that. The spread comes from one column of the price sheet: token rates. Doubling the retrieved context raises the generation cost by 75–86%, while doubling the output length raises it by 14–25%.
My practical rule of thumb: don't spend your evaluation time agonizing over embedding models. Spend it on generation pricing and on how much context you stuff into each prompt, because that's where the bill actually gets written.
What each vendor bills
A enterprise Retrieval-Augmented Generation (RAG) system has three stages: embedding incoming user queries, reranking retrieved document passages, and generating the final synthesis with a frontier LLM. Each vendor prices these pieces a little differently, so here's the lay of the land.
Cohere offers specialized enterprise retrieval infrastructure. Command A runs at $2.50 input and $10.00 output per million tokens, while compact variants like Command R ($0.15/$0.60) and Command R7B ($0.0375/$0.15) cater to high-volume routing. Embed 4 costs $0.12 per million text tokens, and Rerank 4 Fast costs $2.00 per 1,000 searches (with Rerank 4 Pro at $2.50 per 1,000 searches). The detail worth remembering: Cohere bills reranking per search query rather than per token.
Mistral AI focuses on high-efficiency enterprise models. Mistral Large 3 lists at $0.50 input and $1.50 output per million tokens, Medium 3.5 at $1.50/$7.50, and Small 4 at $0.15/$0.60. Batch processing offers a 50% discount on supported models. The practical catch: native reranking endpoints and per-token embedding rates are frequently paired with specialized retrieval tools, so this analysis applies standard market retrieval rates alongside Mistral's generation tiers.
Mistral Large 4 lists at $1.36 input and $4.18 output per million tokens. For a RAG query with 6,000 input tokens (prompt plus retrieved context) and a 400-token answer, generation across 1 million queries costs 6,000 × $1.36 + 400 × $4.18 = $8,160 + $1,672 = $9,832 at list price, against $3,600 on Mistral Large 3 (6,000 × $0.50 + 400 × $1.50). Mistral's temporary launch sale price would halve the Large 4 figure to $4,916. These totals cover generation only, not retrieval or embeddings. For model details and other workloads, see Mistral Large 4 pricing.
OpenAI offers text-embedding-3-small at $0.02 per million tokens and text-embedding-3-large at $0.13. For generation, this guide benchmarks GPT-6 Sol at Standard short-context pricing: $2.00 per million input tokens, $0.20 cached input, and $10.00 output (Batch pricing is $1.00 input, $0.10 cached, and $5.00 output). The cost-sensitive GPT-6 Luna runs at $0.10 input and $0.50 output.
Normalized assumptions
To keep things fair, every stack gets the same query shapes, so differences reflect prices and not context choices. All of these are ILLUSTRATIVE.
| Shape | Input tokens (prompt + retrieved context) | Output tokens | Query embedding | Rerank |
|---|---|---|---|---|
| A: lightweight FAQ | 1,500 | 150 | 30 tokens | none |
| B: enterprise knowledge search | 6,000 | 400 | 30 tokens | 1 search |
| C: heavy document-analysis RAG | 30,000 | 1,200 | 30 tokens | 1 search |
Corpus: 1 million chunks of 512 tokens (512 million tokens), embedded once. That is $61.44 on Cohere Embed 4, $10.24 on text-embedding-3-small and $66.56 on text-embedding-3-large, and a similar amount for monthly refreshes. Amortized over 12 months it is about $5 a month or less.
Vendor-native vector and search products (Cohere Compass, OpenAI's hosted file search and vector stores, Mistral's document libraries) either had no dependable public price or were not captured. So the model uses an external vector database as an ILLUSTRATIVE fixed cost: one node at 730 hours of a $0.75 node-hour (about $547.50 a month) up to 1 million queries a month, and four nodes ($2,200) at 10 million. Rerank is priced at Cohere's $2.00 per 1,000 searches for all stacks.
Formulas: generation = (input × input rate + output × output rate) ÷ 1,000,000 per query; embedding = 30 × rate ÷ 1,000,000; rerank = $0.002 per query where used; vector = fixed monthly cost ÷ (queries per month ÷ 1,000,000).
Cost per 1 million queries, by shape
First, generation, embedding and rerank only (before the vector database):
| Shape | Cohere Command A | Mistral Large 3 ($0.50 / $1.50) | OpenAI GPT-6 Sol |
|---|---|---|---|
| A: FAQ | $5,254 | $979 | $4,501 |
| B: knowledge search | $21,004 | $5,604 | $18,001 |
| C: heavy document analysis | $89,004 | $18,804 | $74,001 |
Now including the external vector database at 1 million queries a month (about $548 plus $5 of ingestion per million queries):
| Shape | Cohere Command A | Mistral Large 3 ($0.50 / $1.50) | OpenAI GPT-6 Sol |
|---|---|---|---|
| A | $5,806 | $1,531 | $5,053 |
| B | $21,556 | $6,156 | $18,553 |
| C | $89,556 | $19,356 | $74,553 |
Effect of volume
For shape B, the vector layer is a fixed cost, so it only dominates at low volume:
| Queries per month | Cohere Command A | Mistral Large 3 ($0.50 / $1.50) | OpenAI GPT-6 Sol |
|---|---|---|---|
| 100,000 | $26,530 | $11,130 | $23,527 |
| 1 million | $21,556 | $6,156 | $18,553 |
| 10 million | $21,224 | $5,824 | $18,221 |
At 100,000 queries a month, the vector database alone is $5,475 per million queries, which is 49% of the Mistral cost. At 10 million, it is $220. If you're piloting at low volume, don't be alarmed by the per-query number. Part of it is just a fixed node waiting for traffic to show up.
Contribution by component
Here's where the money goes for shape B at 1 million queries a month:
| Component | Cohere | Mistral Large 3 ($0.50 / $1.50) | OpenAI GPT-6 Sol |
|---|---|---|---|
| Generation | 88.2% | 58.5% | 86.3% |
| Rerank | 9.3% | 32.5% | 10.8% |
| Vector database | 2.5% | 8.9% | 3.0% |
| Embeddings | 0.02% | 0.06% | 0.00% |
Embeddings are close to irrelevant on the query side. Rerank, which adds a flat $2,000 per million queries, is a third of the cost on the cheapest generator and about a tenth on Cohere and OpenAI.
Sensitivity to context and output length
Shape B, generation cost per million queries:
| Change | Cohere | Mistral Large 3 ($0.50 / $1.50) | OpenAI GPT-6 Sol |
|---|---|---|---|
| Baseline (6,000 in / 400 out) | $19,000 | $3,600 | $16,000 |
| Context doubled to 12,000 | $34,000 (+79%) | $6,600 (+83%) | $28,000 (+75%) |
| Output doubled to 800 | $23,000 (+21%) | $4,200 (+17%) | $20,000 (+25%) |
Input length matters most on stacks with cheap output, and output length matters most where the output rate is high relative to input. In real-world production, that means tightening your retrieval (fewer, better chunks) will usually save you more than any vendor switch.
Caching and batch change the picture further. If half of the input is a cacheable static prefix, shape B generation falls by about 34% on OpenAI at its listed $0.20 cached rate ($16,000 to $10,600 per million queries: 3,000 uncached input tokens at $2, 3,000 cached at $0.20 and 400 output tokens at $10, per million tokens). It falls by about 36% on Cohere if a cached rate of 10% of the input price applied (an ILLUSTRATIVE assumption, since none was captured). Batch pricing applies only to asynchronous workloads: OpenAI's Batch rates ($1.00 input, $5.00 output) take shape B generation to $8,000 per million queries, and a 50% discount reported for most Mistral models takes Mistral's to $1,800.
Budgeting traps
- Comparing headline token prices without normalizing context: A cheap model with bloated prompt context can easily cost more than a pricier model receiving tightly filtered chunks.
- Overlooking reranking unit economics: At $2.00 per 1,000 searches, reranking adds $2,000 per million queries. That's negligible in a small demo, but it can represent up to a third of total spend on low-cost models like Mistral.
- Assuming vendor-native retrieval is cheaper: Hosted search services and vector stores carry baseline monthly fees that can dwarf API inference costs at low volume.
- Relying on floating model aliases: Pointing production traffic to 'latest' aliases risks unexpected price increases when providers promote new flagship versions (nothing says "fun Monday" like a surprise invoice).
What to ask before you buy
Send the same 100 representative queries through each architecture, measuring billed input and output tokens directly. Verify exact reranking and embedding rates in writing, and find out whether prompt caching or batch discounts can be applied to your specific traffic pattern.
Is the pricier option worth it? If your team needs Cohere's retrieval tooling or OpenAI's ecosystem day to day, the premium can make sense. If you just need solid answers over your documents, the cheaper generator will likely get you there, so save your budget for the evaluation run that proves it.