RAG Models: Scalability, Latency and Memory Usage Explained

Every RAG system works at 100,000 vectors. The demo is convincing, retrieval feels instant, and the answers cite the right documents. Then the corpus grows to 10 million chunks, traffic grows to a few hundred queries per second, and three separate walls appear at once: latency you can feel, a memory bill that dictates your architecture, and tail behaviour that no dashboard warned you about.

This guide is about those walls. Not “what is RAG” — if you are reading this, you have one running. It is about the engineering arithmetic of running retrieval-augmented generation at scale: where the milliseconds actually go, what a vector index really costs in RAM, when to quantize versus shard versus replicate, and which optimizations move p95 instead of just the median.

We are Triple Minds, an AI development and LLM model training company. We build and operate RAG systems for clients — and we also train the models inside them, which matters here more than you might expect, because some of the biggest infrastructure wins at scale come from the model side: a fine-tuned embedding model that produces smaller vectors, a distilled reranker that is ten times cheaper to run. Every number in this article is either arithmetic you can check yourself or a typical range we state honestly as a range.

1. The Architecture That Scales — and the One That Doesn’t

The RAG that fails at scale is the one where a single service does everything: embeds documents on upload, holds one monolithic index, embeds queries, searches, and calls the LLM — all in one process. It fails because the two halves of RAG have opposite performance personalities. Ingest wants throughput: big batches, GPU saturation, hours-long index builds. Serving wants latency: single-digit milliseconds, warm caches, predictable tails. Run them on the same nodes and every bulk upload becomes a p99 incident.

Here is the shape that survives production:

Six of these stages deserve their own numbers, and the rest of this article walks through them: chunking and embedding at ingest (①②), the sharded index (③), the semantic cache (④), the reranker (⑤), and generation (⑥). One architectural rule before the numbers: HNSW insertion typically costs on the order of ten times a query, and bulk backfills can inflate serve-path p99 by 2–10×. Keep index building off the serving nodes — build segments on a dedicated builder, then swap them in.

2. Latency: Where a RAG Request Actually Spends Its Time

A production RAG request is a six-stage serial waterfall: embed the query, search the index, rerank candidates, assemble the prompt, wait for the LLM’s first token, then stream the rest. The stages span more than two orders of magnitude in cost, and almost everyone optimizes the wrong ones. Draw the waterfall to scale and the problem is obvious:

Token generation is 70–90% of wall-clock time in virtually every deployment we have measured or audited. The reason is physics, not sloppy engineering: autoregressive decoding must stream essentially all model weights from GPU memory for every generated token, so decode speed is capped by memory bandwidth, not FLOPs:

tokens/sec ceiling ≈ HBM bandwidth / bytes moved per token (weights + KV cache)

Llama-3-8B fp16: 16 GB weights on an A100 (~2 TB/s HBM)
→ single-stream ceiling ≈ 2000/16 ≈ 125 tok/s; measured: 60–100 tok/s

Meanwhile the retrieval stack — the part teams spend months tuning — fits inside the first ~2% of the request. That does not make retrieval latency irrelevant; it makes it a perceived-latency and recall problem rather than a total-latency problem. With streaming, the user experiences your pre-LLM time plus time-to-first-token, so the numbers that matter are the ones before generation starts. Here is a realistic per-stage budget, with honest ranges:

Stage	Typical p50	Typical p95	What moves it
Query embedding (self-hosted GPU)	2–10 ms	15–40 ms	Model size, batching, cold starts
Query embedding (hosted API)	150–300 ms	300–500+ ms	Network + provider queueing — often the largest fixed cost in retrieval
ANN search (in-RAM HNSW, 1–10M vec)	0.4–5 ms	2–20 ms	efSearch, RAM residency, dimension
Managed vector DB (client-observed)	10–100 ms	100–250 ms	Network, TLS, serialization, multi-tenant queueing — server-side search itself is 1–10 ms
Rerank 100 pairs, MiniLM-class (~22M), GPU	50–80 ms	100–200 ms	Candidate count × sequence length
Rerank 100 pairs, 568M-class @ 512 tok, L4/A10	0.8–1.6 s	2–3 s	~58 TFLOPs of compute — model choice dominates
Prompt assembly + tokenization	1–10 ms	~15 ms	Negligible; budget it and move on
LLM TTFT (fast-tier hosted)	200–500 ms	1.6–3.2× p50	Prompt length (prefill), provider queueing
LLM TTFT (frontier hosted, long RAG prompt)	0.5–1.5 s	1–3 s	Same, at frontier scale
Decode (300–800 token answer)	4–20 s	—	Memory bandwidth, batch load, answer length

Three traps hide in that table. First, the embedding API tax: if you call a hosted embedding endpoint per query, you pay 150–300 ms of network and queueing to produce a vector your own GPU could compute in 5 ms — often 30–60× your actual ANN search time. Self-host the query encoder; it is a small model. Second, client-observed versus server-side latency: vendor dashboards report index time (1–10 ms), but your service experiences TLS, serialization of a 6 KB JSON query vector, load-balancer hops and tenant queueing. Measure from your client, keep connections pooled and gRPC where offered, and stay in-region — cross-region alone adds 30–150 ms. Third, the reranker model class: a MiniLM-class cross-encoder scores 100 pairs in 50–80 ms on a modest GPU, while a 568M-parameter reranker at 512 tokens needs roughly 0.8–1.6 seconds for the same batch — the FLOPs arithmetic (2 × 568M params × 512 tok × 100 pairs ≈ 58 TFLOPs) simply does not fit in 100 ms on an L4. Pick the reranker by latency budget, then close the quality gap with domain fine-tuning (section 6).

At p95 the picture inside the retrieval stack shifts — queueing effects concentrate in the reranker:

Tail latency: your users experience your p99, not your median

Two pieces of probability arithmetic explain most production latency pain. The first: a user session with 20 requests has a 64% chance of containing at least one request at or beyond your p95 (1 − 0.95²⁰ = 0.64). Users experience your tail far more often than your percentiles suggest. The second: the moment you shard, every query waits for the slowest shard:

P(at least one slow shard) = 1 − (1 − p)^S     p = per-shard slow probability

p = 1%:  S = 8 → 7.7%   S = 16 → 14.9%   S = 100 → 63.4%

Concretely: 16 shards, each with a 30 ms p99 and a 120 ms p99.9. The probability every shard answers within 30 ms is 0.99¹⁶ = 0.851 — so ~15% of queries wait on a straggler, and fleet-level p99 is set by per-shard p99.9 behaviour: roughly 100–120 ms, four times the per-shard figure. The standard countermeasure is hedged requests — after waiting about the per-shard p95, fire a duplicate to another replica and take whichever answers first. Google’s classic tail-at-scale result cut p99.9 from 1,800 ms to 74 ms for about 2% extra load; expect a few percent overhead when you trigger at p95. This is also the strongest argument for not sharding prematurely, which is where memory comes in.

3. Memory: The Bill Nobody Itemizes Until It Arrives

Vector memory is napkin arithmetic, and doing the napkin math early is the difference between a single-node system and an accidental six-node cluster. The baseline: a raw fp32 vector costs exactly d × 4 bytes. An HNSW graph adds roughly 8·M + 12 bytes per vector on top (4-byte neighbour IDs, M links per layer, layer 0 doubled):

Vectors:      N × d × 4 bytes                     (fp32)
HNSW graph:   N × (8·M + 12) bytes approx.        (M=16 → ~140 B/vec)
Real engines: × 1.1–1.2 allocator + metadata overhead

10M × 768-d, M=16:  30.72 GB vectors + 1.4 GB links ≈ 33–38 GB served

At 10M vectors that is an inconvenience. At 100M × 1536-d it is 614 GB of raw vectors — past any commodity node — and at 1B it is 3.07 TB. Which is why the single most important scaling decision is not “which vector database” but where you sit on the quantization ladder:

Level	Bytes/vector (768-d)	Compression	Typical recall@10 cost	Notes
fp32	3,072	1×	baseline	Default in most engines; rarely necessary
fp16	1,536	2×	≤0.1–0.3% — noise level	The free win; take it first
int8 SQ	768	4×	~0.5–3% with calibration	Claw back with a modest efSearch bump
PQ96	96	32×	2–8% raw; ≤1–2% with rescoring	Needs a full-precision refine tier
PQ64	64	48×	5–15% raw; dataset-dependent	Always pair with oversample + rescore
Binary	96	32×	4–10% raw on modern models; far worse on older ones	Popcount search is 10–40× faster; works best ≥1024-d

The pattern behind the whole ladder: compress the working set, keep a full-precision tier for rescoring. Search over the compressed representation, take 3–10× more candidates than you need, then re-score the top 100–200 against exact vectors — a 0.3–0.6 MB fetch that costs a millisecond or two and recovers most of the recall the compression gave away. Three worked examples from our sizing playbook, arithmetic included:

100M × 1536-d on one node. fp32 + HNSW (M=32): 100M × (6,144 + 268 + 8) B ≈ 642 GB — a 3–4-shard cluster. int8 SQ: 100M × (1,536 + 268 + 8) B ≈ 181 GB — one 256–384 GB node, with the 0.5–3% recall cost recovered by raising efSearch from 100 to ~200 (typically still 1–3 ms per query). Quantization turned a scatter-gather cluster into a single box.

1B × 768-d under 100 GB of RAM. IVF-PQ64 with nlist = 262,144: PQ codes 64 GB + IDs 8 GB + centroids 0.81 GB ≈ 73 GB — a 42× reduction from 3.07 TB flat. A query probes 64 lists (~244k codes, ~15.6M lookup-adds, 1–5 ms/core), then refines the top-200 against fp32 vectors on NVMe (0.6 MB, 1–3 ms parallelized). Raw IVF-PQ recall@10 of ~0.7–0.85 lands at ~0.93–0.98 after the refine step.

The disk-based alternative. DiskANN-style indexes keep PQ codes in RAM and the graph plus full vectors on NVMe: the published configuration serves 1B points from a single 64 GB machine at ≥95% recall@1 and ~5 ms mean latency, riding on 70–100 µs NVMe random reads. This is the step between “buy more RAM” and “shard” that most teams skip because nobody told them it exists.

The memory nobody budgets

• Replicas multiply everything. Cluster RAM = shards × replication factor × per-shard RAM. A 320 GB logical index at RF=3 is 960 GB of fleet memory.
• Index builds spike memory. Budget ~1.2–1.5× steady state on the index components during builds and compactions — and 2× if you rebuild blue-green next to the live index.
• The payload store is not free. Chunk text, metadata, and the document store often rival the vectors themselves; they just page better.
• Chunking is a memory multiplier. Halving your chunk size doubles N — every byte-per-vector decision upstream of the splitter is a corpus-wide multiplier downstream.
• Deletes don’t free memory. HNSW deletes are tombstones: zero bytes reclaimed until a vacuum/rebuild, and recall measurably degrades on heavily-churned graphs. Schedule rebuilds at a deleted-fraction trigger (~10–30% is the common band).

4. Scalability: Two Different Walls, Two Different Fixes

“We need to scale” hides two unrelated problems, and applying the wrong fix to the wrong wall is the most expensive mistake in production RAG.

The MEMORY wall: the index no longer fits the node.
  500M × 768-d fp32 ≈ 1.5 TB — a memory problem, no QPS problem.
  Fix order: quantize → disk-based index → then shard.

The QPS wall: the node no longer keeps up with traffic.
  10M × 768-d HNSW ≈ 5–8 CPU-ms/query → ~1,500–3,000 QPS per 16-vCPU node.
  Fix: replicate. Replication scales reads ~linearly.

The worked example that makes it stick: you need 3,000 QPS at p99 ≤ 100 ms, one node sustains 600 QPS with a 30 ms per-shard p99, and the index fits in RAM. The right answer is six replicas (ceil(3000/600) = 5, plus one for headroom and N−1 failure) — no fan-out, so fleet p99 stays ≈ 30 ms. The tempting wrong answer — “shard 8 ways for speed” — sends every query to 8 shards: P(a slow leg) = 1 − 0.99⁸ = 7.7%, your 100 ms budget is now roughly a p92, and you need hedging just to claw back what sharding cost you. Aggregate CPU per query barely improved. Shard for memory, replicate for traffic — sharding an index that fits in RAM buys tail pain and little else.

Corpus (768-d)	Vector RAM (fp32 / int8)	Sane default architecture
≤500k chunks	≤1.5 GB / 0.4 GB	Brute-force or HNSW in your Postgres (pgvector) — exact search is 1–5 ms here; don’t over-build
500k–10M	≤31 GB / 8 GB	Single-node HNSW (fp16/int8), replicated for traffic and availability
10M–100M	31–307 GB / 8–77 GB	Quantize first (int8; binary+rescore if ≥1024-d), one fat node or first shards; dedicated build node
100M–1B	0.3–3 TB / 77–768 GB	IVF-PQ or DiskANN tier, sharding with hedged requests, semantic sharding where the data clusters
>1B	>3 TB fp32-equivalent	Disk-first index + aggressive PQ in RAM, per-tenant routing, dedicated retrieval fleet

The scaling problems that aren’t about the index

Ingest throughput. Embedding generation spans two orders of magnitude by model size: a MiniLM-class encoder embeds 2,000–8,000 chunks/s on one GPU batched, a 110M-parameter encoder ~800–1,400/s (the FLOPs check: 2 × 110M × 512 tokens ≈ 113 GFLOPs per chunk), and a 7B-class embedder two orders less. Batching is the lever — batch 1 to batch 64–128 is commonly a 10–30× throughput difference. And mind the hosted-API ceiling: re-embedding 50M chunks (~17.5B tokens) under a 5M tokens/minute rate cap is a 2.4-day job minimum; the same corpus on four A100s running a 110M encoder is roughly five hours.

Embedding model migration. Vectors from different encoder versions are mutually incomparable — there is no partial upgrade. The pattern that works: re-embed to a new index on builder infrastructure, dual-write live upserts to both, shadow-query the new index to validate recall and answer quality, then flip the read alias atomically. Budget it like the infrastructure project it is, not a config change.

Multi-tenancy. Per-tenant collections are clean up to a few hundred tenants, then the per-collection overhead (memory, descriptors, optimizer threads) stops scaling; past ~1,000 tenants you want a shared collection with a tenant_id filter — but filtered ANN has a recall trap. Post-filtering returns k × selectivity results in expectation: at 0.1% selectivity you would need ~10,000 candidates to fill a top-10. Engines solve this with tenant-aware graph links (e.g. Qdrant’s payload-partitioned HNSW: m=0 with payload_m=16 on the tenant field); below roughly 10k–100k filtered vectors, just brute-force the filtered subset — a 10k × 768-d exact scan is ~15 MFLOPs, low single-digit milliseconds.

These are the load-bearing decisions in systems like database-connected AI assistants, where corpus growth is continuous and tenant isolation is contractual, not optional.

5. The Recall Dial: Paying Latency for Accuracy Deliberately

Every ANN index has one dial that trades recall against latency — efSearch on HNSW, nprobe on IVF — and most teams have never plotted theirs. The curve is logarithmic, which means both of the common defaults are wrong: the “fast” setting is usually leaving cheap recall on the table, and the “safe” setting is usually paying double latency for recall the reranker would have recovered anyway.

Two operational notes. First, build the gold set: a few hundred queries with exact (brute-force) top-k as ground truth, re-run on every index or model change — this is an afternoon of work that converts every tuning argument into a measurement. Recall failures downstream do not look like search bugs; they look like hallucinations, because the generator confidently answers from the wrong context. Second, the dial is dynamic: under load spikes, dropping efSearch from 400 to 64 sheds roughly 4–6× of search cost in exchange for a few recall points — a far better degradation mode than queueing. IVF is even more predictable: search work scales almost exactly with nprobe/nlist.

6. Serving Optimizations Ranked by What They Actually Move

Technique	Moves	Realistic effect	Cost / catch
Streaming + output-length control	Perceived latency	Perceived latency ≈ pre-LLM + TTFT; a capped 300-token answer halves total time vs an uncapped 600	None. Do this first. Reading speed is ~3.3–5 words/s (≈4.5–6.5 tok/s); any decode ≥8 tok/s outruns the reader
Context dieting (retrieve less, rerank harder)	TTFT + decode + concurrency	Prompt 10k → 2.5k tokens: ~4× less prefill, ~4× smaller KV cache — spending 100–200 ms of rerank to save ~1 s is a good trade	Requires a reranker you trust
Semantic cache	p50 (sometimes dramatically)	Hit ≈ 5–50 ms vs seconds — 20–100× on hits. Hit-rates are workload-shaped: FAQ/support-style traffic caches well (tens of %), long-tail conversational often <10%	Staleness + similarity-threshold tuning (0.90–0.97 band); measure YOUR hit-rate before crediting it
Self-hosted query encoder + embedding cache	p50, fixed cost	Removes the 150–300 ms API tax; cache entries are cheap (768 B int8 — 2M entries ≈ 1.7 GB)	One small GPU or even CPU
Hedged requests (sharded reads)	p99/p999	The canonical result: p999 1,800 → 74 ms for ~2% extra load; trigger at per-shard p95	Needs replicas + idempotent reads; cap hedge volume under overload
Parallel query-rewrite + speculative retrieval	p50	Saving = min(T_rewrite, T_retrieve) — real when an LLM rewrite (300–800 ms) overlaps retrieval	Wasted retrieval on the (10–40%) of requests the rewrite changes
Dynamic batching (embed/rerank/LLM)	Throughput, cost	~10–30× GPU throughput between batch 1 and 64–128	Adds up to one batch-window of queueing latency — size the window against your p95 budget
GPU ANN (CAGRA-class)	Throughput at extreme QPS	~18× QPS at very large batch sizes vs small-batch on the same hardware	Only pays at sustained thousands of QPS; batching latency again
gRPC + connection pooling	p50 a little, p99 more	Kills per-request TLS (+2 RTTs cold) and JSON bloat (6–8 KB → 3 KB per 768-d vector)	Engineering hygiene, not magic

The ranking logic is the waterfall from section 2: anything that shortens or hides generation (streaming, shorter answers, smaller prompts) moves seconds; anything inside retrieval moves milliseconds. The KV-cache arithmetic makes context dieting concrete: at ~128 KB of KV per token (Llama-3-8B-class), an 8k-token RAG context pins ~1 GB of GPU memory per in-flight request — halve the context and you double the concurrency ceiling of the same GPU while cutting TTFT roughly in half. Retrieval quality work and serving cost work are the same work.

7. The Model-Training Lever: Making the Infrastructure Problem Smaller

Everything so far accepts the embedding model as given and engineers around it. As a model training company, we usually attack the other side too, because the model chooses your constants:

• Fine-tuned small beats generic large in-domain. A 33M-parameter, 384-d encoder (bge-small class), contrastively fine-tuned on your domain’s query–document pairs, routinely matches or beats a generic 335M, 1024-d model (e5-large class) on that domain — typical published lifts from domain fine-tuning are 5–15 nDCG@10 points. The infrastructure consequence: 384-d instead of 1024-d is 2.67× less vector memory, bandwidth and search compute, at 10× fewer encoder parameters — before any quantization.
• Matryoshka (MRL) training makes dimensions elastic. An MRL-trained 1536-d model truncates to 512-d at a typical 1–4% recall cost — a 3× memory saving that multiplies with quantization. (Truncating a non-MRL model is catastrophic; this is a training-time property, not a config flag.)
• Quantization-aware embeddings make binary viable. Modern embedding models trained with quantization in mind lose ~4–10% raw recall under binary quantization — recoverable with oversample-and-rescore — where older models lose 15–40% and never recover. If binary indexes are in your future, that is a model-selection criterion today.
• Reranker distillation. Distilling a 568M cross-encoder into a MiniLM-class student on your domain’s pairs keeps most of the in-domain quality at roughly a tenth of the serving cost — turning the 0.8–1.6 s rerank line in the latency table back into a 50–80 ms one.

Run the numbers on a 100M-chunk corpus and the training project usually pays for itself in hardware within quarters: 100M × 1024-d fp32 is 410 GB of vectors; the fine-tuned 384-d model needs 154 GB — int8 takes it to 38 GB. The same corpus, the same recall target, a quarter of the fleet.

8. A Decision Framework You Can Argue With

Sizing questions we ask before any RAG engagement — cost planning falls out of the same arithmetic:

1. N and d, today and in 18 months. N × d × bytes is your memory bill; chunking strategy sets N, model choice sets d, and section 3 sets the bytes.
2. Which wall is closer — memory or QPS? They have different fixes (quantize/disk/shard vs replicate). Most teams hit memory first at tens of millions of vectors; traffic-heavy products hit QPS first at a few thousand queries per second per node.
3. What is the recall target, measured how? No gold set, no target — build the few-hundred-query benchmark before tuning anything.
4. What is the perceived-latency budget? Pre-LLM + TTFT is what users feel. Spend the budget on reranking (quality) rather than raw ANN speed once search is under ~20 ms.
5. Is the embedding model yours or rented? If the corpus is large and the domain is specific, fine-tuning is an infrastructure decision disguised as an ML decision (section 7).

Where Triple Minds Fits

We work on exactly this class of problem across three fronts. Consultation: a sizing and architecture audit — your corpus, traffic, and recall numbers pushed through the arithmetic in this article, with a written recommendation you can execute with or without us. Development: our AI development team builds production RAG systems — ingest pipelines, sharded retrieval, reranking, observability — including database-connected assistants where retrieval spans SQL and vectors. Model training: domain fine-tuned embedding models, MRL and quantization-aware training, and reranker distillation — the lever that shrinks the infrastructure rather than scaling it.

If your RAG system is approaching any of the walls in this article — or the invoice says it already hit one — a 30-minute architecture review is a cheap way to find out which of these levers applies to your numbers.

Frequently Asked Questions