Cost Comparator · RAG / knowledge assistant

What does a RAG knowledge assistant cost to run?

Retrieved context is the hidden cost driver in RAG. It is paid as input tokens on every single query.

RAG (retrieval-augmented generation) apps have a cost shape unlike a normal chatbot: input tokens dominate, not output tokens. Every query pulls back several chunks of retrieved context, typically 2,000 to 5,000 tokens, and that entire block gets billed as input on top of the user's actual question. A RAG app answering short questions can still rack up a large bill purely from the context it stuffs into every call.

This is also why RAG cost does not scale linearly with query volume the way a chatbot's does. Doubling your document corpus without changing retrieval depth (top-k) does not change per-query cost, but doubling top-k to improve answer quality directly doubles your input-token bill.

Who runs this workload

Internal knowledge base assistants over docs, wikis, or support tickets
Customer-facing "ask our docs" search-replacement features
Legal, medical, or compliance tools answering from a fixed document set

Cost drivers

What actually moves the bill.

Retrieval depth (top-k)

Each additional retrieved chunk is pure added input-token cost. Going from top-3 to top-8 retrieval can nearly triple the per-query bill.

Chunk size

Larger chunks improve context but cost more per retrieval. Most production RAG systems settle on 300 to 800 token chunks.

Model choice for the generation step

Since input tokens dominate, a model with a cheaper input rate (not just cheaper output) matters more here than in a typical chatbot.

FAQ

Common questions.

Why is my RAG app more expensive than my chatbot?

Because every query pays for retrieved context (typically 2,000 to 5,000 tokens) as input, on top of the actual question. A five-word question in a RAG system can cost as much as a full paragraph in a plain chatbot once retrieval is added.

How do I lower RAG API costs without hurting answer quality?

Reduce top-k (retrieved chunk count) before you reduce model quality, since retrieval depth is usually the larger lever. Re-ranking to keep only the 2 to 3 most relevant chunks instead of retrieving 8 broadly can cut input tokens 60% or more with minimal quality loss.

Does a bigger document corpus increase RAG costs?

Not directly. Corpus size affects your vector database and indexing cost, not your per-query LLM bill, which is driven by top-k and chunk size, not total documents indexed.

Other workloads