Medical RAG Chatbot

The interesting problem is refusal

Retrieval is the easy part. Chunk a corpus, embed it, search it, hand the top results to a model — that is a weekend, and every RAG tutorial ends there. The part that decides whether the thing is usable is what happens when the corpus does not contain the answer.

A model handed thin or irrelevant context will still answer. It will answer fluently, in the register of the documents it did see, and a reader has no way to tell that from a grounded answer. In a medical context that is not a quality problem, it is the whole risk.

How it is put together

A PDF corpus is split into 500-character chunks, embedded with all-MiniLM-L6-v2 and stored in Pinecone with each chunk's source and page. At query time the two nearest chunks go to Llama 2 7B Chat (4-bit GGML, on CPU) with a prompt that restricts it to that context and gives it an exact sentence to use when the context lacks the answer. Every response comes back with the pages it drew on, shown in the chat UI, so any answer can be checked against its source. Flask serves it; LangChain wires retrieval and prompting together.

What the probing showed

Refusal here is prompt-level, and probing a local replica of the pipeline showed exactly where that bends: asked how COVID-19 is treated — a topic the corpus predates — the model said its context did not cover it and then answered from its own knowledge anyway.

It also showed why the obvious fix is not enough. A similarity threshold separates off-topic questions easily (top-1 cosine around 0.2), but out-of-corpus medical questions scored 0.51–0.64, overlapping the 0.55–0.85 of questions the corpus does answer. A threshold alone would pass exactly the dangerous case.

What I am building next

A groundedness check between generation and response, backed by a similarity floor for off-topic questions — and an evaluation set of in-scope, out-of-corpus and off-topic questions, so refusal quality becomes a measured rate rather than an observation.