I’m trying to create a RAG chatbot (without chat function) for my company but I am struggling with finding the right appraoch. I’ve been working on it for over a month and really need a solution soon.
THE SETTING: I have to build a system that answers very specific medical questions about pharmaceuticals. Potential sources include, among others, millions of studies from pubmed. The model we are using is GPT-4 and we have access to Azure.
THE GOAL: Get a comprehensive answer that starts broad but then narrows down and also resembles the current state of scientific literature.
THE PROBLEM: The system needs to provide a broad answer but also be factually accurate. Using only GPT-4 (no context provided by me, just question), I get a great answer. It’s relevant, starts broad and narrows down. However, factual accuracy can’t be verified and studies are almost always hallucinated. Using GPT-4 + RAG powered by cognitive search, the answer is often narrow and just a summary of the retrieved literature, sometimes includes semantically similar literature that isn’t relevant towards the answer, but is factually accurate and provides real scientific references. I basically need the general expertise of GPT-4, augmented with factual accuracy from our own sources.
POTENTIAL SOLUTION?: I thought of combining the output of both models, GPT-4 without RAG and the other with RAG. This should give me the best of both worlds with a broad answer that also features some relevant literature.
Does anyone have a potential solution that could solve my problem? I'd appreciate any help