Use ConversationalRetrievalChain without history/memory , only from one document

Question

I am trying a simple question answer scoring model .Each question has a document for answer . Issue comes when then answer is unrelated to current question but in the answer there are some text which matches previously answered question .

e.g

1. Question - What is capital of India ? Ans - Delhi is capital of India Score - 10/10

2. Question - Where was XYZ born ? Ans - XYZ was born in a small town in USA Score - 10/10

3. Question - Where is the name of president of US? Ans - I am going to Delhi

Now in question 3 , Answer is completely unrelated and wrong but it is finding some reference text "Delhi" in this case and scoring and giving reason based on that . So the question is how do I cleanup all previous history before submitting next answer , so that it refer only to that answer document and not historical .

Here is the code

    loader = S3FileLoader("s3folderpath",path)
    loader.load()
    
    index = VectorstoreIndexCreator().from_loaders([loader])

    chain = ConversationalRetrievalChain.from_llm(
    llm=ChatOpenAI(model="gpt-3.5-turbo"),
    retriever=index.vectorstore.as_retriever(search_kwargs={"k": 1}),
    result = chain({"question": query, "chat_history": []})
    print(result['answer'])

I even tried using RetrivalQA instead of ConversationalRetrievalChain so that chat history doesn't persist but even that is giving similar result , I know something fundamental is missing , please help and guide here .