Image search on Marqo is returning random results - how can I fix this?

Question

I configured a marqo index for image search as follows:

settings = { "treat_urls_and_pointers_as_images":True, # allows us to find an image file and index it "model":"ViT-L/14" } response = mq.create_index("my-multimodal-index", **settings)

I then added some documents:

mq.index("my-multimodal-index").add_documents(
[
{"image": "https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSBgKtI0_GMZMd1Zm4UGRO4E5RL5bN9kBO0FA&usqp=CAU", "model": "car 1"},
{"image": "https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcS8DWztNRxTzGiqHlE-bzlr4KFooWngblT78Gn7zOuC&s", "model": "car 2"},
{"image": "https://media.istockphoto.com/id/959391798/photo/3d-illustration-of-generic-compact-white-suv.jpg?s=612x612&w=0&k=20&c=2VqUK6i3gs1_ZbYKCEusyjhp-PqakZFsMM7xppqsqeU=", "model": "car 2"}
]) 

I searched the index as follows

mq.index("my-multimodal-index").search("an SUV")

However, the results I retrieved were inaccurate. All of the highlights were from text fields as opposed to images.

How can I get marqo to detect the match in the images as opposed to the irrelevant text?

Answer 1

Like most embedding models, the CLIP model you are using is not properly normalised in terms of the scores it returns when comparing text against text vs text to image. Therefore i suspect in this case its therefore always returning any text match above any image-to-text match.

Therefore, to get accurate results, you'll most likely need to set the searchable attributes to just search the image data (in this case your field labeled "image").

mq.index("my-multimodal-index").search("an SUV", searchable_attributes=["image"]) 

ref: https://docs.marqo.ai/0.0.10/API-Reference/search/