For testing purposes I'm building a bot that clicks on buttons in the screen. I have a folder with all the possible buttons that needs to be pressed and the application being tested only has one button shown at a time. There is no situations where two or more buttons appear together on the screen.
My approach is to take a screenshot every few seconds and loop through all the possible buttons and try to find them on the screenshot. If the button is found the bot clicks the center of the button.
My issue is that even if the button is not present on the screenshot the template algorithm I'm using returns a false positive somewhere in the screen. Is there any way to make sure no false positives are returned unless the button exists on the image?
I'm using python with numpy and skimage. For the template matching I'm using skimage.feature.match_template.
I already tried with opencv using SIFT and all those things with not much success and a lot of issues with opencv itself.
P.S.: If you think there is a better way to solve the problem itself (testing an app by pressing buttons) they are welcome too.
Cheers
Edit 1:
These are the images:
Edit 2:
The output of the script (false positive)
As I realised later on, the only way to make sure the detected image is indeed the one you are searching for is to compute the difference between the detected one and the template and use a threshold to allow for imperfections.