OpenCV / Python - constantly 'read' / 'look at' video game window

Question

I have a video game bot I am working on that looks at the screen of a video game and detects objects within that window.

My current solution is to take a screenshot of that window every x seconds, detect the objects in that screenshot, and take actions accordingly.

I know that open-CV works with web camera inputs and I was wondering if there anything like that I can do for a video game screen?

Please note that this is for purely educational purposes only. I am not going to use this bot for anything more than automation in a single player game - But I don't want to read program memory as I am trying to learn about image classification.

Thanks

Answer 1

ImageGrab is slow (as stated here Fastest way to take a screenshot with python on windows) but it suits your needs if you are only taking images every X seconds.

• Periodicity based on frames

import numpy as np
import cv2
from PIL import ImageGrab

# do something every 30 frames
n, read= 0, 30
while True:
    # (x, y, w, h), slect portion of the screen to screenshot
    img = ImageGrab.grab(bbox=(0, 1000, 100, 1100)) 
    img_np = np.array(img)
    frame = cv2.cvtColor(img_np, cv2.COLOR_BGR2GRAY)
    if n % read == 0:
        cv2.imshow("frame", frame)
    if cv2.waitKey(1) & 0Xff == ord('q'):
        break
    n+=1
    
cv2.destroyAllWindows()

• Periodicity based on time

import numpy as np
import cv2
from PIL import ImageGrab
from time import sleep

# do something every second
while True:
    # (x, y, w, h), slect portion of the screen to screenshot
    img = ImageGrab.grab(bbox=(0, 1000, 100, 1100))
    sleep(1)
    img_np = np.array(img)
    frame = cv2.cvtColor(img_np, cv2.COLOR_BGR2GRAY)
    cv2.imshow("frame", frame)
    if cv2.waitKey(1) & 0Xff == ord('q'):
        break
    
cv2.destroyAllWindows()