Not Seeing Cursor Movement in Dockerized FastAPI with noVNC and pyautogui

Question

I'm setting up a Docker container to host a FastAPI application that interacts with a virtual display via noVNC, using pyautogui to control the mouse cursor. The container runs successfully, and I can access the FastAPI docs and use the API to supposedly move and click the mouse cursor. However, when I connect to the virtual display through noVNC, I cannot see the cursor move, even though the API responds with a success message.

My Dockerfile:

# Use an official Ubuntu base image
FROM ubuntu:20.04

# Avoid warnings by switching to noninteractive
ENV DEBIAN_FRONTEND=noninteractive
ENV DISPLAY=:1.0

# Install Ubuntu desktop, VNC server, and noVNC
RUN apt-get update && apt-get install -y \
    xfce4 \
    xfce4-goodies \
    tightvncserver \
    novnc \
    net-tools \
    python3-pip \
    python3-tk \      
    python3-dev \   
    && rm -rf /var/lib/apt/lists/*

# Setup VNC Server
RUN mkdir /root/.vnc
# Set your VNC password here (it should be changed)
RUN echo "your_vnc_password" | vncpasswd -f > /root/.vnc/passwd
RUN chmod 600 /root/.vnc/passwd

# Install Python packages
RUN pip3 install fastapi uvicorn pyautogui

# Copy your FastAPI application to the container
COPY ./app /app

# Expose VNC server port and FastAPI port
EXPOSE 5901 8000

# Set the USER environment variable
ENV USER=root
RUN touch /root/.Xauthority


# Set up the VNC server and noVNC
CMD ["sh", "-c", "vncserver :1 -geometry 1280x800 -depth 24 && websockify -D --web=/usr/share/novnc/ 8080 localhost:5901 && cd /app && uvicorn app:app --host 0.0.0.0 --port 8000"]

My app.py file:

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import pyautogui
import uvicorn

# Define a Pydantic model for the incoming data


class Coordinates(BaseModel):
    x: int
    y: int


app = FastAPI()


@app.post("/move_click/")
async def move_click(coords: Coordinates):
    try:
        # Move the cursor to the specified coordinates
        pyautogui.moveTo(coords.x, coords.y, duration=1)
        # Click at the current position
        pyautogui.click()
        return {"message": f"Cursor moved and clicked at ({coords.x}, {coords.y})"}
    except Exception as e:
        raise HTTPException(status_code=400, detail=str(e))

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

I've verified that the container is running and the FastAPI application is accessible. When I navigate to the noVNC interface at http://localhost:8080/vnc.html, I get a blank screen (as expected with no applications open), but interacting with the FastAPI endpoint to move the cursor doesn't visually change anything on the noVNC display.

• Is there a log or tool within Docker or noVNC that I should be checking to diagnose the issue?
• Could this be a limitation or configuration issue with pyautogui or noVNC when used in this context?
• What steps can I take to debug or resolve this issue so that cursor movement is visible within the noVNC display?