How to access AccessibilityNode and perform click of the secondary display in Android multi-display setup?

Question

I'm building an Android AccessibilityService that will support two displays. Idea is to have a draggable touch area on first display (something like a touchbar) with which the user can drag on to move the cursor that is displayed on the secondary display. Keep in mind that I want to run standalone apps on secondary display, so I'm not talking about Presentation mode for multi-display setup.

I managed to do this part by setting up a Service that draws over apps using android.permission.SYSTEM_ALERT_WINDOW permission (Settings.canDrawOverlays()). It draws a touch area on the first (touchbar) display, and it draws a cursor on the secondary display. As the user drags on the touchbar, the cursor moves I can get the coordinates of the cursor on the secondary display so I know where the cursor is located at any moment.

The issue now is to implement click interaction. My plan was to make that service as a AccessibilityService to allow me to perform clicks on the "presentation display" when user taps on the "touchbar display", but I cannot access the AccessibilityNode from the secondary display, I can only access the currently active display via "rootInActiveWindow".

Here's the current code (keep in mind this is inside the scope of a AccessibilityService):

private fun click(action: Int = AccessibilityNodeInfo.ACTION_CLICK) {
    val nodeInfo = if(displayManager.displays.size > 1){
        windowsOnAllDisplays.get(displayManager.displays[1].displayId, emptyList()).firstOrNull()?.root ?: rootInActiveWindow
    } else {
        rootInActiveWindow
    } ?: return
    val nearestNodeToMouse = findSmallestNodeAtPoint(nodeInfo, cursorLayout.x, cursorLayout.y)
    if (nearestNodeToMouse != null) {
        logNodeHierachy(nearestNodeToMouse, 0)
        nearestNodeToMouse.performAction(action)
    }
    nodeInfo.recycle()
}

private fun findSmallestNodeAtPoint(sourceNode: AccessibilityNodeInfo, x: Int, y: Int): AccessibilityNodeInfo? {
    val bounds = Rect()
    sourceNode.getBoundsInScreen(bounds)
    if (!bounds.contains(x, y)) {
        return null
    }
    for (i in 0 until sourceNode.childCount) {
        val child = sourceNode.getChild(i)
        if(child != null) {
            val nearestSmaller = findSmallestNodeAtPoint(child, x, y)
            if (nearestSmaller != null) {
                return nearestSmaller
            }
        }
    }
    return sourceNode
}

So, the issue is with "rootInActiveWindow" and "windowsOnAllDisplays" provider. For me, windowsOnAllDisplays always returns empty list (so I can't access a window on a secondary display), and "rootInActiveWindow" only returns the root of the currently active display (which is always the "touchbar" display).

Here's an example of the app on the emulator with two displays (at the bottom is the first (touchbar) display, on top is the secondary display with cursor).

Do you have any idea how can I access the AccessibilityNode from the secondary display to be able to perform click, or even a completely different idea on how to achieve this?

Answer 1

If someone hits the same issue, I got it :) Fetching windows on all displays works, I just needed to add additional flags in XML config of accessibility service.

Besides "flagDefault", needed to add "flagRetrieveInteractiveWindows" for it to return non-empty list of windows on all displays.

android:accessibilityFlags="flagDefault|flagRetrieveInteractiveWindows"

After that, windowsOnAllDisplays would return windows (and AccessibilityNodes) on secondary display.