How to crawl recursive structures using Kotlin coroutines?

Question

Given a tree like structure, and an operation to fetch the children of a node, e.g:

typealias NodeReference = URL 

data class Node(
     val data:Data,
     val childrenList:List<NodeReference>)

suspend fun resolve(nodeRef:NodeReference) : Node    

Do you know a blueprint to implement a crawler function having the signature

fun nodeList(rootNode:NodeReference) : List<Node> = 
    runBlocking(...) {
      ...
    }

returning all nodes of the tree using Kotlin coroutines?

Answer 1

To solve this problem efficiently, you should:

1. Resolve rootRef: NodeReference to get a rootNode: Node
2. Recursively asynchronously call nodeList method for all children of the rootNode
3. Await for the results
4. Merge the results and add the rootNode to them

Here is how you can do it:

suspend fun nodesList(rootRef: NodeReference): List<Node> = coroutineScope {
    val rootNode = resolve(rootRef) // 1
    rootNode.childrenList
        .map { async { nodesList(it) } } // 2
        .awaitAll() // 3
        .flatten() + rootNode // 4
}

---

If you want to use the current thread to execute nodesList, you can do it as follows:

fun nodesListBlocking(rootRef: NodeReference): List<Node> = runBlocking { nodesList(rootRef) }