Keep track of completed Futures

Question

I'm spawning a large number (~100.000) http-request tasks as Futures in Scala. This takes a while, so I would like to be able to keep track of how many of those futures have successfully completed or failed by incrementing a global counter. I want, however, to avoid race conditions. Are there options in Scala to make an atomic counter? Or is there better way to go?

The code looks like this:

val futures = for(i <- 0 until nSteps) yield future {
    ...
    val content = blocking { ... http request ... }
    process(content)
}
Await.result(Future.sequence(futures),2 hours)

Answer 1

I would suggest using the standard Java AtomicInteger. You can increment it using the incrementAndGet() method, and obtain the current value via its get() method.

import java.util.concurrent.atomic.AtomicInteger
...
val completed = new AtomicInteger()
val futures = for(i <- 0 until nSteps) yield future {
    ...
    val content = blocking { ... http request ... }
    process(content)
    completed.incrementAndGet()
}
Await.result(Future.sequence(futures),2 hours)
...
print("Completed: %d" format completed.get)