Wait for a list of futures with composing Option in Scala

Question

I have to get a list of issues for each file of a given list from a REST API with Scala. I want to do the requests in parallel, and use the Dispatch library for this. My method is called from a Java framework and I have to wait at the end of this method for the result of all the futures to yield the overall result back to the framework. Here's my code:

def fetchResourceAsJson(filePath: String): dispatch.Future[json4s.JValue]

def extractLookupId(json: org.json4s.JValue): Option[String]

def findLookupId(filePath: String): Future[Option[String]] =
  for (json <- fetchResourceAsJson(filePath))
    yield extractLookupId(json)

def searchIssuesJson(lookupId: String): Future[json4s.JValue]

def extractIssues(json: org.json4s.JValue): Seq[Issue]

def findIssues(lookupId: String): Future[Seq[Issue]] =
  for (json <- searchIssuesJson(componentId))
    yield extractIssues(json)

def getFilePathsToProcess: List[String]


def thisIsCalledByJavaFramework(): java.util.Map[String, java.util.List[Issue]] = {
  val finalResultPromise = Promise[Map[String, Seq[Issue]]]()

  // (1) inferred type of issuesByFile not as expected, cannot get 
  // the type system happy, would like to have Seq[Future[(String, Seq[Issue])]]

  val issuesByFile = getFilePathsToProcess map { f => 
    findLookupId(f).flatMap { lookupId =>
     (f, findIssues(lookupId)) // I want to yield a tuple (String, Seq[Issue]) here
    }
  }
  Future.sequence(issuesByFile) onComplete {
    case Success(x) => finalResultPromise.success(x) // (2) how to return x here?
    case Failure(x) => // (3) how to return null from here?
  }

  //TODO transform finalResultPromise to Java Map
}

This code snippet has several issues. First, I'm not getting the type I would expect for issuesByFile (1). I would like to just ignore the result of findLookUpId if it is not able to find the lookUp ID (i.e., None). I've read in various tutorials that Future[Option[X]] is not easy to handle in function compositions and for expressions in Scala. So I'm also curious what the best practices are to handle these properly.

Second, I somehow have to wait for all futures to finish, but don't know how to return the result to the calling Java framework (2). Can I use a promise here to achieve this? If yes, how can I do it?

And last but not least, in case of any errors, I would just like to return null from thisIsCalledByJavaFramework but don't know how (3).

Any help is much appreciated.

Thanks, Michael

Answer 1

Several points:

• The first problem at (1) is that you don't handle the case where findLookupId returns None. You need to decide what to do in this case. Fail the whole process? Exclude that file from the list?
• The second problem at (1) is that findIssues will itself return a Future, which you need to map before you can build the result tuple
• There's a shortcut for map and then Future.sequence: Future.traverse
• If you cannot change the result type of the method because the Java interface is fixed and cannot be changed to support Futures itself you must wait for the Future to be completed. Use Await.ready or Await.result to do that.

Taking all that into account and choosing to ignore files for which no id could be found results in this code:

// `None` in an entry for a file means that no id could be found
def entryForFile(file: String): Future[(String, Option[Seq[Issue]])] =
  findLookupId(file).flatMap { 
    // the need for this kind of pattern match shows 
    // the difficulty of working with `Future[Option[T]]`
    case Some(id) ⇒ findIssues(id).map(issues ⇒ file -> Some(issues))
    case None     ⇒ Future.successful(file -> None)
  }

def thisIsCalledByJavaFramework(): java.util.Map[String, java.util.List[Issue]] = {
  val issuesByFile: Future[Seq[(String, Option[Seq[Issue]])]] =
    Future.traverse(getFilePathsToProcess)(entryForFile)

  import scala.collection.JavaConverters._
  try
    Await.result(issuesByFile, 10.seconds)
      .collect {
        // here we choose to ignore entries where no id could be found
        case (f, Some(issues)) ⇒ f -> issues
      }
      .toMap.mapValues(_.asJava).asJava
  catch {
    case NonFatal(_) ⇒ null
  }
}