How to return successfully parsed rows that converted into my case class

Question

I have a file, each row is a json array. I reading each line of the file, and trying to convert the rows into a json array, and then for each element I am converting to a case class using json spray.

I have this so far:

       for (line <- source.getLines().take(10)) {
          val jsonArr = line.parseJson.convertTo[JsArray]
          for (ele <- jsonArr.elements) {
            val tryUser = Try(ele.convertTo[User]) 
          }
        }

How could I convert this entire process into a single line statement?

val users: Seq[User] = source.getLines.take(10).map(line => line.parseJson.convertTo[JsonArray].elements.map(ele => Try(ele.convertTo[User])

The error is:

found : Iterator[Nothing]

Answer 1

Note: I used Scala 2.13.6 for all my examples.

---

There is a lot to unpack in these few lines of code. First of all, I'll share some code that we can use to generate some meaningful input to play around with.

object User {

  import scala.util.Random

  private def randomId: Int = Random.nextInt(900000) + 100000
  private def randomName: String = Iterator
    .continually(Random.nextInt(26) + 'a')
    .map(_.toChar)
    .take(6)
    .mkString

  def randomJson(): String = s"""{"id":$randomId,"name":"$randomName"}"""
  def randomJsonArray(size: Int): String =
    Iterator.continually(randomJson()).take(size).mkString("[", ",", "]")

}

final case class User(id: Int, name: String)

import scala.util.{Try, Success, Failure}

import spray.json._
import DefaultJsonProtocol._

implicit val UserFormat = jsonFormat2(User.apply)

This is just some scaffolding to define some User domain object and come up with a way to generate a JSON representation of an array of such objects so that we can then use a JSON library (spray-json in this case) to parse it back into what we want.

Now, going back to your question. This is a possible way to massage your data into its parsed representation. It may not fit 100% what your are trying to do, but there's some nuance in the data types involved and how they work:

val parsedUsers: Iterator[Try[User]] =
  for {
    line <- Iterator.continually(User.randomJsonArray(4)).take(10)
    element <- line.parseJson.convertTo[JsArray].elements
  } yield Try(element.convertTo[User])

First difference: notice that I use the for comprehension in a form in which the "outcome" of an iteration is not a side effect (for (something) { do something }) but an actual value for (something) yield { return a value }).

Second difference: I explicitly asked for an Iterator[Try[User]] rather than a Seq[User]. We can go very down into a rabbit hole on the topic of why the types are what they are here, but the simple explanation is that a for ... yield expression:

1. returns the same type as the one in the first line of the generation -- if you start with a val ns: Iterator[Int]; for (n<- ns) ... you'll get an iterator at the end
2. if you nest generators, they need to be of the same type as the "outermost" one

You can read more on for comprehensions on the Tour of Scala and the Scala Book.

One possible way of consuming this is the following:

for (user <- parsedUsers) {
  user match {
    case Success(user)  => println(s"parsed object $user")
    case Failure(error) => println(s"ERROR: '${error.getMessage}'")
  }

---

As for how to turn this into a "one liner", for comprehensions are syntactic sugar applied by the compiler which turns every nested call into a flatMap and the final one into map, as in the following example (which yields an equivalent result as the for comprehension above and very close to what the compiler does automatically):

val parsedUsers: Iterator[Try[User]] = Iterator
  .continually(User.randomJsonArray(4))
  .take(10)
  .flatMap(line =>
    line.parseJson
      .convertTo[JsArray]
      .elements
      .map(element => Try(element.convertTo[User]))
  )

One note that I would like to add is that you should be mindful of readability. Some teams prefer for comprehensions, others manually rolling out their own flatMap/map chains. Coders discretion is advised.

---

You can play around with this code here on Scastie (and here is the version with the flatMap/map calls).