explanation on scala for comprehension with Option

Question

I have the following definition:

def f: Option[String] = Some(null)

the following evaluates to None:

for {x:String <- f} yield {
  x
}

the following evaluates to Some(null):

for {x <- f} yield {
  x
}

the following evaluates to Some(null):

f.map((x:String) => x)

I'm wondering why is there differences between them ?

Answer 1

The desugaring happens in the parser, so -Xprint:parser shows the difference:

$ scala -Xprint:parser
Welcome to Scala 2.12.1 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_111).
Type in expressions for evaluation. Or try :help.

scala> for (s: String <- (Some(null): Option[String])) yield s
[[syntax trees at end of                    parser]] // <console>
package $line3 {
  object $read extends scala.AnyRef {
    def <init>() = {
      super.<init>();
      ()
    };
    object $iw extends scala.AnyRef {
      def <init>() = {
        super.<init>();
        ()
      };
      object $iw extends scala.AnyRef {
        def <init>() = {
          super.<init>();
          ()
        };
        val res0 = (Some(null): Option[String]).withFilter(((check$ifrefutable$1) => check$ifrefutable$1: @scala.unchecked match {
  case (s @ (_: String)) => true
  case _ => false
})).map(((s: String) => s))
      }
    }
  }
}

res0: Option[String] = None

This surprises me because I thought filtering in this way was a feature people wanted but wasn't implemented.

The type pattern is just an instanceof test, so null fails that test.

Without the filter:

scala> for (s <- (Some(null): Option[String])) yield s
[[syntax trees at end of                    parser]] // <console>
package $line4 {
  object $read extends scala.AnyRef {
    def <init>() = {
      super.<init>();
      ()
    };
    object $iw extends scala.AnyRef {
      def <init>() = {
        super.<init>();
        ()
      };
      object $iw extends scala.AnyRef {
        def <init>() = {
          super.<init>();
          ()
        };
        val res1 = (Some(null): Option[String]).map(((s) => s))
      }
    }
  }
}

res1: Option[String] = Some(null)

In 2.9:

$ scala29
Welcome to Scala version 2.9.3 (OpenJDK 64-Bit Server VM, Java 1.6.0_38).
Type in expressions to have them evaluated.
Type :help for more information.

scala> for (s: String <- (Some(null): Option[String])) yield s
res0: Option[String] = Some(null)

so the filtering feature was added in 2.10.x.

Edit: so actually, this is what you don't get:

scala> for (s: String <- (Some("x"): Option[Any])) yield s
<console>:12: error: type mismatch;
 found   : String => String
 required: Any => ?
       for (s: String <- (Some("x"): Option[Any])) yield s
                      ^

Answer 2

Well... the first thing that I will say is that "When in Scala-land, stay as far as possible from a null-monster". They are dangerous.

Now...as to understand this behaviour, the first thing that you need to try in Scala-shell is following,

scala> val n = null
n: Null = null

So... In Scala null is an instance of this class Null.

Now... lets see what happens when we mix this null with Option,

scala> val nullOpt1 = Option(null)
nullOpt1: Option[Null] = None

scala> val nullOpt2 = Some(null)
nullOpt2: Some[Null] = Some(null)

Notice the difference... between the two... So basically Scala people thought that there may be people who want to wrap a null into an Option... so they allowed them to explicitly use Some.apply to wrap that null into an Option. Where as the use of Option.apply behave more "intelligently" and gives you a None when you try to wrap a null.

Now... lets first look at how map is implemented for an Option,

final def map[B](f: A => B): Option[B] =
  if (isEmpty) None else Some(f(this.get))

And now... it should be clear as to "why Some(null).map((x: String) => x) gives you Some(null) ?".

As for to understand the for cases... that will require a little understanding of how for is implemented for Option with the use of Repr's. And the behaviour will become clear.