Why does $null behave differently in ForEach-Object {} vs foreach()?

Question

Considering the following examples:

$null

foreach ($n in $null) {'This is a $null test'}
(no output)

$null | ForEach-Object {'This is a $null test'}
This is a $null test

$null -in $null
True

$null -contains $null
True

 

[int]1

foreach ($n in [int]1) {'Test'}
Test

[int]1 | ForEach-Object {'Test'}
Test

[int]1 -in [int]1
True

[int]1 -contains [int]1
True

Since $null is a scalar value that contains nothing, in the second $null example, $null will send a single instance of 'nothing' down the pipeline, which explains the output.

Why can't $null be iterated over as a collection even though $null -in $null returns True?

• The performance gains from not iterating over evaluated null objects/values seem obvious, but why does PowerShell treat scalar variables/values as collections?
• Does this behavior persist through all of .NET and the .NET Framework?
• Is there type coercion happening or do I have an fundamental misunderstanding about how collections/lists are structured or behave?

Answer 1

tl;dr

The asymmetry you've observed is definitely unfortunate - but is unlikely to get resolved, so as not to break backward compatibility (see the bottom section for a potential solution).

---

Both
<commandOrExpression> | ... (pipeline) and
foreach ($var in <commandOrExpression>) { ... }
are enumeration contexts:

• If <commandOrExpression> evaluates to an object that is enumerable,^[1] it is automatically enumerated and the enumerated elements are processed one by one.

• If it doesn't, the result is processed as itself; in a manner of speaking: it is treated like a single-element enumerable.

---

• $null, which corresponds to null in C#, is a special something that happens to represent a "scalar (single) nothing".

• This contrasts with the enumerable null - technically the [System.Management.Automation.Internal.AutomationNull]::Value singleton, aka "Automation Null" - which is what you get when a command produces no output.

  • It is a PS-specific representation of an enumerable that is nothing in itself and also has no elements: Its purpose is to signal "I represent nothing - I am not an object myself (unless I'm forced to act as one (as a scalar) in expressions, in which case I'll pretend to be $null), and enumerating me produces no elements".

• As an aside: as of PowerShell 7.4.x it is still nontrivial to determine if a given stored value is an actual $null or an enumerable null (a distinction that situationally does matter):

  # Only $true if $someValue contains the enumerable null 
  $null -eq $someValue -and @($someValue).Count -eq 0
  

  • The following future enhancement has been green-lit, but has not yet been implemented - see GitHub issue #13465:

    # !! Future enhancement - doesn't work as of v7.4.x
    $someValue -is [System.Management.Automation.AutomationNull]
    

---

Therefore, both the foreach loop and the pipeline:

• fortunately do not perform enumeration if the enumerable null is provided as input, given that its very purpose is to signal that there is nothing to enumerate.

  # Capture the output of a command that produces NO output.
  $noOutput = Get-Item *NoSuchFile*
  
  # No output, because the loop body is not executed.
  foreach ($obj in $noOutput) { 'This does not print.' }
  
  # Ditto.
  $noOutput | ForEach-Object { 'This does not print' }
  

• should process $null as itself, and therefore result in a single processing iteration with $null as the (non-)object to process:

  • Fortunately, this is how it works in the pipeline, as evidenced by your sample call to ForEach-Object cmdlet in your question.

  • Unfortunately, this is not how the foreach language statement behaves, which unexpectedly does not process the $null, as you've also observed.

---

As for the reason for this asymmetry:

• Up to version 2 of Windows PowerShell, assigning the output from a command that produced no output to a variable implicitly - and unexpectedly - converted the enumerable null (representing the lack of output) to $null, which caused the following statements not to be equivalent:

  # DIRECT processing of command output - the loop body is NOT entered.
  foreach ($obj in Get-Item *NoSuchFile*) { 'This does not print' }
  
  # INDIRECT processing of command output, via an *intermediate variable*.
  $output = Get-Item *NoSuchFile*
  # In v2, $output was $null, not the enumerable null,
  # causing the loop body to execute (once, with $obj containing $null)
  # In v3+, this is now - fortunately - equivalent to the above.
  foreach ($obj in $output) { 'This does not print' }
  

• In v3, two fundamental changes were made:

  • Variables now do store the enumerable null as such, without quietly converting it to $null.
  • It was decided to ignore $null as foreach input:^[2]

• Given the former v3 change, implementing the latter was not strictly necessary, but presumably:

  • The latter change was motivated by also making non-existent variables result in no processing.

  • The assumption was made that if a pipeline is used, intermediate and therefore potentially non-existent variables aren't needed, in which case the problem doesn't arise.

• However, the problem (still) does arise in the pipeline, namely if non-existent variables or non-existent properties of existing objects are used as input, because they evaluate to $null and therefore (justifiably) cause $null to be sent through the pipeline.
  Also, passing the enumerable null as an argument to a function or script's untyped parameter causes quiet conversion to $null.

---

It follows from the above that a potential solution that should only minimally impact backward compatibility, if at all, is the following:

• Make references to non-existent variables or properties default to the enumerable null rather than $null, which would implicitly prevent enumeration in both the pipeline and in foreach statements.
  Additionally, preserve the enumerable null as function / script arguments when binding to untyped parameters.

  • In expression contexts, the enumerable null behaves like $null, so something like $null -eq $noSuchVariable would continue to work.

• Make foreach too process actual $null values as such, as the pipeline already does, which would eliminate the asymmetry.

---

^{[1] Specifically, types that implement the IEnumerable interface or its generic counterpart are automatically enumerated, but there are a few exceptions: strings, dictionaries (types implementing IDictionary or its generic counterpart), and System.Xml.XmlNode instances. Additionally, System.Data.DataTable is enumerated too, despite not implementing IEnumerable itself, via its .Rows property. See this answer for details.}

^{[2] This change in behavior in v3 was announced on the PowerShell team blog in 2012: New V3 Language Features, section, section "ForEach statement does not iterate over $null",}