Why is 'invalid value encountered in greater' warning thrown in python xarray for nan? Shouldn't the nan commute without any issues?

Question

The following is philosophical question aimed at figuring out why xarrays is the way that it is.

---

I'm having trouble figuring out the Xarrays way to do the following.

positive_values = values.where(values > 0)  

It follows x-arrays syntax, and computes what I want it to do using xarrays, but throws this Runtime Warning.

RuntimeWarning: invalid value encountered in greater if not reflexive  

My question is, how am I abusing Xarrays?

I'd like to make the case that nans are excellent in the sense that they commute across across operations. (Making it easy to spot or deal with missing data values)

IE.

value = np.nan + 1  
final_value = value/2  
#final_value evaluates to 'nan'

This make their representation in X-arrays very useful. xarrays may have missing data, but that shouldn't stop an operation across thousands of points.

Why doesn't > commute the nan without any issue? Should I be doing this some other way and ignore the error if this is the behavior that I want?

Answer 1

This was not an intentional design choice and should be fixed.

NumPy issues warnings when doing comparisons with NaN, because the result may be surprising, at least if you are not familiar with how comparisons with NaN work:

In [6]: np.array([1, -1, np.nan]) > 0
/Users/shoyer/conda/envs/xarray-dev/bin/ipython:1: RuntimeWarning: invalid value encountered in greater
  #!/Users/shoyer/conda/envs/xarray-dev/bin/python
Out[6]: array([ True, False, False], dtype=bool)

Arguably, it would be more intuitive if the result was something like [True, False, NA], but NumPy doesn't have integrated missing value support.

Until recently (in pandas 0.19), pandas silenced NumPy warnings of this type for all operations -- whether or not the operation was done with using pandas. Because xarray imports pandas, this meant these errors were silenced for xarray operations, too. Now, we'll need to add our own code to silence these warningss.

Update: This should be resolved as of xarray v0.10. If you are still encountering it, please file a bug at http://github.com/pydata/xarray/issues