isnan | NumPy | Python Methods and Functions

** About: **

** numpy.isnan (array [, out]): ** Tests element by element if it is NaN or not, returns the result as a boolean array ...

**Parameters :**

array:[array_like] Input array or object whose elements, we need to test for infinityout:[ndarray, optional] Output array placed with result. Its type is preserved and it must be of the right shape to hold the output.

** Return: **

boolean array containing the result. For scalar input, the result is a new boolean with value True if the input is positive or negative infinity; otherwise the value is False. For array input, the result is a boolean array with the same shape as the input and the values are True where the corresponding element of the input is positive or negative infinity; elsewhere the values are False.

** Code 1: **

` ` |

** Output: **

Is NaN: False Is NaN: False Is NaN: True Is NaN: False Is NaN: False Checking for NaN: [0 0 0]

** Code 2: **

` ` |

** Outut: **

[[0 1 2 3] [4 5 6 7] [8 9 10 11] [12 13 14 15] [16 17 18 19]] Is NaN (Not a Number): [[False False False False ] [False False False False] [False False False False] [False False False False] [False False False False]] Is NaN (Not a Number): [[False] [True]]

** Links: **

https://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.isnan.html

** Notes: These codes will not work for online IDs. Please run them on your systems to see how they work. , This article is provided by **

I have an array of floats (some normal numbers, some nans) that is coming out of an apply on a pandas dataframe.

For some reason, numpy.isnan is failing on this array, however as shown below, each element is a float, numpy.isnan runs correctly on each element, the type of the variable is definitely a numpy array.

What"s going on?!

```
set([type(x) for x in tester])
Out[59]: {float}
tester
Out[60]:
array([-0.7000000000000001, nan, nan, nan, nan, nan, nan, nan, nan, nan,
nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
nan, nan], dtype=object)
set([type(x) for x in tester])
Out[61]: {float}
np.isnan(tester)
Traceback (most recent call last):
File "<ipython-input-62-e3638605b43c>", line 1, in <module>
np.isnan(tester)
TypeError: ufunc "isnan" not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ""safe""
set([np.isnan(x) for x in tester])
Out[65]: {False, True}
type(tester)
Out[66]: numpy.ndarray
```

`np.isnan`

can be applied to NumPy arrays of native dtype (such as np.float64):

```
In [99]: np.isnan(np.array([np.nan, 0], dtype=np.float64))
Out[99]: array([ True, False], dtype=bool)
```

but raises TypeError when applied to object arrays:

```
In [96]: np.isnan(np.array([np.nan, 0], dtype=object))
TypeError: ufunc "isnan" not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ""safe""
```

Since you have Pandas, you could use `pd.isnull`

instead -- it can accept NumPy arrays of object or native dtypes:

```
In [97]: pd.isnull(np.array([np.nan, 0], dtype=float))
Out[97]: array([ True, False], dtype=bool)
In [98]: pd.isnull(np.array([np.nan, 0], dtype=object))
Out[98]: array([ True, False], dtype=bool)
```

Note that `None`

is also considered a null value in object arrays.

There are lots of things I have seen make a model diverge.

Too high of a learning rate. You can often tell if this is the case if the loss begins to increase and then diverges to infinity.

I am not to familiar with the DNNClassifier but I am guessing it uses the categorical cross entropy cost function. This involves taking the log of the prediction which diverges as the prediction approaches zero. That is why people usually add a small epsilon value to the prediction to prevent this divergence. I am guessing the DNNClassifier probably does this or uses the tensorflow opp for it. Probably not the issue.

Other numerical stability issues can exist such as division by zero where adding the epsilon can help. Another less obvious one if the square root who"s derivative can diverge if not properly simplified when dealing with finite precision numbers. Yet again I doubt this is the issue in the case of the DNNClassifier.

You may have an issue with the input data. Try calling

`assert not np.any(np.isnan(x))`

on the input data to make sure you are not introducing the nan. Also make sure all of the target values are valid. Finally, make sure the data is properly normalized. You probably want to have the pixels in the range [-1, 1] and not [0, 255].The labels must be in the domain of the loss function, so if using a logarithmic-based loss function all labels must be non-negative (as noted by evan pu and the comments below).

```
import pandas as pd
import numpy as np
import math
# For single variable all three libraries return single boolean
x1 = float("nan")
print(f"It"s pd.isna: {pd.isna(x1)}")
print(f"It"s np.isnan: {np.isnan(x1)}}")
print(f"It"s math.isnan: {math.isnan(x1)}}")
```

**Output**

```
It"s pd.isna: True
It"s np.isnan: True
It"s math.isnan: True
```

This might happen inside scikit, and it depends on what you"re doing. I recommend reading the documentation for the functions you"re using. You might be using one which depends e.g. on your matrix being positive definite and not fulfilling that criteria.

**EDIT**: How could I miss that:

```
np.isnan(mat.any()) #and gets False
np.isfinite(mat.all()) #and gets True
```

is obviously wrong. Right would be:

```
np.any(np.isnan(mat))
```

and

```
np.all(np.isfinite(mat))
```

You want to check wheter any of the element is NaN, and not whether the return value of the `any`

function is a number...

Return

`True`

if x is a NaN (not a number), and`False`

otherwise.

```
>>> import math
>>> x = float("nan")
>>> math.isnan(x)
True
```

The usual way to test for a NaN is to see if it"s equal to itself:

```
def isNaN(num):
return num != num
```

If you"re using numpy for your arrays, you can also use

```
x = x[numpy.logical_not(numpy.isnan(x))]
```

Equivalently

```
x = x[~numpy.isnan(x)]
```

[Thanks to chbrown for the added shorthand]

**Explanation**

The inner function, `numpy.isnan`

returns a boolean/logical array which has the value `True`

everywhere that `x`

is not-a-number. As we want the opposite, we use the logical-not operator, `~`

to get an array with `True`

s everywhere that `x`

**is** a valid number.

Lastly we use this logical array to index into the original array `x`

, to retrieve just the non-NaN values.

This should be faster than iterating and will work regardless of shape.

```
numpy.isnan(myarray).any()
```

Edit: 30x faster:

```
import timeit
s = "import numpy;a = numpy.arange(10000.).reshape((100,100));a[10,10]=numpy.nan"
ms = [
"numpy.isnan(a).any()",
"any(numpy.isnan(x) for x in a.flatten())"]
for m in ms:
print " %.2f s" % timeit.Timer(m, s).timeit(1000), m
```

Results:

```
0.11 s numpy.isnan(a).any()
3.75 s any(numpy.isnan(x) for x in a.flatten())
```

Bonus: it works fine for non-array NumPy types:

```
>>> a = numpy.float64(42.)
>>> numpy.isnan(a).any()
False
>>> a = numpy.float64(numpy.nan)
>>> numpy.isnan(a).any()
True
```

`numpy.isnan(number)`

tells you if it"s `NaN`

or not.

```
np.count_nonzero(~np.isnan(data))
```

`~`

inverts the boolean matrix returned from `np.isnan`

.

`np.count_nonzero`

counts values that is not 0false. `.sum`

should give the same result. But maybe more clearly to use `count_nonzero`

Testing speed:

```
In [23]: data = np.random.random((10000,10000))
In [24]: data[[np.random.random_integers(0,10000, 100)],:][:, [np.random.random_integers(0,99, 100)]] = np.nan
In [25]: %timeit data.size - np.count_nonzero(np.isnan(data))
1 loops, best of 3: 309 ms per loop
In [26]: %timeit np.count_nonzero(~np.isnan(data))
1 loops, best of 3: 345 ms per loop
In [27]: %timeit data.size - np.isnan(data).sum()
1 loops, best of 3: 339 ms per loop
```

`data.size - np.count_nonzero(np.isnan(data))`

seems to barely be the fastest here. other data might give different relative speed results.

As the title promises, this book will introduce you to one of the world’s most popular programming languages: Python. It’s aimed at beginning programmers as well as more experienced programmers wh...

23/09/2020

The Pragmatic Programmer: Your Journey To Mastery, 20th Anniversary Edition (2nd Edition). The Pragmatic Programmer is one of those rare technical books that you will read, reread, and re-read over...

23/09/2021

Pandas 1.x Cookbook: Practical recipes for scientific computing, time series analysis, and exploratory data analysis using Python, 2nd Edition....

05/09/2021

In the last decade, we have seen the impact of exponential advances in technology on the way we work, shop, communicate, and think. At the heart of this change is our ability to collect and gain insig...

10/07/2020

X
# Submit new EBook