Unexpected behavior in cut() with nullable Int64 dtype

[sdmccabe]

Code Sample

import pandas as pd
series = pd.Series([0, 1, 2, 3, 4, pd.np.nan, 6, 7], dtype='Int64')
breaks = [0, 2, 4, 6, 8]

breaks_cut = pd.cut(series, breaks)
breaks_cut

0           NaN
1    (0.0, 2.0]
2    (0.0, 2.0]
3    (2.0, 4.0]
4    (2.0, 4.0]
5           NaN
6    (0.0, 2.0]
7    (6.0, 8.0]
dtype: category
Categories (4, interval[int64]): [(0, 2] < (2, 4] < (4, 6] < (6, 8]]

Problem Description

When using the pd.Int64 nullable integer data type, pd.cut() unexpectedly bins the first non-np.nan value after an np.nan into the lowest interval. In the above example, the number 6 is binned into (0.0, 2.0].

Expected Output

0           NaN
1    (0.0, 2.0]
2    (0.0, 2.0]
3    (2.0, 4.0]
4    (2.0, 4.0]
5           NaN
6    (4.0, 6.0]
7    (6.0, 8.0]
dtype: category
Categories (4, interval[int64]): [(0, 2] < (2, 4] < (4, 6] < (6, 8]]

Note that using an IntervalIndex produces the expected output.

import pandas as pd
series = pd.Series([0, 1, 2, 3, 4, pd.np.nan, 6, 7], dtype='Int64')
breaks = [0, 2, 4, 6, 8]
intervals = [pd.Interval(x, y) for x, y in zip(breaks[:-1], breaks[1:])]
interval_index = pd.IntervalIndex(intervals)

interval_cut = pd.cut(series, interval_index)
interval_cut

Output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.6.final.0
python-bits      : 64
OS               : Linux
OS-release       : 5.0.0-37-generic
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 0.25.3
numpy            : 1.17.3
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 19.3.1
setuptools       : 44.0.0.post20200102
Cython           : None
pytest           : 5.3.2
hypothesis       : None
sphinx           : 2.3.1
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : 4.4.2
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.10.3
IPython          : 7.11.1
pandas_datareader: None
bs4              : 4.8.2
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.4.2
matplotlib       : 3.1.2
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : None
scipy            : 1.4.1
sqlalchemy       : None
tables           : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None

[jschendel]

Looks like there's an additional complication on master with the introduction of pd.NA and its use as the default NA for nullable integer dtypes:

In [1]: import numpy as np; import pandas as pd; pd.__version__
Out[1]: '0.26.0.dev0+1674.gbe1556c46'

In [2]: series = pd.Series([0, 1, 2, 3, 4, np.nan, 6, 7], dtype='Int64')

In [3]: breaks = [0, 2, 4, 6, 8]

In [4]: pd.cut(series, breaks)
---------------------------------------------------------------------------
TypeError: boolean value of NA is ambiguous

[parkerdgabel]

take

[parkerdgabel]

What should the expected behavior be?

[jschendel]

Probably need to more generically fix #30944 before attempting to resolve the issue of the output being incorrect. It's possible that fixing #30944 may also fix this, in which case we'd just want to add some tests here.

[phofl]

This works on main, may need tests

0           NaN
1    (0.0, 2.0]
2    (0.0, 2.0]
3    (2.0, 4.0]
4    (2.0, 4.0]
5           NaN
6    (4.0, 6.0]
7    (6.0, 8.0]

[kkangs0226]

take