DEPR: Deprecate downcast keyword for fillna

[phofl]

This came up recently in #40861
Depending on the way forward this would allow us to get downcast out from Block or be stricter about it

Thougts?

cc @jbrockmendel @jreback

[jbrockmendel]

The more I look at this, the wonkier it seems. #40861 implemented a test case:

df = pd.DataFrame({"col1": [1, np.nan]})

df.fillna({"col1": 2}, downcast={"col1": "int64"})

But similar calls still raise:

>>> df.fillna({"col1": 2}, downcast={"col1": np.int64})
[...]
TypeError: <class 'numpy.int64'>

>>> df.fillna(2, downcast={"col1": "int64"})
[...]
AssertionError: dtypes as dict is not supported yet

>>> df.fillna(2, downcast="int64")
[...]
ValueError: downcast must have a dictionary or 'infer' as its argument

Moreover, there are several places in NDFrame.fillna and Block.fillna where we ignore the 'downcast' keyword so end up using None. There's also a couple of layers of special-casing in Block._maybe_downcast and Block.downcast.

Notwithstanding the places where the downcast keyword is ignored, the status quo is roughly:

if downcast is None:
    if self.dtype == object:
        return datetime_centric_special_case()
    elif self.dtype.kind in ["f", "m", "M"]:
        return no_op()
    elif self.ndim == 1:
        downcast  = "infer"
        return do_inference()
    else:
        return no_op()

else:
    if self.dtype == object:
        return no_op()
    elif self.ndim == 1:
        return one_case_where_we_actually_respect_downcast_keyword()
    elif downcast != "infer":
          raise ValueErrorOrAssertionError
     else:
        return do_inference()

Some of this we can improve by actually respecting the keyword in more places and call that a bugfix. But the rest of the special-casing (object dtype, ndim==1 vs ndim==2) will likely change things and need a deprecation.

AFAICT the useful cases for downcasting after fillna (and interpolate, the other places from which _maybe_downcast is called) are 1) float->int and 2) object->not_object.

If this were greenfield, my inclination would be to make downcast take a bool, with False meaning no downcasting and True meaning "infer". (we could also take a dict but it would get unpacked before making it to the Block methods)

[jbrockmendel]

xref #11537

[jbrockmendel]

More questionable behavior: 1) we do the downcast even if the fillna is a no-op, 2) we ignore inplace=True and downcast anyway.

[jbrockmendel]

Based on a few days of trying to figure this out, along with #11537, I propose we deprecate the 'downcast' keyword in 'fillna', 'interpolate', 'ffill', 'bfill', with the future behavior being to always downcast for object dtype and never for anything else (the only relevant case for "anything else" being floats)

[jreback]

sounds ok to me

[jbrockmendel]

Having some trouble with the "do X instead" part because we don't have a method for downcasting floats (do we?)

[jreback]

we don't downcast floats as a general rule

[jbrockmendel]

update: deprecating the argument seems increasingly invasive. Leaning towards chipping away at the inconsistencies and calling them bugfixes.

[rhshadrach]

@jbrockmendel - I removed the deprecate tag; do you know if this issue remain open for tracking or if it is now duplicative of other issues?

[jbrockmendel]

I've completely lost track

[rhshadrach]

For the four examples in the top of #40988 (comment), the first remains okay, the last is now fixed, and the other two still raise. A quick search for downcast and fillna gives some issues, but none of the nature here.

[jbrockmendel]

@phofl what are your current thoughts here?

[phofl]

Ideally, we would get rid of downcast in fillna and others and provide a proper down casting method?

[jbrockmendel]

Sounds good to me. Want to take point on this?

[phofl]

Yep, can take care of this.

Would add deprecation note and new method for 2.1, so that we can immediately point users to an alternative.

[jbrockmendel]

I got annoyed enough by this to spend some more time on it. Thinking of deprecating in steps:

1. Deprecate any "downcast" other than [None, False, "infer"] (or dict with each value being one of these)
2. Deprecate behavior of downcasting that casts round-floats to ints (once enforced this would allow us to reuse Block.convert instead of Block._downcast_2d)
3. Deprecate downcast keyword altogether