Linear interpolation fails on tz-aware Series

[xoolive]

The following code fails on 0.24.1, doesn't fail on 0.23.4 but result was wrong.

from datetime import datetime, timezone, timedelta

start = datetime(2019, 1, 1, tzinfo=timezone.utc)

df = pd.DataFrame.from_records(
    {
        "date": [start + timedelta(days=i) for i in range(5)],
        "value": list(range(5)),
        "ref_date": [start + timedelta(days=i) for i in range(5)],
    }
)

df.set_index("date").resample("1H").interpolate("linear")

Things work again if:

• one makes start tz-naive;
• one removes column ref_date.

[TomAugspurger]

Right now Block.interpolate is overloaded for .interpolate and .fillna, and dispatches based on whether the method is a fillna method or an interpolation method.

It looks like ExtensionBlock.interpolate, which is what DatetimeTZBlock inherits from, only handle the fillna side of things.

Long-term, we can make our interp methods work with extension arrays (or maybe add interpolate to the interface).

Short term, we can add a .interpolate to DatetimeTZBlock that calls Block.interpolate.

[xoolive]

Indeed, I added the following in my piece of code that uses the library and it makes things work.

from pandas.core.internals import Block, DatetimeTZBlock
DatetimeTZBlock.interpolate = Block.interpolate

I could see what you means about ExtensionBlock here

I could push a PR with the (ugly) fix you suggest+good comments around here but I don't feel good doing that...

Maybe if I understand why Block.interpolate cannot apply to ExtensionBlock, I could give it a shot. Do you have a few pointers for that?

[TomAugspurger]

Basically, we would need to ask the extension array to provide "good" values for our `missing.interpolate_1d` method. For datetimetz, we probably want 1. convert to i8values 2. perform the interpolation 3. convert back to original datetimetz We have similar issues discussing this general point.

…

On Fri, Mar 1, 2019 at 2:15 PM Xavier Olive ***@***.***> wrote: Indeed, I added the following in my piece of code that uses the library and it makes things work. from pandas.core.internals import Block, DatetimeTZBlock DatetimeTZBlock.interpolate = Block.interpolate I could see what you means about ExtensionBlock here <https://github.com/pandas-dev/pandas/blob/ea1d5f52a329ba24f1b4424f1e8f00cf9c8485c8/pandas/core/internals/blocks.py#L1824> I could push a PR with the (ugly) fix you suggest+good comments around here <https://github.com/pandas-dev/pandas/blob/ea1d5f52a329ba24f1b4424f1e8f00cf9c8485c8/pandas/core/internals/blocks.py#L2212> but I don't feel good doing that... Maybe if I understand why Block.interpolate cannot apply to ExtensionBlock, I could give it a shot. Do you have a few pointers for that? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#25508 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIkrsmEbr2JTbP7tftfZ015EgHP3Rks5vSYpegaJpZM4bZYJB> .

[xoolive]

OK, so if I understand well, the difference with tz-naive Series is that you now have to re-attach the timezone after converting back from i8values?

If so, I can try that, then have a look at similar issues.

[xoolive]

With pandas 1.0.0, I had to adapt my fix to this one...
Maybe it is not the proper way to do it.

def _tz_interpolate(data, *args, **kwargs):
    return data.astype(int).interpolate(*args, **kwargs).astype(data.dtype)

DatetimeTZBlock.interpolate = _tz_interpolate