Skip to content

BUG: stack doesn't preserve lower precision floating dtypes #51602

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from
Closed

BUG: stack doesn't preserve lower precision floating dtypes #51602

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
8437249
BUG: stack doesn't preserve lower precision floating dtypes
lithomas1 Feb 23, 2023
679c467
Merge branch 'main' of https://github.com/pandas-dev/pandas into stac…
lithomas1 Feb 23, 2023
397d5ce
Merge branch 'main' into stack-upcasts-nans
lithomas1 Feb 28, 2023
cae36be
Merge branch 'main' of https://github.com/pandas-dev/pandas into stac…
lithomas1 Mar 13, 2023
15eb161
simplify
lithomas1 Mar 13, 2023
fd68eba
Update reshape.py
lithomas1 Mar 15, 2023
b95a1cf
Merge branch 'main' into stack-upcasts-nans
lithomas1 Mar 15, 2023
6123430
Merge branch 'main' into stack-upcasts-nans
lithomas1 Mar 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion pandas/core/reshape/reshape.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -747,7 +747,18 @@ def _convert_level_number(level_num: int, columns: Index):
if slice_len != levsize:
chunk = this.loc[:, this.columns[loc]]
chunk.columns = level_vals_nan.take(chunk.columns.codes[-1])
value_slice = chunk.reindex(columns=level_vals_used).values
# Override fill value to prevent upcasting to float64
# if we have lower precision floats
common_type = find_common_type(chunk.dtypes.tolist())
if np.issubdtype(common_type, np.floating):
fill_value = common_type.type(np.nan)
elif is_extension_array_dtype(common_type):
fill_value = common_type.na_value
else:
fill_value = np.nan
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason for not using common_dtype.na_value?

Right now "Float32" would add np.nan, but this should be pd.NA, Same for Int64 for example

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default fill_value is np.nan for reindex, even for EAs. That's probably something to fix elsewhere, though (IIRC, the fill value gets converted somewhere in the EA methods, maybe take? It's been a while since I looked into it though).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But why can't we supply the correct one?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

value_slice = chunk.reindex(
columns=level_vals_used, fill_value=fill_value
).values
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is something we'll have to figure out at some point, right now this coerces EA dtypes to object. But ok for another pr since we had this problem before as well...

else:
subset = this.iloc[:, loc]
dtype = find_common_type(subset.dtypes.tolist())
Expand Down
13 changes: 13 additions & 0 deletions pandas/tests/frame/test_stack_unstack.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1481,6 +1481,19 @@ def test_stack(self, multiindex_year_month_day_dataframe_random_data):
expected = ymd.unstack(0).stack(0)
tm.assert_equal(result, expected)

def test_stack_nans_doesnt_upcast_float(self):
# GH 51059
df = DataFrame({("n1", "q1"): [1], ("n2", "q2"): [2]}, dtype="float32")
res = df.stack(level=0)
exp = DataFrame(
{
"q1": np.array([1.0, np.nan], dtype="float32"),
"q2": np.array([np.nan, 2.0], dtype="float32"),
},
index=MultiIndex.from_product([[0], ["n1", "n2"]]),
)
tm.assert_frame_equal(res, exp)

@pytest.mark.parametrize(
"idx, columns, exp_idx",
[
Expand Down