BUG: GroupBy aggregation changes dtype of boolean column

[mfortin-adata]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
df = pd.DataFrame(data={'id': [1,1,1,1], 'my_bool': [True, False, False, True]})
df.groupby('id').diff() # returns incorrectly [NaN, -1, 0, 1], pandas 1.4 returns [NaN, True, False, True]
df['my_bool'].diff() # returns desirable output [NaN, True, False, True]

# other example
df2 = pd.DataFrame(data={'id': pd.Series([], dtype='int64'), 'my_bool': pd.Series([], dtype='bool')})
df2['my_bool'].diff() # correct result: empty Series
df2.groupby('id')['my_bool'].diff() # raises an exception TypeError: numpy boolean subtract, the `-` operator, is not supported

Issue Description

In Pandas 1.5and above (tested 1.5.2 via pip, and on main 2.0.0 via source install)
DataFrameGroupBy.diff:

• on boolean-typed columns returns an incorrectly typed Series,
• on empty boolean-typed columns triggers a numpy exception due to the incorrect operator - use.

Expected Behavior

First example

import pandas as pd
>>> df = pd.DataFrame(data={'id': [1,1,1,1], 'my_bool': [True, False, False, True]})
>>> df.groupby('id').diff()
  my_bool
0     NaN
1    True
2   False
3    True

Second example

Should not raise an exception

>>> df2 = pd.DataFrame(data={'id': pd.Series([], dtype='int64'), 'my_bool': pd.Series([], dtype="bool")})
>>> df2.groupby('id')['my_bool'].diff() # should return Series([], Name: my_bool, dtype: object)
Series([], Name: my_bool, dtype: bool)

Installed Versions

INSTALLED VERSIONS

commit : e16569a
python : 3.10.8.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.22621
machine : AMD64
processor : Intel64 Family 6 Model 167 Stepping 1, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 2.0.0.dev0+1077.ge16569a004
numpy : 1.23.5
pytz : 2022.7
dateutil : 2.8.2
setuptools : 63.2.0
pip : 22.3.1
Cython : 0.29.32
pytest : 7.2.0
hypothesis : 6.61.0
sphinx : 5.3.0
blosc : 1.11.1
feather : None
xlsxwriter : 3.0.6
lxml.etree : 4.9.2
html5lib : 1.1
pymysql : 1.0.2
psycopg2 : 2.9.5
jinja2 : 3.1.2
IPython : 8.8.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : 1.3.5
brotli :
fastparquet : 2022.12.0
fsspec : 2022.11.0
gcsfs : 2022.11.0
matplotlib : 3.6.2
numba : 0.56.4
numexpr : 2.8.4
odfpy : None
openpyxl : 3.0.10
pandas_gbq : None
pyarrow : 10.0.1
pyreadstat : 1.2.0
pyxlsb : 1.0.10
s3fs : 2022.11.0
scipy : 1.10.0
snappy :
sqlalchemy : 1.4.46
tables : 3.8.0
tabulate : 0.9.0
xarray : 2022.12.0
xlrd : 2.0.1
zstandard : 0.19.0
tzdata : 2022.7
qtpy : None
pyqt5 : None

[rhshadrach]

From the docstring of Series.diff:

For boolean dtypes, this uses operator.xor() rather than operator.sub(). The result is calculated according to current dtype in Series, however dtype of the result is always float64.

It seems this has not been implemented correctly. I'm in favor of how the docstring says this method should be implemented, so the result for df['my_bool'].diff() in example 1 would be:

0    NaN
1    1.0
2    0.0
3    1.0
Name: my_bool, dtype: float64

Agreed that groupby should switch to xor with Boolean inputs as well (or alternatively both shouldn't), and that empty inputs should work here (Example 2).

One other thought is that perhaps we should preserve e.g. Int64 dtypes since they can hold NA values (unlike int64). But I think that's best for a separate issue.

[jbrockmendel]

ATM groupby.diff ends with return obj - shifted. Instead we need to mirror what the DataFrame method does, which calls algorithms.diff, which for bool dtypes uses operator.xor.