Merge remote-tracking branch 'upstream/main' into cow-ea-readonly

Author: jorisvandenbossche

.github/workflows/unit-tests.yml

@@ -358,8 +358,6 @@ jobs:
 
       - name: Run Tests
         uses: ./.github/actions/run-tests
-        # TEMP allow this to fail until we fixed all test failures (related to chained assignment warnings)
-        continue-on-error: true
 
   # NOTE: this job must be kept in sync with the Pyodide build job in wheels.yml
   emscripten:

.github/workflows/wheels.yml

@@ -13,6 +13,8 @@
 name: Wheel builder
 
 on:
+  release:
+    types: [published]
   schedule:
   # 3:27 UTC every day
   - cron: "27 3 * * *"
@@ -101,7 +103,7 @@ jobs:
         - [macos-14, macosx_arm64]
         - [windows-2022, win_amd64]
         - [windows-11-arm, win_arm64]
-        python: [["cp311", "3.11"], ["cp312", "3.12"], ["cp313", "3.13"], ["cp313t", "3.13"]]
+        python: [["cp311", "3.11"], ["cp312", "3.12"], ["cp313", "3.13"], ["cp313t", "3.13"], ["cp314", "3.14"], ["cp314t", "3.14"]]
         include:
         # Build Pyodide wheels and upload them to Anaconda.org
         # NOTE: this job is similar to the one in unit-tests.yml except for the fact
@@ -216,3 +218,41 @@ jobs:
           source ci/upload_wheels.sh
           set_upload_vars
           upload_wheels
+
+  publish:
+    if: >
+      github.repository == 'pandas-dev/pandas' &&
+      github.event_name == 'release' &&
+      startsWith(github.ref, 'refs/tags/v')
+
+    needs:
+      - build_sdist
+      - build_wheels
+
+    runs-on: ubuntu-latest
+
+    environment:
+      name: pypi
+    permissions:
+      id-token: write         # OIDC for Trusted Publishing
+      contents: read
+
+    steps:
+      - name: Download all artefacts
+        uses: actions/download-artifact@v4
+        with:
+          path: dist          # everything lands in ./dist/**
+
+      - name: Collect files
+        run: |
+          mkdir -p upload
+          # skip any wheel that contains 'pyodide'
+          find dist -name '*pyodide*.whl' -prune -o \
+                    -name '*.whl'   -exec mv {} upload/ \;
+          find dist -name '*.tar.gz' -exec mv {} upload/ \;
+
+      - name: Publish to **PyPI** (Trusted Publishing)
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          packages-dir: upload
+          skip-existing: true

doc/source/development/contributing_codebase.rst

@@ -214,7 +214,8 @@ With custom types and inference this is not always possible so exceptions are ma
 pandas-specific types
 ~~~~~~~~~~~~~~~~~~~~~
 
-Commonly used types specific to pandas will appear in `pandas._typing <https://github.com/pandas-dev/pandas/blob/main/pandas/_typing.py>`_ and you should use these where applicable. This module is private for now but ultimately this should be exposed to third party libraries who want to implement type checking against pandas.
+Commonly used types specific to pandas will appear in `pandas._typing <https://github.com/pandas-dev/pandas/blob/main/pandas/_typing.py>`__ and you should use these where applicable. This module is private and is meant for pandas development.
+Types that are meant for user consumption should be exposed in `pandas.api.typing.aliases <https://github.com/pandas-dev/pandas/blob/main/pandas/api/typing/aliases.py>`__ and ideally added to the `pandas-stubs <https://github.com/pandas-dev/pandas-stubs>`__ project.
 
 For example, quite a few functions in pandas accept a ``dtype`` argument. This can be expressed as a string like ``"object"``, a ``numpy.dtype`` like ``np.int64`` or even a pandas ``ExtensionDtype`` like ``pd.CategoricalDtype``. Rather than burden the user with having to constantly annotate all of those options, this can simply be imported and reused from the pandas._typing module
 

doc/source/development/maintaining.rst

@@ -451,9 +451,10 @@ which will be triggered when the tag is pushed.
    - Set as the latest release: Leave checked, unless releasing a patch release for an older version
      (e.g. releasing 1.4.5 after 1.5 has been released)
 
-5. Upload wheels to PyPI::
-
-    twine upload pandas/dist/pandas-<version>*.{whl,tar.gz} --skip-existing
+5. Verify wheels are uploaded automatically by GitHub Actions
+   via `**Trusted Publishing** <https://docs.pypi.org/trusted-publishers/>`__
+   when the GitHub `*Release* <https://docs.github.com/en/repositories/releasing-projects-on-github/about-releases>`__
+   is published. Do not run ``twine upload`` manually.
 
 6. The GitHub release will after some hours trigger an
    `automated conda-forge PR <https://github.com/conda-forge/pandas-feedstock/pulls>`_.

doc/source/getting_started/index.rst

@@ -527,7 +527,6 @@ Data sets often contain more than just numerical data. pandas provides a wide ra
                 </div>
             </div>
         </div>
-
     </div>
     </div>
 

doc/source/getting_started/intro_tutorials/03_subset_data.rst

@@ -54,6 +54,8 @@ I’m interested in the age of the Titanic passengers.
 To select a single column, use square brackets ``[]`` with the column
 name of the column of interest.
 
+For more explanation, see `Brackets in Python and pandas <https://python-public-policy.afeld.me/en/columbia/brackets.html>`__.
+
 .. raw:: html
 
         </li>

doc/source/reference/aliases.rst

@@ -0,0 +1,96 @@
+{{ header }}
+
+.. _api.typing.aliases:
+
+======================================
+pandas typing aliases
+======================================
+
+**************
+Typing aliases
+**************
+
+.. currentmodule:: pandas.api.typing.aliases
+
+The typing declarations in ``pandas/_typing.py`` are considered private, and used
+by pandas developers for type checking of the pandas code base.  For users, it is
+highly recommended to use the ``pandas-stubs`` package that represents the officially
+supported type declarations for users of pandas.
+They are documented here for users who wish to use these declarations in their
+own python code that calls pandas or expects certain results.
+
+.. warning::
+
+    Note that the definitions and use cases of these aliases are subject to change without notice in any major, minor, or patch release of pandas.
+
+Each of these aliases listed in the table below can be found by importing them from :py:mod:`pandas.api.typing.aliases`.
+
+==================================== ================================================================
+Alias                                Meaning
+==================================== ================================================================
+:py:type:`AggFuncType`               Type of functions that can be passed to :meth:`agg` methods
+:py:type:`AlignJoin`                 Argument type for ``join`` in :meth:`DataFrame.join`
+:py:type:`AnyAll`                    Argument type for ``how`` in :meth:`dropna`
+:py:type:`AnyArrayLike`              Used to represent :class:`ExtensionArray`, ``numpy`` arrays, :class:`Index` and :class:`Series`
+:py:type:`ArrayLike`                 Used to represent :class:`ExtensionArray`, ``numpy`` arrays
+:py:type:`AstypeArg`                 Argument type in :meth:`astype`
+:py:type:`Axes`                      :py:type:`AnyArrayLike` plus sequences (not strings) and ``range``
+:py:type:`Axis`                      Argument type for ``axis`` in many methods
+:py:type:`CSVEngine`                 Argument type for ``engine`` in :meth:`DataFrame.read_csv`
+:py:type:`ColspaceArgType`           Argument type for ``colspace`` in :meth:`DataFrame.to_html`
+:py:type:`CompressionOptions`        Argument type for ``compression`` in all I/O output methods except :meth:`DataFrame.to_parquet`
+:py:type:`CorrelationMethod`         Argument type for ``correlation`` in :meth:`corr`
+:py:type:`DropKeep`                  Argument type for ``keep`` in :meth:`drop_duplicates`
+:py:type:`Dtype`                     Types as objects that can be used to specify dtypes
+:py:type:`DtypeArg`                  Argument type for ``dtype`` in various methods
+:py:type:`DtypeBackend`              Argument type for ``dtype_backend`` in various methods
+:py:type:`DtypeObj`                  Numpy dtypes and Extension dtypes
+:py:type:`ExcelWriterIfSheetExists`  Argument type for ``if_sheet_exists`` in :class:`ExcelWriter`
+:py:type:`ExcelWriterMergeCells`     Argument type for ``merge_cells`` in :meth:`to_excel`
+:py:type:`FilePath`                  Type of paths for files for I/O methods
+:py:type:`FillnaOptions`             Argument type for ``method`` in various methods where NA values are filled
+:py:type:`FloatFormatType`           Argument type for ``float_format`` in :meth:`to_string`
+:py:type:`FormattersType`            Argument type for ``formatters`` in :meth:`to_string`
+:py:type:`FromDictOrient`            Argument type for ``orient`` in :meth:`DataFrame.from_dict`
+:py:type:`HTMLFlavors`               Argument type for ``flavor`` in :meth:`pandas.read_html`
+:py:type:`IgnoreRaise`               Argument type for ``errors`` in multiple methods
+:py:type:`IndexLabel`                Argument type for ``level`` in multiple methods
+:py:type:`InterpolateOptions`        Argument type for ``interpolate`` in :meth:`interpolate`
+:py:type:`JSONEngine`                Argument type for ``engine`` in :meth:`read_json`
+:py:type:`JSONSerializable`          Argument type for the return type of a callable for argument ``default_handler`` in :meth:`to_json`
+:py:type:`JoinHow`                   Argument type for ``how`` in :meth:`pandas.merge_ordered` and for ``join`` in :meth:`Series.align`
+:py:type:`JoinValidate`              Argument type for ``validate`` in :meth:`DataFrame.join`
+:py:type:`MergeHow`                  Argument type for ``how`` in :meth:`merge`
+:py:type:`MergeValidate`             Argument type for ``validate`` in :meth:`merge`
+:py:type:`NaPosition`                Argument type for ``na_position`` in :meth:`sort_index` and :meth:`sort_values`
+:py:type:`NsmallestNlargestKeep`     Argument type for ``keep`` in :meth:`nlargest` and :meth:`nsmallest`
+:py:type:`OpenFileErrors`            Argument type for ``errors`` in :meth:`to_hdf` and :meth:`to_csv`
+:py:type:`Ordered`                   Return type for :py:attr:`ordered`` in :class:`CategoricalDtype` and :class:`Categorical`
+:py:type:`ParquetCompressionOptions` Argument type for ``compression`` in :meth:`DataFrame.to_parquet`
+:py:type:`QuantileInterpolation`     Argument type for ``interpolation`` in :meth:`quantile`
+:py:type:`ReadBuffer`                Additional argument type corresponding to buffers for various file reading methods
+:py:type:`ReadCsvBuffer`             Additional argument type corresponding to buffers for :meth:`pandas.read_csv`
+:py:type:`ReadPickleBuffer`          Additional argument type corresponding to buffers for :meth:`pandas.read_pickle`
+:py:type:`ReindexMethod`             Argument type for ``reindex`` in :meth:`reindex`
+:py:type:`Scalar`                    Types that can be stored in :class:`Series` with non-object dtype
+:py:type:`SequenceNotStr`            Used for arguments that require sequences, but not plain strings
+:py:type:`SliceType`                 Argument types for ``start`` and ``end`` in :meth:`Index.slice_locs`
+:py:type:`SortKind`                  Argument type for ``kind`` in :meth:`sort_index` and :meth:`sort_values`
+:py:type:`StorageOptions`            Argument type for ``storage_options`` in various file output methods
+:py:type:`Suffixes`                  Argument type for ``suffixes`` in :meth:`merge`, :meth:`compare` and :meth:`merge_ordered`
+:py:type:`TakeIndexer`               Argument type for ``indexer`` and ``indices`` in :meth:`take`
+:py:type:`TimeAmbiguous`             Argument type for ``ambiguous`` in time operations
+:py:type:`TimeGrouperOrigin`         Argument type for ``origin`` in :meth:`resample` and :class:`TimeGrouper`
+:py:type:`TimeNonexistent`           Argument type for ``nonexistent`` in time operations
+:py:type:`TimeUnit`                  Time unit argument and return type for :py:attr:`unit`, arguments ``unit`` and ``date_unit``
+:py:type:`TimedeltaConvertibleTypes` Argument type for ``offset`` in :meth:`resample`, ``halflife`` in :meth:`ewm` and ``start`` and ``end`` in :meth:`pandas.timedelta_range`
+:py:type:`TimestampConvertibleTypes` Argument type for ``origin`` in :meth:`resample` and :meth:`pandas.to_datetime`
+:py:type:`ToStataByteorder`          Argument type for ``byteorder`` in :meth:`DataFrame.to_stata`
+:py:type:`ToTimestampHow`            Argument type for ``how`` in :meth:`to_timestamp` and ``convention`` in :meth:`resample`
+:py:type:`UpdateJoin`                Argument type for ``join`` in :meth:`DataFrame.update`
+:py:type:`UsecolsArgType`            Argument type for ``usecols`` in :meth:`pandas.read_clipboard`, :meth:`pandas.read_csv` and :meth:`pandas.read_excel`
+:py:type:`WindowingRankType`         Argument type for ``method`` in :meth:`rank`` in rolling and expanding window operations
+:py:type:`WriteBuffer`               Additional argument type corresponding to buffers for various file output methods
+:py:type:`WriteExcelBuffer`          Additional argument type corresponding to buffers for :meth:`to_excel`
+:py:type:`XMLParsers`                Argument type for ``parser`` in :meth:`DataFrame.to_xml` and :meth:`pandas.read_xml`
+==================================== ================================================================

doc/source/reference/index.rst

@@ -55,6 +55,7 @@ to be stable.
    extensions
    testing
    missing_value
+   aliases
 
 .. This is to prevent warnings in the doc build. We don't want to encourage
 .. these methods.

doc/source/whatsnew/v2.3.3.rst

@@ -22,6 +22,7 @@ become the default string dtype in pandas 3.0. See
 
 Bug fixes
 ^^^^^^^^^
+- Fix bug in :meth:`Series.str.replace` using named capture groups (e.g., ``\g<name>``) with the Arrow-backed dtype would raise an error (:issue:`57636`)
 - Fix regression in ``~Series.str.contains``, ``~Series.str.match`` and ``~Series.str.fullmatch``
   with a compiled regex and custom flags (:issue:`62240`)
 

doc/source/whatsnew/v3.0.0.rst

@@ -220,11 +220,13 @@ Other enhancements
 - Implemented :meth:`Series.str.isascii` and :meth:`Series.str.isascii` (:issue:`59091`)
 - Improve the resulting dtypes in :meth:`DataFrame.where` and :meth:`DataFrame.mask` with :class:`ExtensionDtype` ``other`` (:issue:`62038`)
 - Improved deprecation message for offset aliases (:issue:`60820`)
+- Many type aliases are now exposed in the new submodule :py:mod:`pandas.api.typing.aliases` (:issue:`55231`)
 - Multiplying two :class:`DateOffset` objects will now raise a ``TypeError`` instead of a ``RecursionError`` (:issue:`59442`)
 - Restore support for reading Stata 104-format and enable reading 103-format dta files (:issue:`58554`)
 - Support passing a :class:`Iterable[Hashable]` input to :meth:`DataFrame.drop_duplicates` (:issue:`59237`)
 - Support reading Stata 102-format (Stata 1) dta files (:issue:`58978`)
 - Support reading Stata 110-format (Stata 7) dta files (:issue:`47176`)
+- Switched wheel upload to **PyPI Trusted Publishing** (OIDC) for release-tag pushes in ``wheels.yml``. (:issue:`61718`)
 -
 
 .. ---------------------------------------------------------------------------
@@ -603,6 +605,7 @@ Other API changes
   an empty ``RangeIndex`` or empty ``Index`` with object dtype when determining
   the dtype of the resulting Index (:issue:`60797`)
 - :class:`IncompatibleFrequency` now subclasses ``TypeError`` instead of ``ValueError``. As a result, joins with mismatched frequencies now cast to object like other non-comparable joins, and arithmetic with indexes with mismatched frequencies align (:issue:`55782`)
+- :meth:`CategoricalIndex.append` no longer attempts to cast different-dtype indexes to the caller's dtype (:issue:`41626`)
 - :meth:`ExtensionDtype.construct_array_type` is now a regular method instead of a ``classmethod`` (:issue:`58663`)
 - Comparison operations between :class:`Index` and :class:`Series` now consistently return :class:`Series` regardless of which object is on the left or right (:issue:`36759`)
 - Numpy functions like ``np.isinf`` that return a bool dtype when called on a :class:`Index` object now return a bool-dtype :class:`Index` instead of ``np.ndarray`` (:issue:`52676`)
@@ -974,8 +977,8 @@ Indexing
 - Bug in reindexing of :class:`DataFrame` with :class:`PeriodDtype` columns in case of consolidated block (:issue:`60980`, :issue:`60273`)
 - Bug in :meth:`DataFrame.loc.__getitem__` and :meth:`DataFrame.iloc.__getitem__` with a :class:`CategoricalDtype` column with integer categories raising when trying to index a row containing a ``NaN`` entry (:issue:`58954`)
 - Bug in :meth:`Index.__getitem__` incorrectly raising with a 0-dim ``np.ndarray`` key (:issue:`55601`)
+- Bug in adding new rows with :meth:`DataFrame.loc.__setitem__` or :class:`Series.loc.__setitem__` which failed to retain dtype on the object's index in some cases (:issue:`41626`)
 - Bug in indexing on a :class:`DatetimeIndex` with a ``timestamp[pyarrow]`` dtype or on a :class:`TimedeltaIndex` with a ``duration[pyarrow]`` dtype (:issue:`62277`)
--
 
 Missing
 ^^^^^^^
@@ -1083,6 +1086,7 @@ Reshaping
 - Bug in :meth:`DataFrame.join` when a :class:`DataFrame` with a :class:`MultiIndex` would raise an ``AssertionError`` when :attr:`MultiIndex.names` contained ``None``. (:issue:`58721`)
 - Bug in :meth:`DataFrame.merge` where merging on a column containing only ``NaN`` values resulted in an out-of-bounds array access (:issue:`59421`)
 - Bug in :meth:`DataFrame.unstack` producing incorrect results when ``sort=False`` (:issue:`54987`, :issue:`55516`)
+- Bug in :meth:`DataFrame.unstack` raising an error with indexes containing ``NaN`` with ``sort=False`` (:issue:`61221`)
 - Bug in :meth:`DataFrame.merge` when merging two :class:`DataFrame` on ``intc`` or ``uintc`` types on Windows (:issue:`60091`, :issue:`58713`)
 - Bug in :meth:`DataFrame.pivot_table` incorrectly subaggregating results when called without an ``index`` argument (:issue:`58722`)
 - Bug in :meth:`DataFrame.pivot_table` incorrectly ignoring the ``values`` argument when also supplied to the ``index`` or ``columns`` parameters (:issue:`57876`, :issue:`61292`)
@@ -1093,7 +1097,7 @@ Reshaping
 - Bug in :func:`melt` where calling with duplicate column names in ``id_vars`` raised a misleading ``AttributeError`` (:issue:`61475`)
 - Bug in :meth:`DataFrame.merge` where user-provided suffixes could result in duplicate column names if the resulting names matched existing columns. Now raises a :class:`MergeError` in such cases. (:issue:`61402`)
 - Bug in :meth:`DataFrame.merge` with :class:`CategoricalDtype` columns incorrectly raising ``RecursionError`` (:issue:`56376`)
--
+- Bug in :meth:`DataFrame.merge` with a ``float32`` index incorrectly casting the index to ``float64`` (:issue:`41626`)
 
 Sparse
 ^^^^^^
@@ -1158,6 +1162,7 @@ Other
 - Bug in printing a :class:`DataFrame` with a :class:`DataFrame` stored in :attr:`DataFrame.attrs` raised a ``ValueError`` (:issue:`60455`)
 - Bug in printing a :class:`Series` with a :class:`DataFrame` stored in :attr:`Series.attrs` raised a ``ValueError`` (:issue:`60568`)
 - Deprecated the keyword ``check_datetimelike_compat`` in :meth:`testing.assert_frame_equal` and :meth:`testing.assert_series_equal` (:issue:`55638`)
+- Fixed bug in the :meth:`Series.rank` with object dtype and extremely small float values (:issue:`62036`)
 - Fixed bug where the :class:`DataFrame` constructor misclassified array-like objects with a ``.name`` attribute as :class:`Series` or :class:`Index` (:issue:`61443`)
 - Fixed regression in :meth:`DataFrame.from_records` not initializing subclasses properly (:issue:`57008`)
 -