|
| 1 | +--- |
| 2 | +title: "NumPy 2.0: an evolutionary milestone" |
| 3 | +date: 2024-06-17 |
| 4 | +draft: false |
| 5 | +description: " |
| 6 | +We announce the release of NumPy 2.0, which brings major improvements in |
| 7 | +functionality and usability. Improvements to NumPy internals set the stage |
| 8 | +for future development. We discuss the motivation behind breaking changes |
| 9 | +and how those impact users, as well as some of the history of the 2.0 |
| 10 | +development process. |
| 11 | +" |
| 12 | +tags: ["News", "numpy"] |
| 13 | +displayInList: true |
| 14 | +author: ["NumPy Developers"] |
| 15 | +--- |
| 16 | + |
| 17 | +Eighteen years since the release of NumPy 1.0, we are thrilled to announce the |
| 18 | +launch of NumPy 2.0! This major release marks a significant milestone in the |
| 19 | +evolution of NumPy, bringing a wealth of enhancements and improvements to users, |
| 20 | +and setting the stage for future feature development. |
| 21 | + |
| 22 | +NumPy has improved and evolved over the past 18 years, with many old releases bringing |
| 23 | +significant performance, usability, and consistency improvements. |
| 24 | +That said, our approach for a long time has been to make only incremental changes while |
| 25 | +carefully managing backward compatibility. This approach minimizes user breakage, |
| 26 | +but also limits the scope of improvements that can be made, both to the API and its underlying implementation. |
| 27 | +Therefore, for this one-off major release, we are breaking backward |
| 28 | +compatibility to implement significant improvements in NumPy's type system. The |
| 29 | +type system is fundamental to NumPy, and major behavioral changes could not be |
| 30 | +made incrementally without mixing two different |
| 31 | +type systems, which would be a recipe for disaster. |
| 32 | + |
| 33 | +The journey to an actual 2.0 release has been long, and it was difficult to |
| 34 | +build the necessary momentum. In part, this may be because, for a time, the |
| 35 | +NumPy developers associated a NumPy 2.0 release with nothing less than a |
| 36 | +revolutionary rewrite of significant key pieces of the code base. Many of these |
| 37 | +rewrites and changes happened over the years, but because of backward |
| 38 | +compatibility concerns they remained largely invisible to the users. NumPy 2.0 |
| 39 | +is the culmination of these efforts, allowing us to discard some legacy |
| 40 | +ABI (Application Binary Interface) that prevented future improvements. |
| 41 | + |
| 42 | +Some major changes to NumPy internals—required for key features in |
| 43 | +2.0—have been in the works since 2019 at least. |
| 44 | +We started concrete plans for the 2.0 release more than a year ago, at a four hour |
| 45 | +long [public planning meeting](https://github.com/numpy/archive/tree/main/2.0_developer_meeting) |
| 46 | +in April 2023. Many of the key changes were proposed and discussed. The key goals |
| 47 | +we decided on there were perhaps even larger and more ambitious in scope than |
| 48 | +some of us expected. This also unlocked some extra energy - which has been great to see. |
| 49 | +After the meeting and over the course of the last year, NumPy enhancement |
| 50 | +proposals ([NEPs](https://numpy.org/neps/)) were written, |
| 51 | +reviewed, and implemented for each major change. |
| 52 | + |
| 53 | +Some key highlights are: |
| 54 | + |
| 55 | +- Cleaned-up and streamlined Python API ([NEP 52](https://numpy.org/neps/nep-0052-python-api-cleanup.html)): |
| 56 | + The Python API has undergone a thorough cleanup, making it easier to learn |
| 57 | + and use NumPy. The main namespace has been reduced by approximately 10%, and |
| 58 | + the more niche `numpy.lib` namespace has been reduced by about 80%, providing |
| 59 | + a clearer distinction between public and private API elements. |
| 60 | + |
| 61 | +- Improved scalar promotion rules: The scalar promotion rules have been |
| 62 | + updated, as proposed in [NEP 50](https://numpy.org/neps/nep-0050-scalar-promotion.html) |
| 63 | + addressing surprising behaviors in type promotion, e.g. with zero dimensional arrays. |
| 64 | + |
| 65 | +- Powerful new DType API and a new string dtype: NumPy 2.0 introduces a new API |
| 66 | + for implementing user-defined custom data types as proposed by |
| 67 | + [NEP 41](https://numpy.org/neps/nep-0041-improved-dtype-support.html). We used |
| 68 | + this new API to implement `StringDType`, offering efficient and painless |
| 69 | + support for variable length strings which was proposed in |
| 70 | + [NEP 55](https://numpy.org/neps/nep-0055-string_dtype.html). And it is our hope |
| 71 | + that enable future new data types with interesting new capabilities in the |
| 72 | + PyData ecosystem and in NumPy itself. |
| 73 | + |
| 74 | +- Windows compatibility enhancements: The default 32-bit integer representation |
| 75 | + on Windows has been updated to 64-bit on 64-bit architectures, addressing one |
| 76 | + of the most common problems with having NumPy work portably across operating |
| 77 | + systems. |
| 78 | + |
| 79 | +- Support for the Python array API standard: This is the first release to |
| 80 | + include full support for the array API standard (v2022.12), made possible |
| 81 | + by the new promotion rules, APIs, and API cleanup mentioned above. |
| 82 | + We also aligned existing APIs and behavior with the standard, |
| 83 | + as proposed in [NEP 56](https://numpy.org/neps/nep-0056-array-api-main-namespace.html). |
| 84 | + |
| 85 | +These are just some of the more impactful changes in behavior and usability. In addition, |
| 86 | +NumPy 2.0 contains significant performance and documentation improvements, |
| 87 | +and more much - for an extensive list of changes, see |
| 88 | +the [NumPy 2 release notes](https://numpy.org/devdocs/release/2.0.0-notes.html). |
| 89 | + |
| 90 | +To adopt this major release, users will likely need to adjust existing code, but we |
| 91 | +worked hard to strike a balance between improvements and ensuring that the |
| 92 | +transition to NumPy 2.0 is as seamless as possible. We wrote a comprehensive |
| 93 | +[migration guide](https://numpy.org/devdocs/numpy_2_0_migration_guide.html), |
| 94 | +and a [ruff plugin](https://numpy.org/devdocs/numpy_2_0_migration_guide.html#ruff-plugin) |
| 95 | +that helps to update Python code so it will work with both NumPy 1.x and |
| 96 | +NumPy 2.x. |
| 97 | + |
| 98 | +While we do require C API users to recompile their projects to support |
| 99 | +NumPy 2.0, we prepared for this in NumPy 1.25 already. The build process was |
| 100 | +simplified so that you can now compile with the latest NumPy version, |
| 101 | +and remain backward compatible. |
| 102 | +This means that projects build with NumPy 2.x are "magically" compatible with |
| 103 | +1.x. It also means that projects no longer need to build their binaries using |
| 104 | +the oldest supported version of NumPy. |
| 105 | + |
| 106 | +We knew throughout development that rolling out NumPy 2.0 |
| 107 | +would be (temporarily) disruptive, because of the backwards-incompatible API and |
| 108 | +ABI changes. We spent an extraordinary amount of effort communicating these |
| 109 | +changes, helping downstream projects adapt, tracking compatibility of popular |
| 110 | +open source projects (see, e.g., |
| 111 | +[numpy#26191](https://github.com/numpy/numpy/issues/26191), and completing the |
| 112 | +release process at limited pace to provide time for adoption. No |
| 113 | +doubt, the next few weeks will bring to light some new challenges, however we fully expect these |
| 114 | +to be manageable and well worth it in the long run. |
| 115 | + |
| 116 | +The NumPy 2.0 release is the result of a collaborative, largely volunteer, |
| 117 | +effort spanning many years and involving contributions from a diverse community |
| 118 | +of developers. In addition, many of the changes above would not have been |
| 119 | +possible without funders and institutional sponsors allowing several team members |
| 120 | +to work on NumPy as part of their day jobs. We'd like to acknowledge in particular: |
| 121 | +the Gordon and Betty Moore Foundation, the Alfred P. Sloan Foundation, |
| 122 | +NASA, NVIDIA, Quansight Labs, the Chan Zuckerberg Initiative, and Tidelift. |
| 123 | + |
| 124 | +We are excited about future improvements to NumPy, many of which will be |
| 125 | +possible due to changes in NumPy 2.0. See [the NumPy roadmap](https://numpy.org/neps/roadmap.html) |
| 126 | +for some features in the pipeline or on the wishlist. Let's |
| 127 | +continue working together to improve NumPy and the scientific Python and PyData |
| 128 | +ecosystem! |
0 commit comments