Implement `#[init]` method attribute in `#[pymethods]` #4951

ahlinc · 2025-03-01T15:00:33Z

This allows to control objects initialization flow in the Rust code in case of inheritance from native Python types.

A simple example, you implement a class inherited from PyDict but want to build a custom constructor interface. With just the #[new] constructor it's not possible and the base PyDict __init__ method will be called too with all arguments that were passed to the constructor as a standard objects initialization behavior.

This allows to control objects initialization flow in the Rust code in case of inheritance from native Python types.

ahlinc · 2025-03-02T21:36:36Z

It seems that most of codecov/patch complains are unrelated to this PR changes.

mejrs

My apologies for the delay. Thanks for this, it seems useful 👍

Please add some more tests, for stuff like combinations of attributes:

    #[init]
    #[classmethod]

    #[init]
    #[new]

and a pyclass that inherits another pyclass, that init cannot be on functions and so on.

guide/src/class.md

mejrs · 2025-04-13T20:52:59Z

pyo3-macros-backend/src/method.rs

                        _kwargs: *mut #pyo3_path::ffi::PyObject
                    ) -> #pyo3_path::PyResult<*mut #pyo3_path::ffi::PyObject> {
                        use #pyo3_path::impl_::callback::IntoPyCallbackOutput;
-                        let function = #rust_name; // Shadow the function name to avoid #3017


Is this no longer needed?

pytests/src/pyclasses.rs

guide/src/class.md

mbway · 2025-04-20T20:15:52Z

btw I also have basic support for defining __init__ in my PR because it is required for defining a metaclass. I haven't compared the implementations but mine has the limitation of not supporting inheritance properly so this is probably better.

#4678 (comment)

There is a limitation with extending PyType that tp_new cannot be set so only tp_init is available. My workaroud for now is to initialize to Default::default() before calling the user's init function but this doesn't work with multi-level inheritance (currently I catch this case with an assert). I decided not to invest more into this because it might be entirely scrapped in favour of re-purposing #[new] in the case of a metaclass but that's open to discussion. I thought the init approach would at least be simpler to implement for my proof of concept.

davidhewitt

Thanks @ahlinc for the PR. Based on the long gap here without response to the review, I will probably take this over in the coming weeks to unblock #4678. I think that supporting __init__ properly may also have benefits for #5137. Please forgive me if you did intend to return to this 🙏

I think the general implementation here looks good, we can probably drop the #[init] attribute. I also wonder what would happen if users did not define #[new] and just defined __init__, based on the current implementation users would get an error saying the class cannot be instantiated. That might be confusing; I wonder if we should consider other options like allowing users to just define __init__ (and make the type unusable before __init__ is called)?

I think given the possible interplay with inheritance, what I might do next is write up an issue summarising the current state of __new__ and __init__, versus what we might like to have in the future.

guide/src/class.md

Co-authored-by: Bruno Kolenbrander <59372212+mejrs@users.noreply.github.com>

Icxolu · 2025-11-10T22:53:33Z

I've refactored this to current main, as well as #5551, for now I'm switching this to draft until that lands. While I was at it, I also implemented most of the review feedback from above.

Icxolu · 2025-11-14T22:07:29Z

With #5551 landed, I think this is now also in a reviewable state again.

davidhewitt

Implementation looks great, thank you! A few thoughts which might prompt doc adjustments or follow up features.

guide/src/class.md

davidhewitt · 2025-11-15T06:41:58Z

guide/src/class.md

+        kwargs: Option<&Bound<'_, PyDict>>,
+    ) -> PyResult<()> {
+        // call the super types __init__
+        slf.py_super()?


We might need to double check this and improve the py_super API, IIRC if slf is a subclass of MyDict then this infinitely recurses because of the way py_super gets the type from the object at runtime.

Yep, you are right here, I just tried this. If replaced by PySuper::new(&PyDict::type_object(slf.py()), slf)? it starts working as expected. Maybe this means that py_super should really look like this instead:

fn py_super<T: PyTypeInfo>(&self) -> PyResult<Bound<'py, PySuper>> { PySuper::new(&T::type_object(self.py()), self) }

This would require specifying the correct base to use.

I think quite possibly, yes. I am also not entirely sure what the interaction with module state (#5600) will be with our "global" type objects, so we may not want to rush to change py_super too fast.

To not block this PR, I guess we can:

use the direct PySuper::new call here

create follow up tasks to:

document py_super with the caveat on how it works

design a better replacement

Sounds good. Do we want to look into making this available on PyPy/GraalPy or defer that to followup as well?

guide/src/class.md

davidhewitt · 2025-11-15T06:51:27Z

guide/src/class/protocols.md

 PyO3 makes it possible for every magic method to be implemented in `#[pymethods]` just as they would be done in a regular Python class, with a few notable differences:

- `__new__` and `__init__` are replaced by the [`#[new]` attribute](../class.md#constructor).
+- `__new__` is replaced by the [`#[new]` attribute](../class.md#constructor).


Not a thing for here but I hope one day maybe we can have __new__ work exactly like Python (maybe it already does), the #[new] attribute might just make sense for some special cases or as a shorthand for #[pyo3(name = "__new__")].

Agreed, maybe we can even deprecate and remove it entirely at some point, so that all dunder methods work similarly and the constructor is less special.

I think there are possible edge cases about __new__ probably wanting to always take (cls, *args, **kwargs) signature (not sure?) which might allow #[new] to have some special semantics.

Given the churn downstream my feeling is that deprecating #[new] is unlikely to be justified and documenting it as a supported shorthand is probably good enough.

Sure, time will tell. Having it as an optional equivalent to #[pyo3(name = "__new__")] would still be quite nice.

davidhewitt · 2025-11-15T06:54:07Z

pytests/src/pyclasses.rs

    }
 }

+#[cfg(not(any(Py_LIMITED_API, PyPy, GraalPy)))] // Can't subclass native types on abi3 yet, PyPy/GraalPy don't support `py_super`


We might want to make py_super() work by calling the python builtin directly on those interpreters? Maybe a follow up task depending what else we decide regarding that API.

I've exposed it in 70981d7. If you agree I think this is ready to land then.

davidhewitt

Looks great, thanks everyone for getting __init__ support implemented!

davidhewitt · 2025-11-23T13:51:59Z

guide/src/class.md

+# #[cfg(not(Py_LIMITED_API))]
+use pyo3::types::{PyDict, PyTuple, PySuper};
+# #[cfg(not(Py_LIMITED_API))]
+use crate::pyo3::PyTypeInfo;


It will be so nice to solve #1344 one day 😂

davidhewitt · 2025-11-23T13:56:18Z

tests/ui/invalid_pyclass_init.stderr

👍 it'd be nice to tidy this error message up with a specialized return value somehow, this is totally fine to merge though.

davidhewitt · 2025-11-23T13:59:12Z

pytests/stubs/pyclasses.pyi


+@final
+class SubClassWithInit:
+    def __init__(self, /, *args, **kwargs) -> Any: ...


Looks like this type hint is wrong, I'll open an issue for this cc @Tpt

Indeed, it should be None. Will have a look at it in a follow-up when this MR is merged.

Icxolu · 2025-11-23T16:06:42Z

Looks like the PyPy failure is real. I've managed to reproduce it on linux (wsl) but not windows.

Icxolu · 2025-11-23T17:47:13Z

Looks like some kind of memory corruption that occurs when we try to run the base initializer:

self_.py_super()?.call_method("__init__", args.to_owned(), kwargs)?;

Not sure why tho 🤔

davidhewitt · 2025-11-24T20:17:29Z

Hmm, I will try to help investigate ASAP, probably Wednesday. This is probably good reason to delay #5646 just in case it needs a patch to the existing code.

Tpt · 2025-11-26T10:25:04Z

pytests/src/pyclasses.rs


+#[cfg(not(any(Py_LIMITED_API)))] // Can't subclass native types on abi3 yet
+#[pyclass(extends = PyDict)]
+struct SubClassWithInit;


Small nit: I would move this test with the other subclassing tests in subclassing.rs

davidhewitt · 2025-11-27T04:23:52Z

Hmm, I will try to help investigate ASAP, probably Wednesday. This is probably good reason to delay #5646 just in case it needs a patch to the existing code.

Not forgotten about this, have had some family stuff come up this week which is keeping me away from the linux desktop.

Icxolu · 2025-11-27T20:12:13Z

No worries. I tried to debug a bit yesterday, but without much success. At least the slf, args and kwags pointers look still fine when we call into Python. Might be something inside the object(s) that gets corrupted.

davidhewitt · 2025-11-28T10:53:09Z

From examining valgrind output, it seems to me that we have an issue where the PyPy GC is attempting to interact with the dict subclass after it's been deallocated. Presumably calling __init__ on the dict subclass is causing the PyPy GC to track it in some form.

I can replace the use of py_super with a handwritten call and I still get the crash, so it is definitely not related to py_super itself:

use pyo3::PyTypeInfo;

let mut full_args = vec![self_.clone().into_any()];
for arg in args {
    full_args.push(arg);
}

let args = PyTuple::new(self_.py(), full_args)?;
PyDict::type_object(self_.py()).call_method("__init__", args, kwargs)?;

I can also trigger the crash more reliably by forcing gc collection after deleting the dict subclass:

$ git diff tests
diff --git a/pytests/tests/test_pyclasses.py b/pytests/tests/test_pyclasses.py
index e2ef59ef0..2db6db71b 100644
--- a/pytests/tests/test_pyclasses.py
+++ b/pytests/tests/test_pyclasses.py
@@ -1,3 +1,4 @@
+import gc
 import platform
 import sys
 from typing import Type
@@ -177,3 +178,9 @@ def test_class_init_method():

     d = SubClassWithInit({"a": 1}, b=2)
     assert d == {"__init__": True, "a": 1, "b": 2}
+
+    del d
+    for i in range(10):
+        gc.collect(0)
+        gc.collect(1)
+        gc.collect(2)

Not yet sure whether the issue is in our implementation of classes on PyPy, or an upstream PyPy issue. I'll try to minimise the repro.

davidhewitt · 2025-11-28T12:10:27Z

#5653 fixes 🎉

* Implement #[init] method attribute in #[pymethods] This allows to control objects initialization flow in the Rust code in case of inheritance from native Python types. * Apply suggestions from code review Co-authored-by: Bruno Kolenbrander <59372212+mejrs@users.noreply.github.com> * review feedback * expose `PySuper` on PyPy, GraalPy and ABI3 --------- Co-authored-by: David Hewitt <mail@davidhewitt.dev> Co-authored-by: Bruno Kolenbrander <59372212+mejrs@users.noreply.github.com> Co-authored-by: Icxolu <10486322+Icxolu@users.noreply.github.com>

Icxolu · 2025-11-28T17:20:41Z

💯 Thanks for investigating! I suspected it had something to do with the subclassing, but only manifested itself due to the new __init__ call. I'll rebase this once #5653 lands.

Icxolu · 2025-11-29T21:15:26Z

Woohoo 🎉 all green ✅

ahlinc force-pushed the init-method branch 10 times, most recently from f256dc8 to 6dbd5f4 Compare March 2, 2025 20:52

Implement #[init] method attribute in #[pymethods]

a06a2ad

This allows to control objects initialization flow in the Rust code in case of inheritance from native Python types.

ahlinc force-pushed the init-method branch from 6dbd5f4 to a06a2ad Compare March 2, 2025 21:11

ahlinc mentioned this pull request Mar 7, 2025

release: 0.24.0 #4959

Merged

mejrs requested changes Apr 13, 2025

View reviewed changes

mbway mentioned this pull request Aug 20, 2025

Opaque Object Layout #4678

Open

davidhewitt reviewed Sep 21, 2025

View reviewed changes

guide/src/class.md Outdated Show resolved Hide resolved

Apply suggestions from code review

28caee8

Co-authored-by: Bruno Kolenbrander <59372212+mejrs@users.noreply.github.com>

davidhewitt mentioned this pull request Oct 22, 2025

refactor #[new] code generation #5551

Merged

davidhewitt mentioned this pull request Nov 4, 2025

0.28 Release #5599

Open

Icxolu marked this pull request as draft November 10, 2025 22:47

Icxolu force-pushed the init-method branch 2 times, most recently from 4a56a0e to 5e8c8e2 Compare November 12, 2025 18:12

refactor

4753b05

Icxolu force-pushed the init-method branch from 5e8c8e2 to 4753b05 Compare November 14, 2025 22:03

Icxolu marked this pull request as ready for review November 14, 2025 22:06

davidhewitt approved these changes Nov 15, 2025

View reviewed changes

davidhewitt approved these changes Nov 23, 2025

View reviewed changes

davidhewitt mentioned this pull request Nov 23, 2025

Return value hint for __init__ is wrong, should be None #5645

Open

davidhewitt added this pull request to the merge queue Nov 23, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Nov 23, 2025

Icxolu added this pull request to the merge queue Nov 23, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Nov 23, 2025

Tpt reviewed Nov 26, 2025

View reviewed changes

davidhewitt mentioned this pull request Nov 28, 2025

fix FFI definition of PyDictObject on PyPy #5653

Merged

bschoenmaeckers added this pull request to the merge queue Nov 28, 2025

bschoenmaeckers removed this pull request from the merge queue due to a manual request Nov 28, 2025

davidhewitt added the CI-build-full label Nov 28, 2025

Icxolu added 2 commits November 29, 2025 16:01

merge main

45f29d1

expose PySuper on PyPy, GraalPy and ABI3

ab43f5f

Icxolu force-pushed the init-method branch 2 times, most recently from d848e7c to b078588 Compare November 29, 2025 18:39

fix graalpy issue

e5902d7

Icxolu force-pushed the init-method branch from b078588 to e5902d7 Compare November 29, 2025 19:26

bschoenmaeckers added this pull request to the merge queue Nov 29, 2025

Merged via the queue into PyO3:main with commit 1e00af5 Nov 29, 2025
85 checks passed

Implement #[init] method attribute in #[pymethods] #4951

Implement #[init] method attribute in #[pymethods] #4951

Conversation

ahlinc commented Mar 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ahlinc commented Mar 2, 2025

Uh oh!

mejrs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mbway commented Apr 20, 2025

Uh oh!

davidhewitt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Icxolu commented Nov 10, 2025

Uh oh!

Icxolu commented Nov 14, 2025

Uh oh!

davidhewitt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidhewitt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Icxolu commented Nov 23, 2025

Uh oh!

Icxolu commented Nov 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidhewitt commented Nov 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidhewitt commented Nov 27, 2025

Uh oh!

Implement `#[init]` method attribute in `#[pymethods]` #4951

Implement `#[init]` method attribute in `#[pymethods]` #4951

ahlinc commented Mar 1, 2025 •

edited

Loading

Icxolu commented Nov 23, 2025 •

edited

Loading