Skip to content

Conversation

@Yikai-Liao
Copy link

Summary

  • Cache __annotations__ and __text_signature__ on each nb_func/nb_method so nanobind callables expose the PEP 3107/362 metadata that Python tooling expects.
  • Teach nb_func_getattro to return the cached metadata for those attributes, letting inspect.signature and autodoc treat regular nanobind bindings like builtin functions.
  • Document the autodoc integration details in docs/_ext/nanobind_patch_notes.md for downstream projects.

inspect.signature() now succeeds for standard nanobind methods/functions because CPython consumes their text signatures. Static factories still raise ValueError; CPython does not treat nb_func as a builtin descriptor, so wiring up __signature__ or patching inspect remains future work.

Testing

I test it locally in symusic, here are some examples:

>>> import inspect
>>> import symusic.core as core

>>> fn = core.TrackTick.shift_time
>>> fn.__text_signature__
'(offset, inplace = False)'
>>> fn.__annotations__
{'offset': 'int', 'inplace': 'bool', 'return': 'TrackTick'}
>>> inspect.signature(fn)
<Signature (offset, inplace=False)>

>>> getter = core.TrackTick.__dict__['notes'].fget
>>> getter.__text_signature__
'(/)'
>>> getter.__annotations__
{'return': 'symusic.core.NoteTickList'}

@wjakob
Copy link
Owner

wjakob commented Nov 21, 2025

I am in general open to this change, but this implementation is not in a state that can be accepted.

Instead of the messy/complex postprocessing of strings, let's directly generate information in the right format—basically, can you do the same thing but with much less code?

Also: creating signature objects in Python is a slow operation, and one would in general construct them once and cache them anyways. Additional caching in nanobind is ABI-incompatible to prior versions and seems excessive.

@wjakob
Copy link
Owner

wjakob commented Nov 21, 2025

Finally, thorough testing of all cases will be needed. Take some of the existing bindings to test functions with different types, default arguments, keyword arguments, variable length positional and keyword arguments, keyword-only arguments, etc.

@Yikai-Liao
Copy link
Author

Currently, if we want to make inspect work for static nb func, we could use monkey patch like this:

import inspect
import symusic.core as core

nb_func_type = type(core.ScoreTick.__dict__['from_file'])

def _signature_from_nb_func(nb_func, sigcls):
    text = getattr(nb_func, "__text_signature__", None)
    if not text:
        raise ValueError("no text signature")
    sig = inspect._signature_fromstr(sigcls, nb_func, text, skip_bound_arg=False)
    annotations = getattr(nb_func, "__annotations__", {})
    params = [
        param.replace(annotation=annotations.get(param.name, param.annotation))
        for param in sig.parameters.values()
    ]
    return sig.replace(parameters=params,
                       return_annotation=annotations.get("return", sig.return_annotation))

_orig = inspect._signature_from_callable

def _signature_from_callable(obj, **kwargs):
    if isinstance(obj, nb_func_type):
        sigcls = kwargs.get("sigcls", inspect.Signature)
        try:
            return _signature_from_nb_func(obj, sigcls)
        except Exception:
            pass
    return _orig(obj, **kwargs)

inspect._signature_from_callable = _signature_from_callable

print(inspect.signature(core.ScoreTick.from_file))

And the results would be:

(path: 'str | os.PathLike', format: 'typing.Optional[str]' = None) -> 'symusic.core.ScoreTick'

@Yikai-Liao
Copy link
Author

Additionally, I’d like to ask why you previously chose to use __nb_signature__ instead of the standard __signature__? If __signature__ were available, inspect should be able to correctly handle static functions bound with nanobind. Was this decision driven by some architectural design or compatibility considerations?

@wjakob
Copy link
Owner

wjakob commented Nov 21, 2025

It was meant purely as a communication aid between stubgen and nanobind.

Implement PEP-style introspection metadata (`__text_signature__`/`__signature__`) for nanobind functions and methods.

- Update C++ implementation in `src/nb_introspect.cpp`.
- Update tests in `tests/test_classes.py` and `tests/test_functions.*` to validate PEP-compliant signatures and annotations.

This improves tooling compatibility and enables standard Python introspection for wrapped callables.
Do not treat absent/empty per-overload annotations as explicit 'typing.Any' during merge.\n\nCollect only concrete annotation strings across overloads and emit 'typing.Union[...]' when multiple distinct concrete annotations exist. If no concrete annotations are present for a parameter across overloads, emit 'typing.Any'.\n\nAlso relaxed annotation token collection so type tokens ('%') contribute even if ':' was not present. Updated tests to expect 'typing.Union[int, float]' for overloaded static_test. Ran focused tests locally.
@Yikai-Liao
Copy link
Author

I’ve run into some ambiguously defined behavior: for overloaded functions, how should we handle their annotations, text signature, and signature? I’m currently trying to merge the function signatures of overloads whose parameter names and orders match exactly, and for those that don’t match I just return an empty dict, None, or something similar. I’m not sure if this is a reasonable approach.

@Yikai-Liao
Copy link
Author

And for those functions whose nb_signature is manually bound via nb::sig, should I directly parse the user-provided sig string, or should I still generate the text signature and annotations using the general logic?

- add NB_INTROSPECT_SKIP_NB_SIG toggle and raise AttributeError for skipped/incompatible __signature__/__text_signature__
- keep merged annotation/text rendering consistent and update tests/stubs for the new semantics
@Yikai-Liao
Copy link
Author

Current Status

Summary

  • Added a centralized PEP 3107/362 introspection pipeline for nb_func/nb_bound_method that parses nanobind’s descriptor into a shared metadata struct, then derives __annotations__, __text_signature__, and inspect.Signature from that single source of truth.
  • Introduced NB_INTROSPECT_SKIP_NB_SIG (default on) so bindings with explicit nb::sig opt out of auto-generated PEP metadata; incompatible overloads also opt out. Turn off if you want ot treat functions with nb::sig as normal functions.
  • Overload handling now merges only when parameter order/names/kinds/layout match; otherwise PEP attributes are treated as absent (AttributeError) to avoid misleading metadata.
  • For compatible overloads, per-parameter/return annotations are merged (Union when multiple distinct values) without importing Python types, keeping ABI stable.

Testing

  • Tests updated to cover runtime annotations/text signatures/signature behavior, including Callable rendering differences across Python versions and inspect.signature behavior on C callables without __signature__.
  • Overloaded functions, static functions and functions with nb::sig are all tested.

@Yikai-Liao
Copy link
Author

Pybind11 Implementation Reference

I have checked pybind11 repo, and find that they do not support __text_signature__ __signature__ and __annotations__ for a function.

As for their first line of __doc__, when the funciton is overloaded, it will be set as function_name(*args, **kwargs). I think my strategy is better.

And also, pybind11 does not has things similar to nb::sig, so we need to decide how to deal with it by ourselves.

- delete metadata_builder/param_state and parse descriptors inline
- drop custom C-style string helpers in favor of std::string utilities
- remove the global inspect cache; build handles per call instead
- collapse annotation/signature merging into compact STL helpers
- overall file shrinks by ~50% without changing runtime behavior
@Yikai-Liao Yikai-Liao force-pushed the feat/pep-introspection branch from 6ce494d to 26574b4 Compare November 22, 2025 09:37
Explain why build_meta() replicates nb_func.cpp's state machine and strip the
unused collect_anno flag to make the code less confusing.
Ensure PyObject_Call() failures are handled before inserting into the parameter
list and bail out cleanly if PyList_SetItem itself fails.
- strip "nanobind::" in type_name_local() even when not demangling
- add anonymous_signature_type + type caster and exported test function
- add Python test ensuring annotations/docstrings never expose nanobind::
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants