Metadata-Version: 2.4
Name: cffi-mkstub
Version: 0.0.2
Summary: Type stub generator for cffi modules
Author-email: Alba Mendez <me@alba.sh>
License-Expression: GPL-3.0-or-later
Project-URL: Homepage, https://github.com/mildsunrise/cffi-mkstub
Project-URL: Issues, https://github.com/mildsunrise/cffi-mkstub/issues
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Typing :: Typed
Classifier: Topic :: Software Development :: Build Tools
Classifier: Topic :: Software Development :: Code Generators
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# cffi-mkstub

[Type stub][stub-files] generator for Python modules generated by [cffi][]. This allows statically checking that the C APIs are being used correctly and provides autocompletion of:

 - function names
 - global variable names
 - struct or union member names
 - enum members
 - C type expressions passed to `ffi.new`, `ffi.cast` and friends

(Screenshot) Autocompletion of lib globals
(Screenshot) Feedback when calling a function

To get an idea of what generated stubs look like, see [example][] ([source C code][example-source]). They contain:

 - a few generic helper definitions that should ideally be incorporated to typeshed
 - a `types` namespace with definitions for all discovered ctypes
 - a `Lib` class specialization with definitions for all globals (functions, variables, constants)
 - an `FFI` class specialization where methods like `new`, `cast` and such are given precise overloads for every C type expression, member name, etc.

The type stubs would ideally be bundled into the resulting cffi package, but can also be used directly by end users. Can also be used with in-line mode, see [programmatic generation](#programmatic-generation) below.

## Philosophy

It's best to start using the generated stubs from the start. In particular, you should **not** expect existing code to typecheck out-of-the-box when applying them. The intent of this project isn't to rule out wrong usages of the API and 'stand down' on usages that *could* be correct, because due to cffi's design, most of your code would fall in the latter category. Instead we aim to expose a reasonable subset of cffi's API that can be reliably checked.

A major consequence of this can be seen in `ffi` methods that accept C type expressions, like `ffi.new('MyStruct_t *')`. cffi will happily parse and normalize your C type expression, so the string `MyStruct_t*` would also work, but this can't possibly be done from a type stub. Therefore, rather than falling back to returning a generic `CData` object if an unrecognized string is passed, the stubs require you to enter the exact (normalized) C type expression. Any difference, like a missing space, triggers a typechecker error. This was deemed a reasonable tradeoff because when using an appropriate language server, you will usually get autocompletion of the accepted strings when writing your code:

(Screenshot) Autocompletion of C type passed to ffi.new()

Another consequence is type conversions. cffi is very lenient with what it accepts, e.g. on a parameter of C type `int` you can pass a Python `int` or an integer CData or an enum CData. The stubs currently only allow `int`, because this is what cffi returns when converting in the other direction.

## Status

Despite the above, the aim is still to eventually expose all *functionality* of cffi (even if through a restricted set of API usages). Even though this project is in the proof of concept stage (and I'm depressed so manage your expectations about my ability to improve and maintain it) it's already functional enough to cover most of the functionality, and I'm using it on my projects. Some notable limitations:

 - Pylance (or maybe Pyright itself, or the VSCode extension) seems to be very bad at handling methods with lots of overloads. In particular it freezes when the cursor is inside of a call like `ffi.new()` and it attempts to display the parameters, to the point where I have to restart the language server. Short of cffi adding new per-ctype APIs (which would greatly reduce the number of overloads in a single function) there isn't much we can do to work around it.

 - When calling `ffi.addressof()` with a field name, Pylance will fail to filter valid names according to the type being passed (it will show all member names of all types). Typechecking still works correctly, i.e. an error will result if the member name of another struct is passed.

 - When using `ffi.def_extern`, the name needs to be passed as an argument:
   ~~~python
   @ffi.def_extern('my_callback')
   def my_callback():
       ...
   ~~~

 - Included FFI objects (`ffi/ffibuilder.include(other_ffi)`) are currently ignored.

One of the priorities right now is to make it (both the generated stubs, and with less priority, the generator itself) more compatible with older Python versions. It has only been tested with Python 3.14 so far.

Works in [both ABI and API modes][cffi-modes], but most of the testing has been done on the former, so some wrinkles may still be present when targetting API mode. It has only been tested with Pylance/PyRight so far.

## Getting started

### Generating the stubs

cffi-mkstub works by interrogating your specific `ffi` object. Because of this, cffi-mkstub doesn't actually pull cffi as a dependency -- but because `ffi` objects need to import `_cffi_backend` to function, it presumes that module to be available for import.

ctypes backend support
The above is not entirely true; although this is currently not very well supported by cffi, when in in-line ABI mode, cffi can be told to use its ctypes backend (removing any dependency on custom native extensions at all). To support this case, cffi-mkstub will tolerate `_cffi_backend` failing to import when `cffi` could be imported, and if an in-line ABI FFI object is passed, it will use its undocumented `_backend` property to know which backend was requested.

> [!IMPORTANT]
> Although we try to use documented cffi APIs whenever possible, their current introspection APIs are simply not enough. We have an [open PR][cffi-introspection-apis] that implements more APIs. **This project needs cffi to have that patch applied to run.** Note that the patch is only needed to run the type stub generator; your project can then use the generated type stubs with a vanilla version of cffi, as the (non-introspection) APIs remain unchanged.

A simple CLI interface is provided which should be enough for most users; run it passing `-m` and the name of the module to import:

~~~bash
cffi-mkstub -m my_lib._foo_cffi
~~~

This should be the module that `ffibuilder.compile()` produced, e.g. what you passed in the first argument to `ffibuilder.set_source()`. cffi-mkstub will import it and fetch its `ffi` attribute. Alternatively you can pass a filesystem path with `-p`:

~~~bash
cffi-mkstub -p my_lib/_foo_cffi.py
~~~

By default, the stubs will be written to a `.pyi` file next to the imported module file. This should cause tools to find them automatically. If this is not desired (for example, when the original module is in an unwritable system directory) pass `-o <output path>`.

### Using the stubs

Note that the stub declares type aliases, classes and namespaces that do not really exist at runtime in the real module. Never attempt to use those at runtime (with e.g. `instanceof`). If you need to reference types in an annotation...

~~~python
import _foo_cffi
lib = _foo_cffi.ffi.dlopen('my_foo.so')

def pat(animal: _foo_cffi.types.Animal_t) -> _foo_cffi.Pointer[int]:
    return lib.foo_pat(animal)
~~~

...the code will only run on Python 3.14+, where [annotations are evaluated lazily][PEP-649]. To support older versions, put the expression inside a string literal:

~~~python
def pat(animal: '_foo_cffi.types.Animal_t') -> '_foo_cffi.Pointer[int]':
    return lib.do_pat(animal)
~~~

You can also do `from __future__ import __annotations__` ([supported since 3.7+][PEP-563]) which will cause the interpreter to do that for you, but in the future it [will get deprecated and eventually removed][PEP-749].

If you want to import the `types` namespace to reduce verbosity, or even some of the types themselves, you can use `typing.TYPE_CHECKING` to skip doing so at runtime (where it would fail). You can even bring specific types into scope (but you can't use `import` for that, since `types` isn't a submodule):

~~~python
from typing import TYPE_CHECKING
from _foo_cffi import ffi

if TYPE_CHECKING:
    from _foo_cffi import types, Pointer
    Animal_t = types.Animal_t

lib = ffi.dlopen('my_foo.so')

animals: list['Pointer[Animal_t]'] = []
...
~~~

### Programmatic generation

cffi-mkstub can also be invoked programmatically through the Python API, which is ideal in e.g. build scripts after compiling with ffibuilder. The simple API (used by the CLI) is `write_type_stub` which takes either a module object or a name to import, and allows overriding the output path through `output_path`:

~~~python
from cffi_mkstub import write_type_stub
write_type_stub('my_lib._foo_cffi')

import external_lib._cffi
write_type_stub(external_lib._cffi, output_path='external_cffi.pyi')
~~~

If more control is desired, the `format_type_hints` API can be used directly. This takes the `ffi` object (not the module) and returns Python code as a string, and accepts a number of arguments to customize the output (but not a lot yet). Keep in mind the returned Python code defines the `FFI` and `Lib` classes, but does not assume instances of those to be present in the `ffi` and/or `lib` attributes of your module -- when using this API, you need to declare these yourself:

~~~python
from cffi_mkstub import format_type_hints
from my_lib._foo_cffi import ffi

hints = format_type_hints(ffi, ffi_cls_name='FooFFI', lib_cls_name='FooLib')
hints += '\n\n' 'ffi: FooFFI' '\n' 'lib: FooLib'
with open('src/my_lib/_foo_cffi.pyi', 'w') as f:
    f.write(hints)
~~~

This API can be used to support [in-line usage][cffi-inline-abi], e.g. with ABI mode:

~~~python
from typing import TYPE_CHECKING, cast
from cffi import FFI
from cffi_mkstub import format_type_hints

ffi = FFI()
ffi.cdef("""
    int printf(const char *format, ...);   // copy-pasted from the man page
""")

# on the first run (see below) the type stub file will be populated and the typechecker will be able to load it
with open('build/_gen_ffi_stubs.pyi', 'w') as f:
    f.write(format_type_hints(ffi))
if TYPE_CHECKING:
    from build._gen_ffi_stubs import FFI
    ffi = cast(FFI, ffi)

lib = ffi.dlopen(None)
arg = ffi.new("char[]", b"world")
lib.printf(b"hi there, %s.\n", arg)
~~~

This could ideally be used as a compact way to generate type stubs (in-line ABI mode doesn't need a working toolchain, a library to link against nor a temporary output directory) to distribute in a stub package and use them later with the compiled module. However keep in mind there are some subtle differences between the four mode combinations in cffi, most notably that in ABI mode no overloads for `def_extern` are generated. In the future we could provide flags to 'simulate' other modes to enable this use case.

## Wishlist for cffi

[PEP-484][] did not exist when cffi was started, so it's understandable that its design doesn't lend very well to static typechecking. While I'd say we can get a very decent result already, some changes on their side would allow us to improve the developer experience (sometimes even outside cffi-mkstub as well) even more:

- An option to retain documentation comments in the produced module. It would then be exposed via `__doc__` or a dedicated ctype API. This would also include function argument names.

- Expose the base type of an enum, i.e. `_CTypeEnum.base_type`.

- Per-type `addressof`, `offsetof`, `new`, `cast` APIs (see above for motivation).

- Retain `typedef`s as their own `CType` kind. Being able to tell which particular typedef, if any, was used at a specific spot (say, a struct member, or a function argument) to reference a given type.

These are changes that could ideally be implemented in a backwards compatible way.

## License

cffi-mkstub itself is GPL licensed for now, but you are of course free to use the stubs it generates for whatever purpose.


[cffi]: https://cffi.readthedocs.io/
[cffi-modes]: https://cffi.readthedocs.io/en/stable/overview.html#abi-versus-api
[cffi-inline-abi]: https://cffi.readthedocs.io/en/stable/overview.html#simple-example-abi-level-in-line
[stub-files]: https://peps.python.org/pep-0484/#stub-files
[PEP-484]: https://peps.python.org/pep-0484
[PEP-563]: https://peps.python.org/pep-0563/
[PEP-649]: https://peps.python.org/pep-0649/
[PEP-749]: https://peps.python.org/pep-0749/
[cffi-introspection-apis]: https://github.com/python-cffi/cffi/pull/231
[example]: https://github.com/mildsunrise/eci-analysis/blob/aeec1e36b231dea0c976abc70781b500e6ead3f6/bindings/_eci_cffi.pyi
[example-source]: https://github.com/mildsunrise/eci-analysis/blob/aeec1e36b231dea0c976abc70781b500e6ead3f6/synthDrivers/eloquence/eci.h
