Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Explain how to add an extension module #1350

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 46 commits into from
Sep 11, 2024
Merged
Changes from 1 commit
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
086ded3
explain how to add modules
picnixz Jul 13, 2024
96fb998
Update extension-modules.rst
picnixz Jul 13, 2024
5f8797c
Update extension-modules.rst
picnixz Jul 13, 2024
e5d41f8
Update extension-modules.rst
picnixz Jul 13, 2024
b740114
Update extension-modules.rst
picnixz Jul 13, 2024
c74df67
Update extension-modules.rst
picnixz Jul 13, 2024
e645869
Update extension-modules.rst
picnixz Jul 14, 2024
5a9a57c
Update extension-modules.rst
picnixz Jul 14, 2024
26a18eb
Update extension-modules.rst
picnixz Jul 14, 2024
1b4d73f
Update extension-modules.rst
picnixz Jul 14, 2024
94086dd
Address Hugo's feedback
picnixz Jul 14, 2024
9a78a3b
Update extension-modules.rst
picnixz Jul 14, 2024
e22c278
Update extension-modules.rst
picnixz Jul 14, 2024
1f51497
improvements
picnixz Jul 15, 2024
bdf09e5
fixup! sphinx
picnixz Jul 15, 2024
ef7cf3e
fixup! indents
picnixz Jul 15, 2024
1a405ba
fixup! warnings
picnixz Jul 15, 2024
e39cbc2
improve sections
picnixz Jul 15, 2024
35f207f
fix markup
picnixz Jul 15, 2024
316b00d
improve titles
picnixz Jul 15, 2024
523dece
improve presentation
picnixz Jul 15, 2024
d1cdd1d
fixup! markup
picnixz Jul 15, 2024
defb31e
simplify snippets
picnixz Jul 15, 2024
6213438
improvements
picnixz Jul 15, 2024
f6e5d55
improvements
picnixz Jul 15, 2024
d1a1ed5
some rewordings and cleanups
picnixz Jul 15, 2024
86e3e54
simplify wording
picnixz Jul 16, 2024
65f62e7
address Erlend's review
picnixz Jul 17, 2024
4b7c7d8
fix indents?
picnixz Jul 17, 2024
7abb6f1
add ref to clinic everywhere when needed
picnixz Jul 17, 2024
da9b58b
fix typos
picnixz Jul 17, 2024
783e6db
address encukou's review
picnixz Jul 18, 2024
8906ebd
improve the page flow
picnixz Jul 18, 2024
128c81c
use sentence case
picnixz Jul 18, 2024
7d6c8d6
add podman tip
picnixz Jul 18, 2024
01c25bc
address rest of the review
picnixz Jul 18, 2024
56910fb
address Alyssa's review
picnixz Jul 18, 2024
3ee1cea
add details
picnixz Jul 18, 2024
3d235f4
address review
picnixz Aug 4, 2024
7b0b234
Make it easier to update the required ubuntu version
picnixz Aug 4, 2024
7fa94bf
Merge remote-tracking branch 'upstream/main' into add-extensions-tuto…
picnixz Aug 4, 2024
ec48fdb
fixup!
picnixz Aug 4, 2024
1843e3d
fixup!
picnixz Aug 4, 2024
18c4d91
improve comment
picnixz Aug 14, 2024
8224bbd
use double quotes instead of single quotes
picnixz Aug 14, 2024
2f63053
Address Carol's review.
picnixz Sep 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Address Carol's review.
  • Loading branch information
picnixz committed Sep 11, 2024
commit 2f630532c42caeacdfaa43e033851a540ede998e
68 changes: 32 additions & 36 deletions 68 developer-workflow/extension-modules.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,15 @@
Standard library extension modules
==================================
picnixz marked this conversation as resolved.
Show resolved Hide resolved

In this section, we are interested in extending the CPython project with
an :term:`extension module`. We will not explain how to write the module
in C but rather explain how to configure the project and make it compile.
In this section, we explain how to configure and compile the CPython project
with a C :term:`extension module`. We will not explain how to write a C
extension module and prefer to give you some links where you can read good
documentation:

* https://docs.python.org/dev/c-api/
* https://docs.python.org/dev/extending/
* :pep:`399`
* https://pythonextensionpatterns.readthedocs.io/en/latest/

Some modules in the standard library, such as :mod:`datetime` or :mod:`pickle`,
have identical implementations in C and Python; the C implementation, when
Expand All @@ -18,14 +24,6 @@ Other modules mainly implemented in Python may import a C helper extension
providing implementation details (for instance, the :mod:`csv` module uses
the internal :mod:`!_csv` module defined in :cpy-file:`Modules/_csv.c`).

For writing an extension module in C, we prefer to give you some links
where you can read good documentation:

* https://docs.python.org/dev/c-api/
* https://docs.python.org/dev/extending/
* :pep:`399`
* https://pythonextensionpatterns.readthedocs.io/en/latest/

Classifying extension modules
=============================

Expand Down Expand Up @@ -60,15 +58,14 @@ Extension modules can be classified into two categories:
especially if they already have a pure Python implementation.

According to :pep:`399`, *new* extension modules MUST provide a working and
tested pure Python implementation, unless a special dispensation is given.
Please ask the :github:`Steering Council <python/steering-council>` if such
dispensation is needed.
tested pure Python implementation, unless a special dispensation from
the :github:`Steering Council <python/steering-council>` is given.

Adding an extension module to CPython
=====================================

Assume that the standard library contains a pure Python module :mod:`!foo`
together with the following :func:`!foo.greet` function:
with the following :func:`!foo.greet` function:

.. code-block:: python
:caption: Lib/foo.py
Expand All @@ -77,8 +74,8 @@ together with the following :func:`!foo.greet` function:
return "Hello World!"

Instead of using the Python implementation of :func:`!foo.greet`, we want to
use its corresponding C implementation exposed in some :mod:`!_foo` module
written in C. Ideally, we want to modify :cpy-file:`!Lib/foo.py` as follows:
use its corresponding C extension implementation exposed in the :mod:`!_foo`
module. Ideally, we want to modify :cpy-file:`!Lib/foo.py` as follows:

.. code-block:: python
:caption: Lib/foo.py
Expand All @@ -93,14 +90,14 @@ written in C. Ideally, we want to modify :cpy-file:`!Lib/foo.py` as follows:

.. note::

Accelerator modules should *never* be imported directly, whence the
convention is to mark them as private implementation details with the
underscore prefix (namely, :mod:`!_foo` in this example).
Accelerator modules should *never* be imported directly. The convention is
to mark them as private implementation details with the underscore prefix
(namely, :mod:`!_foo` in this example).

In order to incorporate the accelerator module, we need to determine:

- where to place the extension module source code in the CPython project tree,
- which files to modify in order to compile the CPython project, and
- where to update the CPython project tree with the extension module source code,
- which files to modify to configure and compile the CPython project, and
- which ``Makefile`` rules to invoke at the end.

Updating the CPython project tree
Expand All @@ -110,7 +107,10 @@ Usually, accelerator modules are added in the :cpy-file:`Modules` directory of
the CPython project. If more than one file is needed for the extension module,
it is more convenient to create a sub-directory in :cpy-file:`Modules`.

For our extension module :mod:`!_foo`, we consider the following working tree:
In the simplest example where the extension module consists of one file, it may
be placed in :cpy-file:`Modules` as ``Modules/_foomodule.c``. For a non-trivial
example of the extension module :mod:`!_foo`, we consider the following working
tree:

- :ref:`Modules/_foo/_foomodule.c` --- the extension module implementation.
- :ref:`Modules/_foo/helper.h` --- the extension helpers declarations.
Expand All @@ -121,11 +121,6 @@ is called ``<NAME>module.c``, where ``<NAME>`` is the name of the module that
will be later imported (in our case :mod:`!_foo`). In addition, the directory
containing the implementation should also be named similarly.

One could imagine having more files, or no helper files at all. Here,
we wanted to illustrate a simple example without making it too trivial. If
the extension module does not require additional files, it may directly be
placed in :cpy-file:`Modules` as ``Modules/_foomodule.c``.

.. code-block:: c
:caption: Modules/_foo/helper.h
:name: Modules/_foo/helper.h
Expand Down Expand Up @@ -276,9 +271,9 @@ placed in :cpy-file:`Modules` as ``Modules/_foomodule.c``.
Configuring the CPython project
-------------------------------

Now that we have implemented our extension module, we need to update some
configuration files in order to compile the CPython project on different
platforms.
Now that we have added our extension module to the CPython source tree,
we need to update some configuration files in order to compile the CPython
project on different platforms.

Updating :cpy-file:`!Modules/Setup.{bootstrap,stdlib}.in`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand Down Expand Up @@ -402,7 +397,8 @@ Updating :cpy-file:`configure.ac`
[test "$ARCH_RUN_32BIT" = "true"],
[-I\$(srcdir)/Modules/_foo], [])

More generally, the status of the extension is determined as follows:
More generally, the host's configuration status of the extension is
determined as follows:

+-----------+-----------------+----------+
| Enabled | Supported | Status |
Expand Down Expand Up @@ -479,8 +475,8 @@ We describe the minimal steps for compiling on Windows using MSVC.
...

Each item in ``_PyImport_Inittab`` consists of the module name to import,
here :mod:`!_foo`, together with the corresponding ``PyInit_*`` function
correctly suffixed.
here :mod:`!_foo`, with the corresponding ``PyInit_*`` function correctly
suffixed.

* Update :cpy-file:`PCbuild/pythoncore.vcxproj`:

Expand Down Expand Up @@ -539,7 +535,7 @@ We describe the minimal steps for compiling on Windows using MSVC.
Compiling the CPython project
-----------------------------

Now that everything is in place, it remains to compile the project:
Now that the configuration is in place, it remains to compile the project:

.. code-block:: shell

Expand Down Expand Up @@ -571,7 +567,7 @@ Troubleshooting
---------------

This section addresses common issues that you may face when following
this tutorial.
this example of adding an extension module.

No rule to make target ``regen-configure``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand Down
Loading
Morty Proxy This is a proxified and sanitized view of the page, visit original site.