CPython and the Python/C API
The CPython interpreter (aka, "python") works by compiling Python source code to intermediate bytecodes, and then operating on those. CPython, which is written in C, is also accompanied by an Application Programming Interface (API) that enables communication between Python and C (and thus basically to any other language that C can interoperate with). The Python/C API allows for compiled pieces of code to be called from Python programs or executed within the CPython interpreter. This process of producing compiled code for use by CPython is generally known as "extending" Python and the compiled pieces of code to be used are known as "extension modules".
Extension modules are in the form of shared libraries (e.g., .so shared object files on Linux/Mac, or .pyd dynamically linked Python libraries on Windows) which contain both underlying code executing the operations of interest and calls to functions in the Python/C API that interface that code to the interpreter. Much of the core functionality of the Python language and standard library are written in C (stored typically in the directory $PYTHON_HOME/lib/pythonX.Y/lib-dynload, where $PYTHON_HOME refers to the root of the python installation and X.Y refers to a specific python version). Extension modules can be imported into the Python interpreter in the same way that pure Python code can, exposing functions and other objects for use within the interpreter.
Extension modules can be written by hand using the Python/C API, but typically, they are generated by third-party tools that automate the generation of interface code based upon specifications, such as function declarations in a library that someone wants to wrap up and call from Python. With some third-party tools, compiled code is auto-generated and imported back into running Python code automatically. We will see examples of these sorts of operations in the tutorial material that follows.
If you're curious and are on a Linux or Mac system, you can examine in a bit more detail what the insides of an extension module look like. The math
module is part of the Python Standard Library, providing a Python interface to the C Math Library, available within Python via the statement import math
. First execute this code block to see what file contains the contents of the math module:
On a Linux or Mac system, you can use the nm
utility on the command line to examine the symbol table of the shared object file that contains the math module. (There might be a similar function available under Windows, but it appears not to be part of a default installation, and is instead provided by additional tools such as Visual Studio.) You could run nm
separately in your shell, or you could run it within python via a system call using the os
module from the Python Standard Library:
Alternatively, within ipython or jupyter, you could do a shell escape to run that command (note the extra dollar sign $ to extract the name of the file for the shell command):
The output of the nm command is long, but within it you should see both the names of some functions that are implemented in the C Math library, such as:
as well as some functions from the Python/C API that are called in the interface to the library, such as: