JBjboy·Nov 17, 2015
This post highlights an interesting dichotomy in the Python scientific computing community. Everyone knows that PyPy runs faster than CPython for many common tasks [1].
[1] PyPy is on average 7x faster than CPython: http://speed.pypy.org/
But those in the Python community who are serious about scientific computing (or image processing, like my startup) are already using Numpy & Scipy, which provide hand-coded C implementations of most matrix-related operations. Everyone knows that Python for-loops are "slow" [2], and storing a large 2-D matrix as a list-of-list-of-Python-int-object would require a huge amount of memory & indirection. So, Numpy offers an N-dimensional array type, implemented in C: C arrays of densely-packed C primitive types, with for-loops in C to iterate over the matrix elements. Then Scipy builds a lot of Matlab-lik
Read the rest
e functionality as modules on top of this fundamental Numpy array type.
[2] Python for-loops are slow: https://jakevdp.github.io/blog/2014/05/09/why-python-is-slow...
So the most expensive operations in a Python number-crunching program are likely already implemented using Numpy & Scipy operations, which run in compiled C (and additionally, often make use of Blas/Atlas/LAPACK/etc, for even greater speedups in sustained number-crunching).
But unfortunately, Numpy & PyPy do not naturally work together. Being written in C, Numpy makes substantial use of the CPython C-API -- and in fact, Numpy provides its own C-API [3]! The official Numpy package doesn't work with PyPy; the PyPy project very thoughtfully provides its own PyPy-compatible Numpy package [4].
[3] Numpy C-API: http://docs.scipy.org/doc/numpy-1.10.0/reference/c-api.html
[4] PyPy-compatible Numpy package: http://pypy.org/download.html#installing-numpy
Furthermore, Numpy is fantastic, but it can't offer all possible permutations of matrix operations. In particular, there are certain image-processing operations that are awkward (and thus, computationally-inefficient) to express using Numpy operations. So you might ultimately need to go to the Numpy C-API anyway.
This is why we created (and, just a few days ago, open-sourced) Pymod: https://github.com/jboy/nim-pymod
Pymod is a Nim+Python project that auto-generates all the Python C-API boilerplate & auto-compiles a Python C extension module that wraps the functions in a Nim module. Pymod enables us to write our Numpy array-processing code in Nim, then compile it (for C++-like speeds) as a well-behaved Python module. Nim made this very easy, because it compiles to C.
After considering our Python-integration options (CPython C-API, `ctypes` and `cffi`), we decided to go with the CPython C-API & Numpy C-API. We explained this decision in greater detail in the "Implementation details" section of the Pymod README [5]; the executive summary is that `ctypes` seems better suited to wrapping C types in Python, rather than exposing existing Python types in C, while the CPython C-API code could be generated & compiled with the C code that Nim was going to produce anyway.
[5] Pymod implementation details: https://github.com/jboy/nim-pymod#implementation-details
That said, we would be delighted for Pymod-produced Python modules to be able to run under PyPy. We've been strongly considering implementing a `cffi` back-end for Pymod, but this won't necessarily solve the Numpy issue. It would be even better if PyPy could support all the CPython C-API extension modules in the world in one fell swoop.