Distributing

Recently I tried to learn about distributing Python packages through PyPI. While that seems more or less straightforward for pure Python projects, the story gets more complicated for our case, where extensions in another language need to be compiled in a platform-dependent manner. I ended up with some insights, but also a lot of remaining confusion:

For 1. I found very little information on how to actually implement that. One option seems to build the extensions using distutils in the setup.py script, although I am not sure if that requires the user to install from source, instead of installing using pip. One post mentions that they used build_ext with pip to build a C extension during installation, but I could just not find any information on how exactly that works.

I found much more information on option 2. The preferred way to package binary extensions through are wheels. >A wheel is a built package that can be installed without needing to go through the build process. Installing wheels is substantially faster for the end user than installing from a source distribution.

In our case we would need to create a Platform Wheel, which is a platform dependent wheel that contains compiled extensions. PyPI currently supports wheels for Windows and OS X but only a compatible subset of linux distributions. Wheels appear to be the preferred and more standardized way of packaging versus the older egg format (see this discussion) and are used e.g. to manage Numpy distribution.

Another option is to use conda to package our project. This is where I am still confused. Building a conda package from an existing PyPI project seems straightforward. Those can apparently be build for one platfrom, converted for others and then uploaded to anaconda.org. But I am not sure if that is only true for Python packages, or for binary extensions too, and how it works in the latter case (do the extensions already have to be on PyPI?). There is also a tutorial how to build a conda package from scratch for any language, for which no PyPI project is required. But it is still a bit obscure to me how exactly that works, and what are the advantages over using PyPI with wheels. I found some thoughts on the latter question here under Myth#6

Hopefully I can update with a better understanding soon.