<![CDATA[Integrating CBS Tools in the Nipy community]]>https://juhuntenburg.github.io/gsoc2017http://github.com/juhuntenburg/gsoc2017/blob/gh-pages/images/blog_cover.pngIntegrating CBS Tools in the Nipy communityhttps://juhuntenburg.github.io/gsoc2017RSS for NodeMon, 28 Aug 2017 17:08:46 GMT60<![CDATA[Final report]]>

As it is the last week of Google Summer of Code, this post is also my final project report, summarizing what I have worked on during the past 3 months, what I have learned, and what is left to do.

The result

I worked on developing Nighres — a Python package for processing of high-resolution neuroimaging data. It provides a Python framework for the Java-based CBS High-Res Brain Processing Tools, aiming to make those tools easier to install, use and extend.

The experience

It is cool to have built a software package. But even more satisfying to know that every step to get there meant learning something I had never done before. Without a course, or a book, simply by doing it. I have written about most of these steps in previous posts, but here is an overview with links:

What I have learned
What I got better at

  • Writing good Python code

  • Writing good docstrings

  • Using Git and Github

  • Using virtualenv for clean Python environments

  • Raising exceptions, errors, warnings in Python

The future

Open issues can be found on the Github issue tracker. In my opinion, these are the most crucial next steps:

  • Writing unittests and testing against different versions of Python, Numpy and Nibabel on Travis

  • Supporting different platforms and architectures

  • Becoming part of the Nipy community of practice

  • Writing instructions for developers to make contributing easy

  • Wrapping more CBS Tools modules and writing the interfaces for them

  • Adding more examples and make them executable on readthedocs (currently too big)

  • Providing a docker image

]]>https://juhuntenburg.github.io/gsoc2017/2017/08/25/final_report.htmlhttps://juhuntenburg.github.io/gsoc2017/2017/08/25/final_report.htmlFri, 25 Aug 2017 00:00:00 GMT<![CDATA[Travis CI]]>

This week I set up continuous integration (CI) with travis for our project. I knew very little about travis — or CI — before, and had thought of it more in terms of running automatic tests. And we don’t even have unit tests yet.

But it turns out travis also very useful for another reason: to automatically build the package — including compiling of Java classes, wrappers and all — and deploy it to PyPI. With the build script from Brainhack Vancouver and, as usual, lots of peeking into other packages, it was easier than I had thought to put together the travis.yml script.

We currently deploy to TestPyPI every time changes are pushed to master. But once Nighres is ready for the official release, it’s easy to switch to the real PyPI server and deploy only tagged releases.

And when we have proper tests for the Python code, we can extend the travis script to run the tests against different versions of Python and the dependencies (Numpy and Nibabel), to ensure that changes to the existing code don’t break current functionality.

With the current script, travis builds on Linux Ubuntu Trusty x86_64. It will be important to set up similar builds for other architectures and platforms, so that we can make the precompiled package available for more users. But one step after the other.

For now, at the end of the second last week of GSoC, what could be more satisfying:

Build Status

]]>https://juhuntenburg.github.io/gsoc2017/2017/08/18/travis.htmlhttps://juhuntenburg.github.io/gsoc2017/2017/08/18/travis.htmlFri, 18 Aug 2017 00:00:00 GMT<![CDATA[Documentation Part II]]>

Through contributing to other projects I have learned how to write docstrings. But I have never set up the actual online documentation myself. For Nighres, I looked into two approaches: Github wiki and Sphinx

Summary

Github wiki

One idea was to document the package with a github wiki. Github wikis are easy to set up and there are some really nice examples.

What I don’t like about the wiki is that all documentation has to be written and edited by hand. That means, if someone changes a function’s docstring or adds a new function, they also have to make the respective changes in the sepearte wiki repo (nighres.wiki).

Sphinx

The other option I looked into is sphinx, which I have seen used for many other Python projects. The big advantage of sphinx is that functions (classes, modules, etc) can be documented automatically using the autodoc extension. That means changes in the main code are automatically integrated when rebuilding the documentation. Or maybe one line needs to be added for a new function.

Now sphinx is much harder to get started on than github wiki. There is good documentation on the basics (e.g. this tutorial), but it took me days to understand how to use the actually interesting features and make it look nice.

What helped me most was to dig into the documentations of similar projects (in my case Nilearn and Nipype), imitate what they do and then start to make changes.

Read The Docs

For hosting the documentation I tried out read the docs. If you write your documentation in sphinx and your code is on github, it is really easy (and free) to set up a searchable documentation website under the .readthedocs.io domain. The best about it: the page automatically rebuilds, whenever you push changes to your repo. Or whenever you make a new stable release, depending on your settings.

Read the docs also has their own custom sphinx theme, which I am using for now. It has many built-in features that make the documentation look nice without much styling. The theme needs to be installed separately:

pip install sphinx-rtd-theme

And then in the sphinx conf.py add:

html_theme = sphinx_rtd_theme

This demo shows how different reStructuredText constructs come out in the read the docs theme. It is also possible to modify the theme, as I found out by peeking into the sphinx gallery documentation (see here and here). I have also played around with a more Nipy-like documentation style, and generally not too many changes would be necessary to switch. But before someone has more time to fiddle with making a pretty custom website, the rtd theme seems like the best choice to me.

Sphinx Gallery

Finally, inspired by the Nilearn documentation, I also learned how to document examples using sphinx gallery. To understand how it is used in Nilearn, and learn from that, I had to build the Nilearn documentation locally to see all the folders and files.

Since it took me a moment to understand the error messages when trying that, here is what you need to install in order to build documentation using sphinx gallery:

pip install sphinx-gallery
pip install matplotlib
pip install pillow

Once correctly set up in the config.py file, sphinx gallery beautifully renders the example code and creates pretty click-able fields (=the gallery), which are easy to include in the documentation. Each function can be set to auto-reference the examples that it is used in.

]]>https://juhuntenburg.github.io/gsoc2017/2017/08/11/Documentation.htmlhttps://juhuntenburg.github.io/gsoc2017/2017/08/11/Documentation.htmlFri, 11 Aug 2017 00:00:00 GMT<![CDATA[Nighres]]>

This week was special, the official birth of Nighres — Processing tools for high-resolution neuroimaging.

Nighres is the name that we decided on for our package. Ni for consistency with other neuroimaging tools in the Nipy (Neuroimaging in Python) community. And nighres in analogy to highres (as in high-resolution data), and because it’s easy to remember and type, and not yet taken on github.

Just born and baptized, nighres soon started making its first step into the software world. It got its own github organization, including a website, and inherited the repo I had been working on so far during GSoC. Conveniently, github makes it super easy to transfer a repo with all history and even issues and project boards.

Excited to help nighres grow into a real software package during the next weeks, so that it soon start to make friends in the Nipy community!

]]>https://juhuntenburg.github.io/gsoc2017/2017/07/28/Nighres.htmlhttps://juhuntenburg.github.io/gsoc2017/2017/07/28/Nighres.htmlFri, 28 Jul 2017 00:00:00 GMT<![CDATA[MANIFEST.in(g)]]>

When setting up the new nighres repo I also worked on the build script from Brainhack Vancouver. It now pulls the raw Java classes from the original cbstools repo, compiles and wraps them using JCC and finally builds the PyPI-ready wheel. The wheel installs nighres, the cbstools wrappers which are called internally (but can also be accessed by advanced users), as well as atlas data that some of the algorithms need.

When dealing with those atlas file I learned that including data files recursively using the setup function is a pain. But thanks to this post I switched to specifying the data directories in a MANIFEST.in file, which is automatically read when setting include_package_data=True in the setup function. Simple and works like a charm.

So far everything is only tested on my linux setup, so I have to look into other systems and continuous integration. But yet another step closer to nice and clean distribution.

]]>https://juhuntenburg.github.io/gsoc2017/2017/07/28/MANIFEST-in.htmlhttps://juhuntenburg.github.io/gsoc2017/2017/07/28/MANIFEST-in.htmlFri, 28 Jul 2017 00:00:00 GMT<![CDATA[Like a real blog]]>

Today I moved my blog from pure github pages to hubpress (which is built on github pages). Takes a few hours of figuring out the setup and switching from Markdown to AsciiDoc.* But from there writing and publishing new posts is simple, and it just looks a bit more like a real blog (maybe I even figure out the cover image eventually). It’s cool how many different things I get to learn through GSoC, for which I probably wouldn’t take time otherwise during my PhD.

* I found atom's AsciiDoc preview and syntax highlighting extensions to help a lot with that

]]>https://juhuntenburg.github.io/gsoc2017/2017/07/27/A-real-blog.htmlhttps://juhuntenburg.github.io/gsoc2017/2017/07/27/A-real-blog.htmlThu, 27 Jul 2017 00:00:00 GMT<![CDATA[Licenses]]>

Trying to decide how to license our package I was reading around and found that https://choosealicense.com/ was helpful for a quick overview on the basics, and https://opensource.org/licenses for detailed info on all licenses.

What eventually helped me most was reaching out to the community via the brainhack slack once again.

The final choice came down to MIT versus Apache 2.0. While both are similarly permissive regarding use and re-use of the code, Apache 2.0 additionally addresses the question of patents. I didn’t really understand what that means in practice, and if it was important for us, but found two posts that helped me get at least a rough idea (here and here).

We eventually decided for Apache 2.0, it seems there is no loss but potential gain in comparison to MIT. Also, while MIT came out of academia and might be more naive regarding corporate issues, Apache 2.0 (or similar) is used by big software projects such as Google’s tensorflow and Mozilla.

]]>https://juhuntenburg.github.io/gsoc2017/2017/07/26/Licenses.htmlhttps://juhuntenburg.github.io/gsoc2017/2017/07/26/Licenses.htmlWed, 26 Jul 2017 00:00:00 GMT<![CDATA[Github project management]]>

Following a workshop at Brainhack Vancouver by the amazing @kirstie_j, I started organizing my GSoC work using github project boards. I probably use it a bit differently than intended — mostly to keep track of my own tasks and asking questions to my mentors — but find it extremely useful. We still have more detailed or organizational discussions on slack, but the project board is great to keep an overview without yet another tool (such as Trello & Co.)

]]>https://juhuntenburg.github.io/gsoc2017/2017/07/21/Github-project-management.htmlhttps://juhuntenburg.github.io/gsoc2017/2017/07/21/Github-project-management.htmlFri, 21 Jul 2017 00:00:00 GMT<![CDATA[A documentation for documentation]]>

These days I am writing the Python interfaces to the JCC-wrapped Java code, that will eventually be exposed to the user. This also means writing comprehensive docstrings. Up until now I mostly freestyled docstrings, roughly trying to imitate what I had seen (and liked) in other packages, or other functions of the package I worked on. But searching for something docstring-related I stumbled upon the PEP257 docstring conventions. Maybe not so surprising that they exist. More surprising, there are specific NumPy/SciPy docstring conventions (and examples). It makes sense for us to follow the Scientific Python community convenstions, and so far I find the style clean, easy to follow and flexible enough for package specifics. Plus the warm and fuzzy feeling of following standards (#GermanAfterAll). Hooray!

]]>https://juhuntenburg.github.io/gsoc2017/2017/07/18/A-documentation-for-documentation.htmlhttps://juhuntenburg.github.io/gsoc2017/2017/07/18/A-documentation-for-documentation.htmlTue, 18 Jul 2017 00:00:00 GMT<![CDATA[Brainhack Vancouver]]>

From June 22-24 I was at the OHBM Hackathon 2017 in Vancouver and pitched a project to implement the distribution for cbstools-python. I mostly hoped that some of the folks there had some advice, but did not really expect anyone to get overly excited about joining this pretty specific project. Turns out I still underestimate the power of brainhack — right after the pitch kofalt came up to me and said he would love to join.

Within two & 1/2 intense and fun days we managed to pool his expertise on software packaging and distribution, my recently acquired knowledge about PyPI and what we want for cbstools-python, and some helpful insights from other brainhackers into a single, but powerful script. It performs the following essential steps:

  1. Download the MIPAV and JIST dependencies (should become redundant soon)

  2. Compile the Java code

  3. Wrap the Java classes using JCC

  4. Build the right folder structure and finally a wheel

This automates all steps necessary for continuous integration and distribution via PyPI. It will need to be adapted once the high-level Python interfaces have been added and we have decided on the final code organization. Also a few small issues regarding continuous integration are still to be solved. But it is still a massive step forward and much more than I ever expected to get done in that short time.

As usual, I left brainhack tired but thrilled: about this awesome, inspiring community, and about having learned more in a few days than in a regular month or more back at university. If you are into neuroscience and coding, you should definitely check out the next brainhack in your region, or just organize one yourself.

]]>https://juhuntenburg.github.io/gsoc2017/2017/07/16/Brainhack-Vancouver.htmlhttps://juhuntenburg.github.io/gsoc2017/2017/07/16/Brainhack-Vancouver.htmlSun, 16 Jul 2017 00:00:00 GMT<![CDATA[Distributing]]>

Recently I tried to learn about distributing Python packages through PyPI. While that seems more or less straightforward for pure Python projects, the story gets more complicated for our case, where extensions in another language need to be compiled in a platform-dependent manner. I ended up with some insights, but also a lot of remaining confusion:

Generally there are two options:

  1. compiling the extensions during installation

  2. distributing pre-compiled binaries

For 1. I found very little information on how to actually implement that. One option seems to build the extensions using distutils in the setup.py script, although I am not sure if that requires the user to install from source, instead of installing using pip. One post mentions that they used build_ext with pip to build a C extension during installation, but I could just not find any information on how exactly that works.

I found much more information on option 2. The preferred way to package binary extensions through are wheels. >A wheel is a built package that can be installed without needing to go through the build process. Installing wheels is substantially faster for the end user than installing from a source distribution.

In our case we would need to create a Platform Wheel, which is a platform dependent wheel that contains compiled extensions. PyPI currently supports wheels for Windows and OS X but only a compatible subset of linux distributions. Wheels appear to be the preferred and more standardized way of packaging versus the older egg format (see this discussion) and are used e.g. to manage Numpy distribution.

Conda

Another option is to use conda to package our project. This is where I am still confused. Building a conda package from an existing PyPI project seems straightforward. Those can apparently be build for one platfrom, converted for others and then uploaded to anaconda.org. But I am not sure if that is only true for Python packages, or for binary extensions too, and how it works in the latter case (do the extensions already have to be on PyPI?). There is also a tutorial how to build a conda package from scratch for any language, for which no PyPI project is required. But it is still a bit obscure to me how exactly that works, and what are the advantages over using PyPI with wheels. I found some thoughts on the latter question here under Myth#6

Hopefully I can update with a better understanding soon.

]]>https://juhuntenburg.github.io/gsoc2017/2017/06/16/Distributing.htmlhttps://juhuntenburg.github.io/gsoc2017/2017/06/16/Distributing.htmlFri, 16 Jun 2017 00:00:00 GMT<![CDATA[To Docker or not to Docker]]>

Reaching out to the neuroimaging community regarding means of software distribution got me many helpful replies and I learned quite a bit about Docker and related tools along the way. My main take-aways:

  • Docker as a scientific tool in the neuroimaging community is of growing importance, as exemplified by BIDS Apps or CBRAIN. However, it still in an early stage so that widespread adoption will take time and installation procedures might need to be tweaked due to changes to the Docker project itself.

  • Default Docker images such as AlpineLinux do not add a lot of overhead (they are much smaller than virtual machines) and should not result in a significant performance penalties unless one is dealing with super efficient numerical code.

  • Using Docker in combination with Singularity helps to run containers without root access and without Docker being actually installed, e.g. on HPCs

  • When distributing through Docker it is even more important to have good regression tests across platforms and software versions. There are two projects trying to make this easier: neurodocker and niceman

  • One interesting approach is, in addition to the Docker image, to publish a standalone script that generates Docker commands for users through PyPI, so it can be install with pip.

One of the most helpful comments regarding our project I found to be this (on neurostars.org):

"If you are looking to distribute a library that is intended to be integrated with other software wrapping it in Docker will make it hard if not impossible. If you are looking to distribute a command line tool with complex set of dependencies Docker is a good fit."

From the replies and talking to my mentors I got the sense that it will be the best choice for us to focus on more traditional ways of download and installation for now. Especially, because integrating our tools with other software is a main objective of this project. But we will also explore if it makes sense to additionally provide a Docker Image later on.

Thanks for all the helpful comments and suggestions!

]]>https://juhuntenburg.github.io/gsoc2017/2017/06/03/To-Docker-or-not-to-Docker.htmlhttps://juhuntenburg.github.io/gsoc2017/2017/06/03/To-Docker-or-not-to-Docker.htmlSat, 03 Jun 2017 00:00:00 GMT<![CDATA[Wisdom of the crowd]]>

This week, GSoC started with the "community bonding" phase. I already feel pretty bonded with my software community (in fact I constitute 1/3 of it), so I use the time to reach out to the larger neuroimaging community regarding one of the first issues I want to address:

What is the best way to distribute the Python version of CBS Tools?

I have previously discussed this question with my mentors and other colleagues, and one suggestion that came up is to deploy the tools via Docker. In order to find out what neuroimaging folks think about docker I have started discussion threads on neurostars and in the brainhack slack team.

I know very little about docker myself, so I also started watching some tutorials. Whether we end up using it or not, my feeling is it won’t hurt in the future to know a thing or two about docker.

]]>https://juhuntenburg.github.io/gsoc2017/2017/05/11/Wisdom-of-the-crowd.htmlhttps://juhuntenburg.github.io/gsoc2017/2017/05/11/Wisdom-of-the-crowd.htmlThu, 11 May 2017 00:00:00 GMT<![CDATA[Google Summer of Code (GSoC)]]>

On this site I will document my work during the Google Summer of Code 2017 with INCF. Follow the links for a project description in short or long.

]]>https://juhuntenburg.github.io/gsoc2017/2017/05/08/Google-Summer-of-Code-G-SC.htmlhttps://juhuntenburg.github.io/gsoc2017/2017/05/08/Google-Summer-of-Code-G-SC.htmlMon, 08 May 2017 00:00:00 GMT