Research setup with git and python

Making research is a difficult task.

There are many iterations and how you do it may remind random walk.

How to make it more sane? How to make computations reversible?

Here are some examples of reproducible academic publications with ipython notebook.

I want to share with you my current research setup. It's probably far from optimal, but I'm still researching and experimenting (with it).

Tools

Git and Github

For my thesis I created git repository and connected it to github.

Git is amazing. It makes your work managable. With git I tend to work on smaller more managable chunks, clean up my work properly and end what I just started.

After all it's a version control system, so my code and writings are secured and I can always return to the older/different version.

I have a github account and for every project I set up a new repository. Open source repositories are free.

Github makes collaboration relly easy.

In [13]:
from IPython import display

display.YouTubeVideo("SCZF6I-Rc4I")
Out[13]:

Jenkins and waffle.io

What is jenkins?

It's a continuous integration system and it's cool with lots of users, documentation and plugins.

Jenkins workflow

After every bigger change in code I push my changes to github and jenkins is triggered.

Jenkins does his jobs. Jobs are sets of tasks regarding particular project.

I want to be sure that doing those changes I don't break anything. I also want to have backwards compatibility. What's worse than stale, broken code?

My solution to this is to automatically test my code with jenkins. Jenkins pulls code, builds it from scratch, test it patiently, collect metrics and display all of this to me. It's really great :).

I have my linter connected to jenkins with coverage plugin.

Waffle - dashboard

At work we use atlassian jira which is a great software to help you manage your projects and collaboration. I especially love its board.

Sometimes simple TODO list isn't enough. Waffle is for managing tasks and issues. I like to micromanage myself from time to time.

In [12]:
display.Image('/home/att/waffle_io.png')
Out[12]:

Writing

But how do I actually write my thesis? I introduce you markown. It's a simple markup language which is widely used.

Grab the interactive tutorial.

Comparing to latex an RestructuredText markdown seems really easy to me. It's almost dead simple. After learning some basic tricks I started to write almost any document in markdown. It integrates nicely with github and can be easily converted to html. I write blogposts in markdown and convert it to html with static site generator called nikola.

Why not latex? Latex is very good for scientific notation.

$$ \lambda x = e^{x - x} $$

I can write latex in markown without a problem. It's a common extension.

Powerful notebook

One of my grates tools is IPython notebook. I've written about it in one of my previous posts.

What are the greatest features of notebook:

  • easy collaboration
  • convinience of shell with reusability of the script
  • reevalutaion of cells, possibility of nonlinear work
  • interactive plugins
  • awesome interaction with python - tab completion, argument inspection etc
  • ipython magic
  • support for languages other than python - for example ipython notebook can work with ruby or haskell kernels!

More reasons why giving notebook a try from discussion on reddit

I used notebook to first interactively explore my domain of research and craft some code to later refactor it into appropriate package and to make interactive demos and simulations.

I also used it to write latex and analytical proofs of my models: example notebook

which I embedded on my github repository and share over the internet. Everyone can download this notebook and recreate all results. It's awesome!

Thesis template

I'm pretty excited to see project involving using IPython notebook to write thesis because I'm a notebook enthusiast in process of working on thesis.

Notebook thesis template - looks promising, but probably needs some work to be easily adaptable to various workflows.

I would like to do something as amazing as for example [Python for signal processing] (http://nbviewer.ipython.org/github/unpingco/Python-for-Signal-Processing/blob/master/Buffons_Needle_Sim.ipynb) which is a book written in IPython.

In [15]:
display.Image('/home/att/python_signal.png')
Out[15]:

I think that all those amazing tools make my life as a researcher much easier.

Komentarze

Comments powered by Disqus
Contents © 2014 Justyna Ilczuk - Powered by Nikola - Creative Commons License BY-NC-SA
Share