Monday, March 26, 2012


One of the cool things about working at Hyves is the regular hackathons. The Hyves hackathons are two days where employees can work on projects entirely of their own devising. Typically this results in small teams forming around an idea and trying to produce a working prototype in two days. These events are fun, and a lot can be learned and achieved in the time.

At the last hackathon, our team (Alexander, Chris, Matias and Rodrigo) decided to produce a piece of hardware rather than software. The concept started off simply; we wanted a big, red, fifties-looking button we could use to deploy the entire Hyves system. It grew from there.

The vision

With a little brainstorming prior to the hackathon, we came up with a more complete concept. The vision was now to produce a Deployatron. This would have the following features:
  • Integrate with our existing, Jenkins based, build and deploy process
  • Able to do the two major deploy workflows (deploying the staging and live versions of the site)
  • Look retro-cool
So we got hold of an Arduino, bought the best looking components we could find and set to work.

The design

The first thing we did was get together and do a bit of design. We started off with some system design. We decided that we needed four things: integration with Jenkins, a piece of software running on the Deploymeister's* computer to act as a middle-man, some software on the Arduino and the actual hardware. We decided to keep the Arduino firmware as minimal as possible, and focus the intelligence in the middle layer.

The build

We had several tasks to complete, so we defined some minimal APIs between them and set to work. Matias soon had the Jenkins integration under control using the excellent jenkinsapi library. Alexander set to work on the middleware using Python and Twisted to create an event driven system to talk to the Arduino's serial port. Chris started plugging LEDs into the Arduino and remembering how to solder. By the end of day one we had a working prototype set up, talking via the middleware to the Jenkins server and responding with its flashing LEDs.

The next day we got onto polishing the system. Matias and Rodrigo started work on the case, drilling, cutting and the like, Alexander started putting the finishing touches to the Arduino firmware and Chris started the soldering. With help from various people (William, Sofie) by 4:30pm we had a 90% working system with the following features:
  • Ability to switch between two work-flows
  • Flashing red/green LEDs to indicate build and deploy process
  • Button to build the branch
  • Button that lights up red when it's possible to deploy (when there was a good, recent, build)
  • Two key switches to “arm” the deployatron
All this in a nice looking box, with event-driven middleware communicating between it and Jenkins.

The future

The Deployatron has been used by the Deploymeister to build and deploy the site. There are a few bugs to work out, but its generally functioning fine. In addition, there are a bunch of possible things we might want to do with this in the future:
  • Integrate some other features of the build process – rollback, for instance
  • Make the Deployatron network based, so the software doesn't need to be installed on the Deploymeister's computer
  • Put the middleware directly into a Jenkins plugin, possibly with the ability to just specify which pins which switches and which LEDs are on and what processes they should respond to/start
  • Add indicators for other things – state of the unit tests, for instance


* The Deploymeister is the developer responsible for building and deploying the site in a given week. This role is shared by all developers in a rotating system.

Monday, January 16, 2012

Data Visualization for 'Wie is de Mol?'

For the television show 'Wie is de Mol?' we created a predictions game, in which Hyves members can predict which of the show's candidates is the Mole. For 10 weeks, each week there is an episode of the show, and members participating in the predictions game can change their mind about who is the Mole each week.

With currently more than 125.000 unique members who made a prediction, the project can be called a success, and we have a nice amount of data to create some visualizations.

The chord diagram below shows for each of the show's candidates the number of members predicting he/she is the Mole before the second episode. The chords show how many members transitioned from one Mole to the other between the first and second episode. Note that there are transitions in both directions.

The visualization is created by first dumping the database table containing the predictions to a file. The file is parsed by a Python script, that aggregates the information into a transition matrix. The matrix is visualized with a chord diagram using the D3.js Javascript library.

With the D3.js library you can create interactive diagrams. Hovering the cursor over a part of the diagram belonging to a certain participant, it highlights all the relevant data, as shown below.