Zk | 2015-04-06-dogfooding-1


type: post date: 2015-04-06 slug: dogfooding-1 title: Dogfooding - Pt. 1


I work for Canonical, Ltd, a company focused primarily on open source software, best known for their operating system, Ubuntu, a distribution of Linux focused on both the ease of desktop use, as well as a seamless experience across desktop, server, mobile, and TV. That's not all that we make, though, and our umbrella of products and services stretches across several sectors of the software market.

I am on the Juju UI team. Juju is a dev-ops solution that allows packaging server software into Juju Charms, which act as biased installations of software that are built by people who know how that software can best be deployed. For instance, the Wordpress charm encapsulates an installation of Wordpress, just as the MySQL charm packages an installation of MySQL, such that, if you have a Juju environment up and running, you can simply run:

{% highlight bash %} juju deploy wordpress juju deploy mysql juju add-relation wordpress mysql juju expose wordpress

and wind up with a proper installation of Wordpress talking to a proper installation of MySQL in the cloud of your choice.

I work on more than just the projects I have at Canonical, however. I've bought wholeheartedly into both the ethos and implementation of open source software, and so I work on releasing as much of my own free-time projects as open source as possible. Lately, the project that has been taking up much of my weekends and evenings has been an exploration of Go, a programming language by Google with a focus on simplicity and concurrency. I chose this language for this task because Juju itself, along with many of its tools, are written in Go.

The project I've been working on, Warren, is a pretty simple bit of web software that allows publishing of entities of various types - from blog posts to pictures and so on - and linking of those entities into a web of content. Or, rather, those are my hopes for it; it's still in its early stages. I've learned a lot about Go, many Go web utilities such as Martini, Gin, and Gorilla, testing in Go using GoConvey, as well as the services that back Warren up such as Mongo and ElasticSearch.

Writing a web app such as this, it was a natural solution to charm Warren so that I could manage it, along with its backing services, in a scalable fashion on any of the clouds that Juju supports.

What I Did

The work that I've done to date consists of two parts beyond Warren itself: the Warren charm and the Warren bundle. Whereas charms describe the way in which a singal service is deployed, bundles are groups of charms that work together as a group of services providing a particular functionality. For instance, the Warren bundle includes Warren, Mongo, ElasticSearch, and haproxy for load balancing.

The Warren charm

I went with the idea of a "thin" charm. This means that the charm itself does not contain any of the source code or binaries required for building Warren itself. This is as opposed to "thick" (sometimes referred to as "fat") charms, which contain a binary, a zip of source code, or some other means of deploying the service without reaching out to some external location. There are a few advantages and disadvantages to each:

Thin charms

Thick charms

For Warren, I went with a thin charm for a few reasons:

In the end, the wins that thick charms get me are outweighed by the wins involved in thin charms.

The Warren bundle

Warren in the Juju GUI

The Juju GUI is the project that I have spent the most time with during my time at Canonical. I started out on the GUI, and continue to contribute to it even as focus has shifted to other aspects of the various user interfaces to Juju. One of the things that I like about it is that it's really easy to see just how your entire environment works and fits together. In the instance above, you can see haproxy exposed, which leads to Warren, which depends on Mongo and ElasticSearch. There's a flow of traffic indicated in the diagram - a request to the exposed bit, which is proxied to Warren, which may make a request of either Mongo, ElasticSearch, or both.

Another benefit of the Juju GUI is that it allows you to export the current environment in the format of a Juju bundle, which is a repeatable representation of an environment. These bundles can then be reimported with the GUI or with another of our tools, Juju Quickstart, which allows building and managing Juju environments from the command line.

In this case, I created and exported my Warren bundle, but am not yet able to utilize it. This is because the Warren charm does not yet live in the Juju charmstore - it's only deployed locally - and bundles do not support local charms yet. For now, though, I'm happy to have the bundle here as a reference to use when deploying Warren to a cloud.

The Good

My list of good things from this post will probably just read about the advantages of using Juju. Once I had the charm up and running, it was astoundingly easy to repeat the process: simply juju deploy local:trusty/warren-charm from my charm repository. I have to say, there's no beating that. In the past, I've deployed and managed just about every one of my projects by hand, and the lack of repeatability led to a ton of external cruft. Deployment checklists, extended QA times, slower iterations, and so on. I think that Juju helps alleviate all of these to some extent.

Additionally, in writing and testing the charm, I found all of the weak spots in my code that weren't evident from a development environment. For instance, I had relied on several hacks and shortcuts supplied by the libraries I was using which, while they looked elegant, were not good solutions for a repeatable testing environment or a production environment. For instance, I relied on defaults for hosting static files and templates. While this made my code look clean and simple, it was not repeatable because there were no guarantees where those assets would be stored. Once I charmed Warren, that became evident, and I found ways to create a robust means of dealing with these problems that was both elegant, and non-magical.

The Bad

Docs.

I found the documentation around the python Charm Helpers, a package used for building and supporting charms, to be quite lacking. It feels as though new features were implemented in code while the documentation had not yet been fully written; what docs are there read as though the user should already understand how to use the package.

As a developer, I'm well aware of this pattern, and definitely guilty of it myself. Even so, I was left spinning my wheels for days as I tried to charm up Warren. I've helped with the charming of several different packages that we use within the company, but none of them use the new, undocumented features of charm helpers.

In the end, I wound up working off the old style of creating a python based charm - that is, rather than utilizing the service framework included with charm helpers, I created methods that were decorated as hooks, which were called simply whenever one of those hooks was fired. This is the style I've most commonly seen in the charms hosted in the charmstore, and while it does work, I get the impression that this newer services framework is much more elegant and easy to use. Easy to use once, that is, you understand how.

It just so happens, though, that the task that I've given myself over the next few months of free-project time at work is to help improve the documentation of the products and services that we have, whether that's code-side in the form of docstrings, wider documentation included with the projects, tutorials, or videos. Indeed, that's much of the purpose behind these dogfooding posts! So I'll be working just has hard as the other teams, I hope, to help document Juju and how it can be used for managing cloud deployments.

What's next

Next, I'd like to talk about more specifics about how the Warren charm works and the services with which it's deployed. I hope to make this at least a short series, if not an account of my ongoing adventures with dogfooding Juju for my own purposes.