323 lines
16 KiB
HTML
323 lines
16 KiB
HTML
<!doctype html>
|
|
<html>
|
|
<head>
|
|
<title>Zk | 2015-04-09-dogfooding-2</title>
|
|
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Gentium+Plus&family=Lato&family=Ubuntu+Monodisplay=swap" />
|
|
<link rel="stylesheet" type="text/css" href="/style.css?2024-05-04" />
|
|
<meta name="viewport" content="width=device-width" />
|
|
<meta charset="utf-8" />
|
|
</head>
|
|
<body>
|
|
<main>
|
|
<header>
|
|
<h1>Zk | 2015-04-09-dogfooding-2</h1>
|
|
</header>
|
|
<article class="content">
|
|
<hr />
|
|
<p>type: post
|
|
date: 2015-04-09
|
|
slug: dogfooding-2
|
|
title: Dogfooding - Pt. 2</p>
|
|
<hr />
|
|
<p><em>This is a continuation of the <a href="/posts/tech/2015/04/06/dogfooding-1/">first
|
|
post</a>; you should read that first!</em></p>
|
|
<p>This is part 2 of the “Dogfooding Juju” series that I’m doing. This time, I want
|
|
to go into a little bit of detail about the Warren charm and how I wound up
|
|
structuring it. As I mentioned in the previous post, there are perhaps more
|
|
elegant ways to do this, but I found the documentation to be lacking in ways
|
|
that prevented me from dedicating relatively scant free time to the task.
|
|
Instead, I wound up following the path that I’ve followed before on the job with
|
|
several of the charms we use for our own projects.</p>
|
|
<h2 id="how-a-charm-works">How a Charm Works</h2>
|
|
<p>One can think of a charm as a deployment solution. In a lot of ways, it follows
|
|
the same path as many other dev-ops solutions out there, such as Ansible, Chef,
|
|
or Puppet. In fact, one can use any one of those solutions (or more than one,
|
|
if one is so inclined) in managing what happens during the deployment of a charm
|
|
into a service, as many charms do. Charms are meant to be biased, best-practice
|
|
solutions that install a service and describe the way that service relates to
|
|
other services in the Juju environment.</p>
|
|
<p>A charm is, at its core, primarily built around two concepts: configuration and
|
|
hooks. Configuration describes the way the charm is built and how it can
|
|
interact with other services, while hooks describe how the service responds to
|
|
state changes, both internally and externally. There is a third bit, which we
|
|
won’t get into here, as it’s not relevant, but is worth mentioning as part of
|
|
the charm, which is “actions”. Actions are code within the service that
|
|
responds to requests, either from the user or from the environment, through Juju
|
|
itself.</p>
|
|
<p>A lot of (digital) ink has been spilled on charm building, so I’m not going to
|
|
go too far in depth beyond explaining how this works with the Warren charm.
|
|
Please feel free to check out the <a href="https://jujucharms.com/docs/stable/authors-intro">extensive
|
|
docs</a> if you’re interested in
|
|
diving deeper.</p>
|
|
<h3 id="configuration">Configuration</h3>
|
|
<p>Configuration primarily happens in two files used by the charm: <code>metadata.yaml</code>
|
|
and <code>config.yaml</code>. <code>metadata.yaml</code> contains information about the charm, who
|
|
wrote it, and how it connects to various other charms within the environment,
|
|
while <code>config.yaml</code> contains all of the configuration options that a charm may
|
|
use during execution of any of its numerous hooks.</p>
|
|
<p>Anyone who is familiar with packaging software in any way will be familiar with
|
|
the way these two files work. You can specify the name of the charm, the
|
|
authors, a short description, and some tags in the <code>metadata.yaml</code> file.
|
|
Additionally, if this charm relies on other services, they will be defined in
|
|
the interfaces section. <code>config.yaml</code> is basically a schema describing the
|
|
configuration options that the charm uses. For each option, a type, a
|
|
description, and a default may be provided.</p>
|
|
<h3 id="hooks">Hooks</h3>
|
|
<p>Hooks are where all of the action takes place in a charm. There are a few main
|
|
hooks, and then several which depend on the state of the environment. The main
|
|
hooks are:</p>
|
|
<ul>
|
|
<li><code>install</code> - work that needs to take place as the charm is first being
|
|
installed.</li>
|
|
<li><code>start</code> - actions that take place as the charm is moving from <code>installed</code> to
|
|
<code>started</code> states.</li>
|
|
<li><code>stop</code> - actions that take place as the charm is moving from <code>started</code> to
|
|
<code>stop</code> or <code>dying</code> states.</li>
|
|
<li><code>config-changed</code> - actions that take place when any configuration value has
|
|
changed.</li>
|
|
</ul>
|
|
<p>The other hooks you might encounter are relation hooks. These are fired as the
|
|
state of relations to the charm change. They come in four types, each of which
|
|
includes the name of the relation interface as part of it:</p>
|
|
<ul>
|
|
<li><code>*-relation-joined</code> - fired when the two services first start talking to each
|
|
other.</li>
|
|
<li><code>*-relation-changed</code> - fired when some aspect of the relation is changed, such
|
|
as data about that relation is changed.</li>
|
|
<li><code>*-relation-departed</code> - fired when a relation is removed by the user.</li>
|
|
<li><code>*-relation-broken</code> - fired when the relation is broken between the two
|
|
services for some reason, such as one service being removed.</li>
|
|
</ul>
|
|
<h2 id="the-structure-of-the-warren-charm">The Structure of the Warren Charm</h2>
|
|
<p>The Warren charm is basically typical, as far as charms go. It has its own
|
|
metadata and config files, as well as a full collection of hooks.</p>
|
|
<h3 id="configuration_1">Configuration</h3>
|
|
<h4 id="metadatayaml">metadata.yaml</h4>
|
|
<p>The <code>metadata.yaml</code> file contains a lot of basics that will be familiar at a
|
|
glance. Name, summary, maintainer, description, tags, these are all pretty
|
|
straight forward. Of note, however, are the subordinate element, which declares
|
|
whether or not this service will be subordinate to another (a topic for a later
|
|
date), and the provides/requires elements, which describe how this service can
|
|
relate to others.</p>
|
|
<p>Provides describes the interface that this service will expose to others within
|
|
the Juju environment. Of particular note (mostly because the others haven’t
|
|
been implemented yet) is the website interface, which provides a means of
|
|
hosting content over HTTP/S. This will be used by the haproxy charm, which will
|
|
provide load balancing over this interface.</p>
|
|
<p>Requires describes the interfaces that this service needs other charms to
|
|
provide within the environment in order to run fully. In this case, this means
|
|
Mongo via the mongodb charm, and ElasticSearch via the eponymous charm.</p>
|
|
<pre><code class="language-yaml">
|
|
name: warren-charm
|
|
summary: Warren is a networked content-sharing site.
|
|
maintainer: Madison Scott-Clary <makyo@drab-makyo.com>
|
|
description: |
|
|
Warren is a networked content-sharing site, allowing users to not only post
|
|
their creations, but link them together into a web of their works, and the
|
|
works of others. It manages each post as an abstract entity and uses content
|
|
types to render those abstract types into something viewable within a
|
|
browser.
|
|
tags:
|
|
- social
|
|
- cms
|
|
- applications
|
|
subordinate: false
|
|
provides:
|
|
website:
|
|
interface: http
|
|
nrpe-external-master:
|
|
interface: nrpe-external-master
|
|
scope: container
|
|
local-monitors:
|
|
interface: local-monitors
|
|
scope: container
|
|
requires:
|
|
mongodb:
|
|
interface: mongodb
|
|
elasticsearch:
|
|
interface: elasticsearch
|
|
</code></pre>
|
|
|
|
<h4 id="configyaml">config.yaml</h4>
|
|
<p>Our configuration values for this charm are also pretty straight-forward. You
|
|
can see that we have options for an SMTP server, which will be used for sending
|
|
notification emails, two keys which are used for encrypting session data, the
|
|
database name, the port to listen on, and the source. Source is interesting
|
|
because it’s structured to allow various different ways to fetch the source for
|
|
building Warren. Since this is a thin charm (that is, it does not include any
|
|
of the source for Warren itself), the charm will have to figure out how to fetch
|
|
the source as required. We’ve provided a few ways of specifying that, all of
|
|
which interface with Git: one can specify a branch name, a tag name, or a
|
|
commit SHA.</p>
|
|
<pre><code class="language-yaml">
|
|
options:
|
|
smtp-server:
|
|
default: smtp.example.com
|
|
description: Address for the SMTP server for sending emails from Warren
|
|
type: string
|
|
session-auth-key:
|
|
default: CHANGEME--------
|
|
description: Session authentication key (16 or 32 bytes)
|
|
type: string
|
|
session-encryption-key:
|
|
default: CHANGEME--------
|
|
description: Session encryption key (16, 32, or 64 bytes)
|
|
type: string
|
|
mongo-db:
|
|
default: warren
|
|
description: The mongo database name
|
|
type: string
|
|
listen_port:
|
|
default: 3000
|
|
description: The port to listen on
|
|
type: int
|
|
source:
|
|
default: "branch:master"
|
|
description: A string containing a "branch:", "tag:", or "commit:" followed
|
|
by a branch name, a tag name, or a commit, respectively
|
|
type: string
|
|
</code></pre>
|
|
|
|
<h3 id="hooks_1">Hooks</h3>
|
|
<p>This is where the meat of the charm lives. Hooks are executable bits of code
|
|
within the <code>/hooks</code> directory of the charm, each named appropriately. That is,
|
|
there is an executable file in <code>/hooks</code> named <code>install</code>, one named <code>start</code>, and
|
|
so on for all of the hooks that will be fired for our service. As is standard
|
|
practice for this type of charm, I actually have all of the code in one file,
|
|
<code>hooks.py</code>, and all of the hooks files are simply symlinked to point to that
|
|
file.</p>
|
|
<p>I’m not going to go too in depth here, nor post the <a href="ttps://github.com/warren-community/warren-charm/blob/master/hooks/hooks.py.html">entire
|
|
file</a>,
|
|
which you can look at yourself, but simply outline the way the hooks are called.
|
|
Future posts may go more in depth as to how things work on a more atomic level.</p>
|
|
<p>First is our install hook, as shown by the decorator. This one takes care of
|
|
some initial work that needs to be done to get the service up and running. It
|
|
updates all packages, ensures dependencies (such as golang, git, and bzr), adds
|
|
a user which will be used to run the service, makes source and build
|
|
directories, and installs the source for Warren.</p>
|
|
<pre><code class="language-python">
|
|
@hooks.hook('install')
|
|
def install():
|
|
'''Install required packages, user, and warren source.'''
|
|
apt_get_update()
|
|
ensure_packages(*dependencies)
|
|
host.adduser(owner)
|
|
prep_installation()
|
|
install_from_source()
|
|
</code></pre>
|
|
|
|
<p>The stop hook is similarly simple. It stops the Warren service and deletes the
|
|
upstart file for starting it.</p>
|
|
<pre><code class="language-python">
|
|
@hooks.hook('stop')
|
|
def stop():
|
|
'''Stop the warren service.'''
|
|
log('Stopping service...')
|
|
host.service_stop(system_service)
|
|
if upstart_conf:
|
|
unlink_if_exists(upstart_conf)
|
|
</code></pre>
|
|
|
|
<p>Here’s where the fun begins. As is standard practice for several charms, many
|
|
hooks should behave in the same way. This was put to me by Kapil Thangavelu as,
|
|
“There should only be a config-changed hook, and everything else is subordinate
|
|
to that.” This means that all or most relation hooks, the config-changed hook,
|
|
and the start hook should basically be the same.</p>
|
|
<p>Below, we’ve decorated the <code>main hook</code> method will most of our relation hooks,
|
|
start, and config-changed. The work this does is fairly straight forward. It
|
|
fetches the source and updates to the specified version if necessary, writes the
|
|
Warren config file, writes the upstart file, opens or closes ports as necessary,
|
|
and restarts the service.</p>
|
|
<pre><code class="language-python">
|
|
@hooks.hook('start')
|
|
@hooks.hook('config-changed')
|
|
@hooks.hook('mongodb-relation-joined')
|
|
@hooks.hook('mongodb-relation-departed')
|
|
@hooks.hook('mongodb-relation-broken')
|
|
@hooks.hook('mongodb-relation-changed')
|
|
@hooks.hook('elasticsearch-relation-joined')
|
|
@hooks.hook('elasticsearch-relation-departed')
|
|
@hooks.hook('elasticsearch-relation-broken')
|
|
@hooks.hook('elasticsearch-relation-changed')
|
|
def main_hook():
|
|
'''Main hook functionality
|
|
On most hooks, we simply need to write config files, work with hooks, and
|
|
restart. If the source has changed, we'll additionally need to rebuild.
|
|
'''
|
|
if config.changed('source'):
|
|
log('Source changed; rebuilding...')
|
|
install_from_source()
|
|
write_init_file()
|
|
write_config_file()
|
|
manage_ports()
|
|
restart()
|
|
</code></pre>
|
|
|
|
<p>In our case, the haproxy hooks take a little bit more work, however. The
|
|
haproxy service requires a bit of information from us: the hostname for this
|
|
unit of the Warren service, and the port on which it is listening. For each
|
|
<code>website</code> relation on this service, we simply send (using <code>relation_set</code>) those
|
|
data to the remote service.</p>
|
|
<pre><code class="language-python">
|
|
@hooks.hook('website-relation-joined')
|
|
@hooks.hook('website-relation-departed')
|
|
@hooks.hook('website-relation-broken')
|
|
@hooks.hook('website-relation-changed')
|
|
def website_relation_hook():
|
|
'''Notify all website relations of our address and port.'''
|
|
for relation_id in relations.get('website', {}).keys():
|
|
private_address = hookenv.unit_private_ip()
|
|
hookenv.relation_set(
|
|
relation_id=relation_id,
|
|
relation_settings={'hostname': private_address, 'port': config['listen_port']})
|
|
</code></pre>
|
|
|
|
<p>How are the hooks run? Simple. When the <code>hooks.py</code> file is called, we pass all
|
|
the work on to the charmhelpers library, which will decide which decorated hook
|
|
methods to call:</p>
|
|
<pre><code class="language-python">
|
|
if __name__ == "__main__":
|
|
hooks.execute(sys.argv)
|
|
</code></pre>
|
|
|
|
<h2 id="the-good">The Good</h2>
|
|
<p>There’s just so much to be said for having a repeatable, debuggable (I’ll get
|
|
into <code>juju debug-hooks</code> at some point, promise!) means of deploying a service.
|
|
With this layout for a charm, it’s easy to see what hook does what, and is
|
|
fairly easy to organize your code around that. The configuration files are in a
|
|
familiar and readable format (I’m looking at you, countless <code>*.pom</code> files), and
|
|
the python charmhelpers package keeps our hooks fairly simple.</p>
|
|
<h2 id="the-bad">The Bad</h2>
|
|
<p>I’ll be totally honest and say that a lot of the work that I did on this charm
|
|
came from observing the ways other charms were built, not by reading
|
|
documentation. I don’t mean to harp on this, but I simply had no other path
|
|
forward for creating my charm, there wasn’t much to read. Again, this is
|
|
something I’ll be focusing on helping along, myself.</p>
|
|
<p>My other problems stem from the issues involved with this path forward and may
|
|
be mitigated by utilizing the new services framework.</p>
|
|
<p>The <code>hooks.py</code> file is big, but there are enough hooks and enough code
|
|
repetition that it wouldn’t necessarily make sense to have it any other way.
|
|
There are a few other charms that have gotten big enough to divide the
|
|
deployment strategies into several different files and classes (notably the
|
|
<a href="https://jujucharms.com/juju-gui/">Juju GUI</a> charm) in sensible ways. In the
|
|
case of Warren, though there weren’t obvious break points, and yet the file
|
|
still feels relatively long.</p>
|
|
<h2 id="whats-next">What’s next</h2>
|
|
<p>In the next post, I’d like to go more in depth on the process of developing a
|
|
charm. That means going into <code>debug-hooks</code>, <code>juju ssh</code>, and a few other
|
|
commands that are useful for developing and debugging a charm.</p>
|
|
</article>
|
|
<footer>
|
|
<p>Page generated on 2024-05-04</p>
|
|
</footer>
|
|
</main>
|
|
<script type="text/javascript">
|
|
document.querySelectorAll('.tag').forEach(tag => {
|
|
let text = tag.innerText;
|
|
tag.innerText = '';
|
|
tag.innerHTML = `<a href="/tags.html#${text}">${text}</a>`;
|
|
});
|
|
</script>
|
|
</body>
|
|
</html>
|