throw != raise, catch != rescue

This isn't even sysadmin stuff here, so if you don't care about Ruby, look away.

It is increasingly irritating when I'm trying to google examples of usage of Ruby's throw and catch functionality, that people who are very confused about the terminology of exceptions in Ruby appear to be writing tutorials. I can understand that they're writing about it coming from the perspective of some other language, but in Ruby you don't *throw* an exception, nor do you *catch* it because throw and catch are something different. The terminology is important because one set deals with exceptions (unexpected errors) and the other doesn't.

I want to see examples of throw and catch and it's made very difficult by the fact that everyone seems to use the exception stack for flow control (shudder) and referring to it by throw and catch, meaning I keep ending up reading badly worded tutorials on the exception handling tools which I already understand.

/rant

MCollective and MongoDB registration.

I'm currently working on mashing our mcollective and puppet rigs together into one big nodelessly configured family by using Whack's nodeless setup, hacked about to search node roles from the same mongodb that Volcane's mcollective registration and puppet searching. The idea here is to be able to add user-set "roles" to each node in the database - something that is only ever set by hand unlike the registration data which should only ever be set by a node.

In order to do this, I hacked Volcane's mongodb registration code about a bit so that it would only do atomic updates rather than blatting the entire document whenever it updated, meaning you can put in user defined fields as well. Then the guys who wrote the original code did it on the main branch (albeit as a side effect to fixing other things) but in a much less shitty way because they can code and I tend to do something more akin to hitting ruby with a stick.

I'm now back to working on my frontend to assign these roles through a point and drool interface.

Git

It's been a while.

So this is less of an entry and more of a link to something which has made a lot of sense to me and is a really nice model for developing code - anyone who knows me knows I come from a systems background but have been writing more and more code of late, so this stuff is largely new to me.

http://nvie.com/posts/a-successful-git-branching-model/

This outlines (as you might imagine) a very logical and concise way of laying out a git tree and I love it with mouth because it's tidy.

Edit: http://jeffkreeftmeijer.com/2010/why-arent-you-using-git-flow/ and there's a tool to make it easy too!

Backports and meta-packages

I've just made an edit on the Puppet+Nginx+Unicorn entry a couple of posts back to include:

• rubygems 1.3.4-1~bpo50+1 via backports <- Not actually required for this, but many things require the meta-package as well as the versioned package so getting them together can save you some headaches.

I think this is fairly important as a rule of thumb - if you're installing from backports, you should really check for the associated meta-packages as it can cause dependancy problems later on with other packages should you not have them. I was just installing mcollective on my puppetmaster test rig, and I ran in to this problem.

As I've said before, it's usually easiest to set up your own debian repository using reprepro if you're going to install custom-baked debs simply because it saves on headaches.

Migrating puppetmaster to unicorn+nginx from WEBrick

As a followup to my last post I've been thinking about the best way to migrate over a production server and be able to test the new config without blatting what's currently running so that the new setup can be tested before blowing the old one away, so what I did was to test having my unicorn setup running and then a separate instance running the standard WEBrick.

On the puppetmaster (with the nginx+unicorn setup already running) I ran:

/usr/bin/ruby1.8 /usr/bin/puppet master --masterport=18141 --servertype=webrick --pidfile /var/run/puppet/thingy.pid --no-daemonize --verbose --debug

and on my test client:

puppetd --test --verbose --debug --detailed-exitcodes --masterport 18141

With a verified client, this should run just fine, and you should see a load of output scroll past on your non-daemonized puppetmaster instance. This shows that we can (as the documentation would imply) run multiple instances of puppetmasterd via different webservers on different ports, so it is possible to set up and test an entirely different puppetmaster instance using nginx+unicorn whilst the production instance is running. Then it's just a case of changing the config and changing the ports.

Puppet+Nginx+Unicorn

This page should outline how to set up puppet behind and nginx proxy with a unicorn ruby webserver to replace the default WEBrick. The purpose of doing this is to firstly speed up puppet's file access and script running and then to proxy it so that nginx can handle all of the client requests with cached data.

Prerequisites

Running behind an nginx proxy requires puppet packages of version 2.6.1 or higher including clients as far as I am aware (tested on 2.6.2) due to a bug in 2.6.0 where an extra / gets inserted into the file content path by the client causing the file content to 404 but not the metadata, meaning that any files distributed by puppet exist but are empty.

Package versions

This guide was written using the following package versions on debian lenny, as a result, YMMV:

• rubygems1.8 1.3.4-1~bpo50+1 via backports <- REQUIRED min version for unicorn.
• EDIT:rubygems 1.3.4-1~bpo50+1 via backports <- Not actually required for this, but many things require the meta-package as well as the versioned package so getting them together can save you some headaches.
• librack-ruby 1.1.0-4~bpo50+1 via backports <- REQUIRED min version for unicorn
• nginx 0.7.65-2~bpo50+1 via backports <- if you can use a newer version you may be able to use a conditional nginx config for the handling of certificate signing instead of my config below.
• puppet-common 2.6.2-4~bpo50+1 via backports
  
• puppetmaster 2.6.2-4~bpo50+1 via backports
  

I found that the easiest thing to do was to set up a local repo before starting, as it saves you scping things around and getting in to all sorts of confusing shenanigans.

Setting up the puppetmaster

1. Install Debian
   
2. apt-get install rubygems1.8 librack-ruby puppetmaster puppet-common nginx make ruby1.8-dev
   
3. gem install unicorn
   

Configuration

unicorn

Copy /var/lib/gems/1.8/gems/rack-1.2.1/test/rackup/config.ru to /etc/puppet/config.ru (you will be running unicorn from /etc/puppet, and it will look for the rack config "config.ru" in this directory.

Create /etc/puppet/unicorn.conf as follows:

worker_processes 8
working_directory "/etc/puppet"
listen 'unix:/var/run/puppet/puppetmaster_unicorn.sock', :backlog => 512
timeout 120
pid "/var/run/puppet/puppetmaster_unicorn.pid"

preload_app true
if GC.respond_to?(:copy_on_write_friendly=)
  GC.copy_on_write_friendly = true
end

before_fork do |server, worker|
  old_pid = "#{server.config[:pid]}.oldbin"
  if File.exists?(old_pid) && server.pid != old_pid
    begin
      Process.kill("QUIT", File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end
end

Unicorn listens on an internal unix socket which will be proxied by nginx. As unicorn manages it's own worker processes via a unicorn-master, this means that unlike mongrel only one socket need be used.

Edit /etc/default/puppetmaster and change the following two options:

SERVERTYPE=unicorn
PORT=18140

This is so that a) puppetmaster uses the unicorn webserver and b) so that it doesn't try to run it on the port we will be configuring nginx to listen on.

Many guides try to get you to install the ruby gem "god" at this point to manage the unicorn process, but for the sake of simplicity, clarity and sanity we will use an init script /etc/init.d/unicorn-puppet (don't forget to chmod 755):

#!/bin/bash
# unicorn-puppet         This init script enables the puppetmaster rackup application
#                        via unicorn.
#
# Authors:               Richard Crowley 
#                        Naresh V. 
#
#    Modified for Debian usage by Matt Carroll
#    
#

lockfile=/var/lock/puppetmaster-unicorn
pidfile=/var/run/puppet/puppetmaster_unicorn.pid

RETVAL=0
DAEMON=/var/lib/gems/1.8/bin/unicorn
DAEMON_OPTS="-D -c /etc/puppet/unicorn.conf"


start() {
    sudo -u $USER $DAEMON $DAEMON_OPTS
    RETVAL=$?
    [ $RETVAL -eq 0 ] && touch "$lockfile"
    echo
    return $RETVAL
}

stop() {
    sudo -u $USER kill `cat $pidfile`
    RETVAL=$?
    echo
    [ $RETVAL -eq 0 ] && rm -f "$lockfile"
    return $RETVAL
}

restart() {
    stop
    sleep 1
    start
    RETVAL=$?
    echo
    [ $RETVAL -ne 0 ] && rm -f "$lockfile"
    return $RETVAL
}

condrestart() {
    status
    RETVAL=$?
    [ $RETVAL -eq 0 ] && restart
}

status() {
    ps ax | egrep -q "unicorn (worker|master)"
    RETVAL=$?
    return $RETVAL
}

usage() {
    echo "Usage: $0 {start|stop|restart|status|condrestart}" >&2
    return 3
}

case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    restart)
        restart
        ;;
    condrestart)
        condrestart
        ;;
    status)
        status
        ;;
    *)
        usage
        ;;
esac

exit $RETVAL

Also, don't forget to add this to your startup scripts!

nginx

/etc/nginx/nginx.conf

user www-data;
worker_processes  2;

events {
        worker_connections  1024;
}

http {
        default_type  application/x-raw;
 large_client_header_buffers 16 8k;

        # site specific settings such as access_log, proxy_buffers
        # I use
         proxy_max_temp_file_size 0;
         proxy_buffers 128 4k;

        upstream muppet-upstream {
                server unix:/var/run/puppet/puppetmaster_unicorn.sock;
        }

        server {
                listen 8140;
                include /etc/nginx/conf.d/ssl.conf;
  ssl_verify_client on;
  root /usr/share/empty;
                location / {
                        proxy_pass http://muppet-upstream;
                 include /etc/nginx/conf.d/proxy_set_header.conf;
   proxy_set_header X-Client-Verify SUCCESS;
                }
        }
 server {
  listen 8141;
                include /etc/nginx/conf.d/ssl.conf;
  ssl_verify_client off;
  root /usr/share/empty;
  location / {
   proxy_pass http://muppet-upstream;
   include /etc/nginx/conf.d/proxy_set_header.conf;
   proxy_set_header X-Client-Verify FAILURE;
  }
 }
}

You may wish to change "muppet-upstream" to a more suitable name. There are two servers listening on two different ports, as our version of nginx can't handle SSL certification conditionals, so the standard port 8140 is used for the regular service for signed clients, and 8141 is used for signing requests. The setting of the proxy headers is important, as the nginx proxy is where the SSL layer terminates, so the information regarding certification success or failure and the hostname must be passed on using HTTP headers, most of which are configured in the conf.d/proxy_set_headers.conf:

proxy_redirect         off;
proxy_set_header Host              $host;
proxy_set_header X-Real-IP         $remote_addr;
proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
#proxy_set_header    X-Client-Verify  $ssl_client_verify;
proxy_set_header    X-Client-DN $ssl_client_s_dn;
proxy_set_header    X-SSL-Subject    $ssl_client_s_dn;
proxy_set_header    X-SSL-Issuer     $ssl_client_i_dn;

SSL certification is configured in conf.d/ssl.conf:

ssl on;
ssl_certificate /var/lib/puppet/ssl/certs/muppetmaster.labs.pem;
ssl_certificate_key /var/lib/puppet/ssl/private_keys/muppetmaster.labs.pem;
ssl_ciphers ALL:-ADH:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP;
ssl_client_certificate  /var/lib/puppet/ssl/ca/ca_crt.pem;
 
#ssl_crl                 /var/lib/puppet/ssl/ca/ca_crl.pem;
 

The headers that the puppetmaster should use must now be configured in /etc/puppet/puppet.conf by adding the section:

[master]
ssl_client_header = HTTP_X_CLIENT_DN
ssl_client_verify_header = HTTP_X_CLIENT_VERIFY

(this section was previously called puppetmasterd in 2.6.0 and earlier, so if the section is named that you will get a message telling you it assumes you meant [master]. )

Testing and initial run

Firstly ensure that all of your daemons (puppetmaster, unicorn and nginx) are started up on the master. Then, on the client stop the puppet daemon and run:

puppetd --test --verbose --debug --detailed-exitcodes --ca_port 8141 --waitforcert 10

Which will give you verbose output for the signing request. If all goes smoothly, you can then sign the request on the master, and restart the puppet daemon on the client which will continue to run normally on the default port (8140) now that the SSL has been verified. Signing requests will 404 on this port.

When debugging problems, running daemons in non-daemonized verbose debug mode is your friend.

 

My new blog for PITFA systems administration

Hi, I'm Matt and I'm a sysadmin. At the moment I mainly work on setting up expandible infrastructure in order to make future expansion of systems more manageable, and to introduce automation to everyday tasks such as server provisioning.

This is my blog, "prontab" to explain how I do what I do with certain pieces of software and system setups that have many caveats and pitfalls that I've had to deal with in the past. The idea is that once I've bashed my head against a brick wall, others shouldn't have to!

Feel free to post questions and the like, but don't expect me to know all of the answers or to debug anything for you. My dayjob is to get these things to work to a degree of functionality that is useful for my purposes, no more and no less!