Puppet+Nginx+Unicorn

This page should outline how to set up puppet behind and nginx proxy with a unicorn ruby webserver to replace the default WEBrick. The purpose of doing this is to firstly speed up puppet's file access and script running and then to proxy it so that nginx can handle all of the client requests with cached data.

Prerequisites

Running behind an nginx proxy requires puppet packages of version 2.6.1 or higher including clients as far as I am aware (tested on 2.6.2) due to a bug in 2.6.0 where an extra / gets inserted into the file content path by the client causing the file content to 404 but not the metadata, meaning that any files distributed by puppet exist but are empty.

Package versions

This guide was written using the following package versions on debian lenny, as a result, YMMV:

• rubygems1.8 1.3.4-1~bpo50+1 via backports <- REQUIRED min version for unicorn.
• EDIT:rubygems 1.3.4-1~bpo50+1 via backports <- Not actually required for this, but many things require the meta-package as well as the versioned package so getting them together can save you some headaches.
• librack-ruby 1.1.0-4~bpo50+1 via backports <- REQUIRED min version for unicorn
• nginx 0.7.65-2~bpo50+1 via backports <- if you can use a newer version you may be able to use a conditional nginx config for the handling of certificate signing instead of my config below.
• puppet-common 2.6.2-4~bpo50+1 via backports
  
• puppetmaster 2.6.2-4~bpo50+1 via backports
  

I found that the easiest thing to do was to set up a local repo before starting, as it saves you scping things around and getting in to all sorts of confusing shenanigans.

Setting up the puppetmaster

1. Install Debian
   
2. apt-get install rubygems1.8 librack-ruby puppetmaster puppet-common nginx make ruby1.8-dev
   
3. gem install unicorn
   

Configuration

unicorn

Copy /var/lib/gems/1.8/gems/rack-1.2.1/test/rackup/config.ru to /etc/puppet/config.ru (you will be running unicorn from /etc/puppet, and it will look for the rack config "config.ru" in this directory.

Create /etc/puppet/unicorn.conf as follows:

worker_processes 8
working_directory "/etc/puppet"
listen 'unix:/var/run/puppet/puppetmaster_unicorn.sock', :backlog => 512
timeout 120
pid "/var/run/puppet/puppetmaster_unicorn.pid"

preload_app true
if GC.respond_to?(:copy_on_write_friendly=)
  GC.copy_on_write_friendly = true
end

before_fork do |server, worker|
  old_pid = "#{server.config[:pid]}.oldbin"
  if File.exists?(old_pid) && server.pid != old_pid
    begin
      Process.kill("QUIT", File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end
end

Unicorn listens on an internal unix socket which will be proxied by nginx. As unicorn manages it's own worker processes via a unicorn-master, this means that unlike mongrel only one socket need be used.

Edit /etc/default/puppetmaster and change the following two options:

SERVERTYPE=unicorn
PORT=18140

This is so that a) puppetmaster uses the unicorn webserver and b) so that it doesn't try to run it on the port we will be configuring nginx to listen on.

Many guides try to get you to install the ruby gem "god" at this point to manage the unicorn process, but for the sake of simplicity, clarity and sanity we will use an init script /etc/init.d/unicorn-puppet (don't forget to chmod 755):

#!/bin/bash
# unicorn-puppet         This init script enables the puppetmaster rackup application
#                        via unicorn.
#
# Authors:               Richard Crowley 
#                        Naresh V. 
#
#    Modified for Debian usage by Matt Carroll
#    
#

lockfile=/var/lock/puppetmaster-unicorn
pidfile=/var/run/puppet/puppetmaster_unicorn.pid

RETVAL=0
DAEMON=/var/lib/gems/1.8/bin/unicorn
DAEMON_OPTS="-D -c /etc/puppet/unicorn.conf"


start() {
    sudo -u $USER $DAEMON $DAEMON_OPTS
    RETVAL=$?
    [ $RETVAL -eq 0 ] && touch "$lockfile"
    echo
    return $RETVAL
}

stop() {
    sudo -u $USER kill `cat $pidfile`
    RETVAL=$?
    echo
    [ $RETVAL -eq 0 ] && rm -f "$lockfile"
    return $RETVAL
}

restart() {
    stop
    sleep 1
    start
    RETVAL=$?
    echo
    [ $RETVAL -ne 0 ] && rm -f "$lockfile"
    return $RETVAL
}

condrestart() {
    status
    RETVAL=$?
    [ $RETVAL -eq 0 ] && restart
}

status() {
    ps ax | egrep -q "unicorn (worker|master)"
    RETVAL=$?
    return $RETVAL
}

usage() {
    echo "Usage: $0 {start|stop|restart|status|condrestart}" >&2
    return 3
}

case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    restart)
        restart
        ;;
    condrestart)
        condrestart
        ;;
    status)
        status
        ;;
    *)
        usage
        ;;
esac

exit $RETVAL

Also, don't forget to add this to your startup scripts!

nginx

/etc/nginx/nginx.conf

user www-data;
worker_processes  2;

events {
        worker_connections  1024;
}

http {
        default_type  application/x-raw;
 large_client_header_buffers 16 8k;

        # site specific settings such as access_log, proxy_buffers
        # I use
         proxy_max_temp_file_size 0;
         proxy_buffers 128 4k;

        upstream muppet-upstream {
                server unix:/var/run/puppet/puppetmaster_unicorn.sock;
        }

        server {
                listen 8140;
                include /etc/nginx/conf.d/ssl.conf;
  ssl_verify_client on;
  root /usr/share/empty;
                location / {
                        proxy_pass http://muppet-upstream;
                 include /etc/nginx/conf.d/proxy_set_header.conf;
   proxy_set_header X-Client-Verify SUCCESS;
                }
        }
 server {
  listen 8141;
                include /etc/nginx/conf.d/ssl.conf;
  ssl_verify_client off;
  root /usr/share/empty;
  location / {
   proxy_pass http://muppet-upstream;
   include /etc/nginx/conf.d/proxy_set_header.conf;
   proxy_set_header X-Client-Verify FAILURE;
  }
 }
}

You may wish to change "muppet-upstream" to a more suitable name. There are two servers listening on two different ports, as our version of nginx can't handle SSL certification conditionals, so the standard port 8140 is used for the regular service for signed clients, and 8141 is used for signing requests. The setting of the proxy headers is important, as the nginx proxy is where the SSL layer terminates, so the information regarding certification success or failure and the hostname must be passed on using HTTP headers, most of which are configured in the conf.d/proxy_set_headers.conf:

proxy_redirect         off;
proxy_set_header Host              $host;
proxy_set_header X-Real-IP         $remote_addr;
proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
#proxy_set_header    X-Client-Verify  $ssl_client_verify;
proxy_set_header    X-Client-DN $ssl_client_s_dn;
proxy_set_header    X-SSL-Subject    $ssl_client_s_dn;
proxy_set_header    X-SSL-Issuer     $ssl_client_i_dn;

SSL certification is configured in conf.d/ssl.conf:

ssl on;
ssl_certificate /var/lib/puppet/ssl/certs/muppetmaster.labs.pem;
ssl_certificate_key /var/lib/puppet/ssl/private_keys/muppetmaster.labs.pem;
ssl_ciphers ALL:-ADH:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP;
ssl_client_certificate  /var/lib/puppet/ssl/ca/ca_crt.pem;
 
#ssl_crl                 /var/lib/puppet/ssl/ca/ca_crl.pem;
 

The headers that the puppetmaster should use must now be configured in /etc/puppet/puppet.conf by adding the section:

[master]
ssl_client_header = HTTP_X_CLIENT_DN
ssl_client_verify_header = HTTP_X_CLIENT_VERIFY

(this section was previously called puppetmasterd in 2.6.0 and earlier, so if the section is named that you will get a message telling you it assumes you meant [master]. )

Testing and initial run

Firstly ensure that all of your daemons (puppetmaster, unicorn and nginx) are started up on the master. Then, on the client stop the puppet daemon and run:

puppetd --test --verbose --debug --detailed-exitcodes --ca_port 8141 --waitforcert 10

Which will give you verbose output for the signing request. If all goes smoothly, you can then sign the request on the master, and restart the puppet daemon on the client which will continue to run normally on the default port (8140) now that the SSL has been verified. Signing requests will 404 on this port.

When debugging problems, running daemons in non-daemonized verbose debug mode is your friend.