Brokenhaze

The thoughts of a SysAdmin

Archive for the ‘Uncategorized’ Category

LOPSA Board – My Candidate Statement

without comments

A little about me

Hi, I’m George and I’m running for one of the open seats on the LOPSA Board.

I’ve been involved in LOPSA for about four years now. I’ve been involved with the NYC local since it started, and have been part of the LOPSA-EAST (Formally PICC) committee for the past three years.

I’ve been working as a System Administrator for the last 14 years. Currently I’m an SRE at Stack Exchange.

I believe that this organization has a lot of promise to be a leader in the field of system administration and would like to be part of those that have been entrusted by the membership to guide the organization to that position.

Ideas for Change

I think you would agree with me that LOPSA as an organization needs to evolve, and change so that it is around and more importantly relevant in the coming years.

The best ways to make sure an organization is relevant is to grow, and have a louder voice. “How do we grow?” “How do we have a louder voice?” The former comes up almost every year at election time, the latter I’ve heard come up more and more in the past year.

Below are my ideas on how to make LOPSA better.

How do we grow

1. Keep working on locals

I’m not going to spend much time on this, but obviously we need to continue the work on growing, and starting new local groups. Everyone says this every year, and I think we do a pretty good job of starting new locals and growing the ones we already have.

2. Online membership

Personally I think by not having some sort of “Online Local” we are leaving a lot on the table by not having some sort of viable online membership. As of 2012 the US Bureau of Labor Statistics says that there are 366,400 people what are classified as Network and Computer Systems Administrators. They are not all in the 15 localities that we have locals.

We need to setup a good online community that all members can access, but it should be geared towards trying to replicate some of the experience of a physical local.

  • Simulate the talks
    • Monthly Articles
    • Monthly Video Presentation
    • Dedicated talk area – forum, possibly comments section – for the above items
  • A way to emulate the before and after hang out sessions

How do we become more vocal

As an organization we need to become more vocal. We need to be putting out statements and positions on the important issues of they day. SOPA, Snowden, and Net Neutrality are all perfect examples of topics that LOPSA should have a vocal and strong opinion on that represents the majority of our membership.

One thing we need to make sure to do is not get stuck trying to get everyone to agree on. Any group will have differing opinions on every topic. One very important part of being vocal and putting our ideals out there is that you can’t put out a statement three weeks after the news has broken. What we need to do is have a group of people that are trusted by the membership to convey the majority opinion.

Finally, Focus

We need to look at everything that we are trying to do as an organization with a critical eye. There needs to be an examination of everything we are doing and see if it fits one of three categories 1) better the sysadmin community, 2) Being a leader in the community, 3) Growing the organization. If anything we are working on does not fit into one of those three categories we need to take a hard look at it and decide if we should continue with that venture.

It doesn’t need to be a hard stop, we will never pick this back up later, but we should stop spending limited resources on things that don’t move us closer to our goals. As we grow we will have more resources in time and money to be able to start re-introducing some of these initiatives that had to be stopped.

Written by George Beech

May 2nd, 2014 at 7:38 pm

Posted in Uncategorized

Puppet Workflow with Vagrant

without comments

I’ve spent a good deal of my time working with puppet over the last few years. And like most other things I’ve spent some time trying to optimize my workflow to avoid the annoying things. Just like anything else developing a good workflow for puppet has taken some time and I’d like to share the workflow that we have come to use over at Stack Exchange that seems to be pretty-darn-good.

Until recently our puppet dev workflow wasn’t horrible, however was a bit painful. Normally you would see something like (pg version):

commit -> wait for CI to pick up commit -> build failed -> damn -> fix typo -> commit -> wait for build -> build failed -> grrr -> fix new error -> commit … and on and on.

O, that way madness lies; let me shun that;
No more of that.
-William Shakespeare

Our new way of doing things is much, much saner. Which has in general raised team spirits!

The Dev Tier

We currently use Vagrant to do local dev work on new modules, and changes production modules. I have to say once it clicked that I can use Vagrant for puppet it was like that first sip of rocket fueled coffee in the morning. Everything suddenly became clear.

We have a vagrant file that uses two boxes – one is the client and the other is the server. They are actually seperate boxes. The client one is a box standard CentOS 6.4 minimal install. The server is the same but with all the puppetmaster bits all setup and ready to go. This includes Apache + Passenger, and all the CA goodness. It’s in a state just before the first run of puppetmasterd that creates the certificates.

First let’s look at the master config. There isn’t much special here but I do want to point out few things.

# Setup the Puppet master
  config.vm.define :master do |master|
    master.vm.box = "centos64-puppetm"
    master.vm.box_url = "http://<internal_server>/vagrant/vagrant-sei-puppetm-centos.x64.vb.box"
    master.vm.hostname = "master.local"
    master.vm.synced_folder "../../puppet-dev", "/etc/puppet"
    master.vm.synced_folder "../../scripts", "/root/scripts"
    master.vm.network :private_network, ip: "172.28.19.20"
    master.vm.provision :shell, :path => "master.sh"
    # Customize the actual virtual machine
    master.vm.provider :virtualbox do |vb|
      # Uncomment this, and adjust as needed to add memory to vm
      vb.customize ["modifyvm", :id, "--memory", 2048]
      # Because Virtual box is stupid - change the default nat network
      vb.customize ["modifyvm", :id, "--natnet1", "192.168.0.0/16"]
    end
  end

The first two highlights there are setting up synced folders. One is to the local puppet development repo on your disk, and the second is to a folder of our utility scripts.

This is actually the real magic. with our puppet development folder on our local host machine mounted at /etc/puppet this allows us to work in our local enviroment with all of our editing tools that we have come to love, and all we have to do is simply save our work and it is active in our Vagrant environment.

The next thing you will see highlighted is the master.vm.provision stanza. This is telling vagrant to use a shell (bash) based provisioner and to run the script that if finds in ./ on the host as it is starting up. There are a bunch of provisioners available for Vagrant. I’m using a shell provisioner here because I just want to do a couple of very very basic things to get the box in working order.

service ntpd stop
ntpdate pool.ntp.org
service ntpd start
service httpd stop
service puppetmaster start
service puppetmaster stop
service httpd start

As you can see, there is not much going on here. Syncing the time, stopping Apache/Passenger, starting then immediately stopping puppetmater to generate the certificates, then starting Apache/Passenger back up and we are ready to go.

The last two lines are simply modify the base VM to add more memory (Vagrant defaults to allocating 512M) and changing the NAT IP. We had to do the second one because VirtualBox defaults to using 10.2.0.0/24 for the nat network – which is a production network for us.

Our client configuration is almost exactly the same except the puppet folder is synced to /root/puppet for convenience and it pulls a different box down.

The provisioning script is also very basic for the client.

echo "172.28.19.20 puppet" >> /etc/hosts
service ntpd stop
ntpdate pool.ntp.org
service ntpd start
service puppet stop
rm -rf /var/lib/puppet/ssl/
service puppet start

The two biggest pieces here are lines one and six. Line one adds a hosts entry for the master puppet server so the client can bootstrap itself. And the sixth line makes sure any certs that might have been there are destroyed and lets puppet recreate them.

You can expand below to see the full source of our Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile which sets up one puppet master and one puppet client.
# Assumes "puppet-dev" and "scripts" repos are cloned into the same
# base directory.

Vagrant.configure("2") do |config|
  # All Vagrant configuration is done here. The most common configuration
  # options are documented and commented below. For a complete reference,
  # please see the online documentation at vagrantup.com.

  # Every Vagrant virtual environment requires a box to build off of.
  config.vm.box = "centos64"

  # Setup the Puppet master
  config.vm.define :master do |master|
    master.vm.box = "centos64-puppetm"
    master.vm.box_url = "http://<internal_server>/vagrant/vagrant-sei-puppetm-centos.x64.vb.box"
    master.vm.hostname = "master.local"
    master.vm.synced_folder "../../puppet-dev", "/etc/puppet"
    master.vm.synced_folder "../../scripts", "/root/scripts"
    master.vm.network :private_network, ip: "172.28.19.20"
    master.vm.provision :shell, :path => "master.sh"
    # Customize the actual virtual machine
    master.vm.provider :virtualbox do |vb|
      # Uncomment this, and adjust as needed to add memory to vm
      vb.customize ["modifyvm", :id, "--memory", 2048]
      # Because Virtual box is stupid - change the default nat network
      vb.customize ["modifyvm", :id, "--natnet1", "192.168.0.0/16"]
    end
  end

  # Setup the Puppet client. You can copy and modify this stanza to allow for
  # multiple client, just change all instances of 'client1' to another term
  # such as 'client2'
  config.vm.define :client1 do |client1|
    client1.vm.box = "centos64"
    client1.vm.box_url = "http://<internal_server>/vagrant/vagrant-sei-centos64.x64.vb.box"
    client1.vm.hostname = "client1"
    # Make puppet-dev accessable from the client for easier copying.
    client1.vm.synced_folder "../../puppet-dev", "/root/puppet"
    client1.vm.network :private_network, ip: "172.28.19.21"
    #client1.vm.network :forwarded_port, guest: 8100, host: 8100
    client1.vm.provision :shell, :path => "client.sh"
    client1.vm.provider :virtualbox do |vb|
      # Uncomment this, and adjust as needed to add memory to vm
      vb.customize ["modifyvm", :id, "--memory", 2048]
      # Because Virtual box is stupid - change the default nat network
      vb.customize ["modifyvm", :id, "--natnet1", "192.168.0.0/16"]
    end
  end

  if false
    config.vm.define :client2 do |client2|
      client2.vm.box = "centos64"
      client2.vm.box_url = "http://ny-man02.ds.stackexchange.com/vagrant/vagrant-sei-centos64.x64.vb.box"
      client2.vm.hostname = "client2"
      # Make puppet-dev accessable from the client for easier copying.
      client1.vm.synced_folder "../../puppet-dev", "/root/puppet"
      client2.vm.network :private_network, ip: "172.28.19.22"
      client2.vm.provision :shell, :path => "client.sh"
      # Customize the actual virtual machine
      client2.vm.provider :virtualbox do |vb|
        # Uncomment this, and adjust as needed to add memory to vm
        vb.customize ["modifyvm", :id, "--memory", 2048]
        # Because Virtual box is stupid - change the default nat network
        vb.customize ["modifyvm", :id, "--natnet1", "192.168.0.0/16"]
      end
    end
  end
 
end

So, what exactly does doing a dev setup like this help you with? It keeps you from having to constantly push to your testing environment to test every change. Which is huge – especially when you have a habbit of missing typo’s that puppet-lint doesn’t catch. You can smoke test on a local VM and iterate extremely quickly. Avoid madness, don’t spam internal chat with build messages. It’s a win-win-win to me. For those that don’t know my handle on the SE network is Zypher … so Nick is talking about me descending into madness.

puppet-madnexx

Test and Prod

I’m going to talk about the test and prod environments together here. The only functional difference between the two is the puppet module code that is run in those environments so for the purposes of this exercise they are the same.

What happens after you are done working locally and have a working change? Well that is simple, you push the change set(s) from your local clone to our mercurial server. You can obviously swap mercurial with any VCS system you like.

Once the changes have been pushed up the server, our CI Server (TeamCity) will pull the changes down and “build” them. Since nothing is really compiled with puppet what it is actually doing is running through a battery of tests, then if they pass it pushes the changes out to our puppet masters.

  1. Run a bash script that checks the changeset against puppet validate
  2. Run a bash script that generates the puppet docs for the changeset
  3. Push the changes to the puppet master servers

We have bot in our internal chat that will post a message with the success/failure of the build and a link to the build info page inside TeamCity. Once you have a successful build then your change is live in Test or Prod depending on which repo you pushed to.

That’s about it. There is nothing all that special or groundbreaking here. All the magic really happens at the local dev stage with Vagrant. The sanity checks on the server side before deployment are really just to make extra sure since you worked all the bugs out, right?

I’m sure that there is yet more room for improvement to our process but at this point i’m happy to say the decent into madness has been staved off for now.

Written by George Beech

July 25th, 2013 at 10:00 am