Brokenhaze

The thoughts of a SysAdmin

Puppet Workflow with Vagrant

without comments

I’ve spent a good deal of my time working with puppet over the last few years. And like most other things I’ve spent some time trying to optimize my workflow to avoid the annoying things. Just like anything else developing a good workflow for puppet has taken some time and I’d like to share the workflow that we have come to use over at Stack Exchange that seems to be pretty-darn-good.

Until recently our puppet dev workflow wasn’t horrible, however was a bit painful. Normally you would see something like (pg version):

commit -> wait for CI to pick up commit -> build failed -> damn -> fix typo -> commit -> wait for build -> build failed -> grrr -> fix new error -> commit … and on and on.

O, that way madness lies; let me shun that;
No more of that.
-William Shakespeare

Our new way of doing things is much, much saner. Which has in general raised team spirits!

The Dev Tier

We currently use Vagrant to do local dev work on new modules, and changes production modules. I have to say once it clicked that I can use Vagrant for puppet it was like that first sip of rocket fueled coffee in the morning. Everything suddenly became clear.

We have a vagrant file that uses two boxes – one is the client and the other is the server. They are actually seperate boxes. The client one is a box standard CentOS 6.4 minimal install. The server is the same but with all the puppetmaster bits all setup and ready to go. This includes Apache + Passenger, and all the CA goodness. It’s in a state just before the first run of puppetmasterd that creates the certificates.

First let’s look at the master config. There isn’t much special here but I do want to point out few things.

# Setup the Puppet master
  config.vm.define :master do |master|
    master.vm.box = "centos64-puppetm"
    master.vm.box_url = "http://<internal_server>/vagrant/vagrant-sei-puppetm-centos.x64.vb.box"
    master.vm.hostname = "master.local"
    master.vm.synced_folder "../../puppet-dev", "/etc/puppet"
    master.vm.synced_folder "../../scripts", "/root/scripts"
    master.vm.network :private_network, ip: "172.28.19.20"
    master.vm.provision :shell, :path => "master.sh"
    # Customize the actual virtual machine
    master.vm.provider :virtualbox do |vb|
      # Uncomment this, and adjust as needed to add memory to vm
      vb.customize ["modifyvm", :id, "--memory", 2048]
      # Because Virtual box is stupid - change the default nat network
      vb.customize ["modifyvm", :id, "--natnet1", "192.168.0.0/16"]
    end
  end

The first two highlights there are setting up synced folders. One is to the local puppet development repo on your disk, and the second is to a folder of our utility scripts.

This is actually the real magic. with our puppet development folder on our local host machine mounted at /etc/puppet this allows us to work in our local enviroment with all of our editing tools that we have come to love, and all we have to do is simply save our work and it is active in our Vagrant environment.

The next thing you will see highlighted is the master.vm.provision stanza. This is telling vagrant to use a shell (bash) based provisioner and to run the script that if finds in ./ on the host as it is starting up. There are a bunch of provisioners available for Vagrant. I’m using a shell provisioner here because I just want to do a couple of very very basic things to get the box in working order.

service ntpd stop
ntpdate pool.ntp.org
service ntpd start
service httpd stop
service puppetmaster start
service puppetmaster stop
service httpd start

As you can see, there is not much going on here. Syncing the time, stopping Apache/Passenger, starting then immediately stopping puppetmater to generate the certificates, then starting Apache/Passenger back up and we are ready to go.

The last two lines are simply modify the base VM to add more memory (Vagrant defaults to allocating 512M) and changing the NAT IP. We had to do the second one because VirtualBox defaults to using 10.2.0.0/24 for the nat network – which is a production network for us.

Our client configuration is almost exactly the same except the puppet folder is synced to /root/puppet for convenience and it pulls a different box down.

The provisioning script is also very basic for the client.

echo "172.28.19.20 puppet" >> /etc/hosts
service ntpd stop
ntpdate pool.ntp.org
service ntpd start
service puppet stop
rm -rf /var/lib/puppet/ssl/
service puppet start

The two biggest pieces here are lines one and six. Line one adds a hosts entry for the master puppet server so the client can bootstrap itself. And the sixth line makes sure any certs that might have been there are destroyed and lets puppet recreate them.

You can expand below to see the full source of our Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile which sets up one puppet master and one puppet client.
# Assumes "puppet-dev" and "scripts" repos are cloned into the same
# base directory.

Vagrant.configure("2") do |config|
  # All Vagrant configuration is done here. The most common configuration
  # options are documented and commented below. For a complete reference,
  # please see the online documentation at vagrantup.com.

  # Every Vagrant virtual environment requires a box to build off of.
  config.vm.box = "centos64"

  # Setup the Puppet master
  config.vm.define :master do |master|
    master.vm.box = "centos64-puppetm"
    master.vm.box_url = "http://<internal_server>/vagrant/vagrant-sei-puppetm-centos.x64.vb.box"
    master.vm.hostname = "master.local"
    master.vm.synced_folder "../../puppet-dev", "/etc/puppet"
    master.vm.synced_folder "../../scripts", "/root/scripts"
    master.vm.network :private_network, ip: "172.28.19.20"
    master.vm.provision :shell, :path => "master.sh"
    # Customize the actual virtual machine
    master.vm.provider :virtualbox do |vb|
      # Uncomment this, and adjust as needed to add memory to vm
      vb.customize ["modifyvm", :id, "--memory", 2048]
      # Because Virtual box is stupid - change the default nat network
      vb.customize ["modifyvm", :id, "--natnet1", "192.168.0.0/16"]
    end
  end

  # Setup the Puppet client. You can copy and modify this stanza to allow for
  # multiple client, just change all instances of 'client1' to another term
  # such as 'client2'
  config.vm.define :client1 do |client1|
    client1.vm.box = "centos64"
    client1.vm.box_url = "http://<internal_server>/vagrant/vagrant-sei-centos64.x64.vb.box"
    client1.vm.hostname = "client1"
    # Make puppet-dev accessable from the client for easier copying.
    client1.vm.synced_folder "../../puppet-dev", "/root/puppet"
    client1.vm.network :private_network, ip: "172.28.19.21"
    #client1.vm.network :forwarded_port, guest: 8100, host: 8100
    client1.vm.provision :shell, :path => "client.sh"
    client1.vm.provider :virtualbox do |vb|
      # Uncomment this, and adjust as needed to add memory to vm
      vb.customize ["modifyvm", :id, "--memory", 2048]
      # Because Virtual box is stupid - change the default nat network
      vb.customize ["modifyvm", :id, "--natnet1", "192.168.0.0/16"]
    end
  end

  if false
    config.vm.define :client2 do |client2|
      client2.vm.box = "centos64"
      client2.vm.box_url = "http://ny-man02.ds.stackexchange.com/vagrant/vagrant-sei-centos64.x64.vb.box"
      client2.vm.hostname = "client2"
      # Make puppet-dev accessable from the client for easier copying.
      client1.vm.synced_folder "../../puppet-dev", "/root/puppet"
      client2.vm.network :private_network, ip: "172.28.19.22"
      client2.vm.provision :shell, :path => "client.sh"
      # Customize the actual virtual machine
      client2.vm.provider :virtualbox do |vb|
        # Uncomment this, and adjust as needed to add memory to vm
        vb.customize ["modifyvm", :id, "--memory", 2048]
        # Because Virtual box is stupid - change the default nat network
        vb.customize ["modifyvm", :id, "--natnet1", "192.168.0.0/16"]
      end
    end
  end
 
end

So, what exactly does doing a dev setup like this help you with? It keeps you from having to constantly push to your testing environment to test every change. Which is huge – especially when you have a habbit of missing typo’s that puppet-lint doesn’t catch. You can smoke test on a local VM and iterate extremely quickly. Avoid madness, don’t spam internal chat with build messages. It’s a win-win-win to me. For those that don’t know my handle on the SE network is Zypher … so Nick is talking about me descending into madness.

puppet-madnexx

Test and Prod

I’m going to talk about the test and prod environments together here. The only functional difference between the two is the puppet module code that is run in those environments so for the purposes of this exercise they are the same.

What happens after you are done working locally and have a working change? Well that is simple, you push the change set(s) from your local clone to our mercurial server. You can obviously swap mercurial with any VCS system you like.

Once the changes have been pushed up the server, our CI Server (TeamCity) will pull the changes down and “build” them. Since nothing is really compiled with puppet what it is actually doing is running through a battery of tests, then if they pass it pushes the changes out to our puppet masters.

  1. Run a bash script that checks the changeset against puppet validate
  2. Run a bash script that generates the puppet docs for the changeset
  3. Push the changes to the puppet master servers

We have bot in our internal chat that will post a message with the success/failure of the build and a link to the build info page inside TeamCity. Once you have a successful build then your change is live in Test or Prod depending on which repo you pushed to.

That’s about it. There is nothing all that special or groundbreaking here. All the magic really happens at the local dev stage with Vagrant. The sanity checks on the server side before deployment are really just to make extra sure since you worked all the bugs out, right?

I’m sure that there is yet more room for improvement to our process but at this point i’m happy to say the decent into madness has been staved off for now.

Written by George Beech

July 25th, 2013 at 10:00 am

Time Flies

without comments

Excuse me for a bit while I get a little sentimental. Tomorrow (the 27th of September) marks my two year anniversary with Stack Exchange – it was still called Stack Overflow Internet Services when I joined! I realized this about two or three weeks ago when I was setting up my trip to Denver for the Opening Party. It just amazes me how quickly time flies when you are having fun.

It’s amazing to me how much the company has grown, how much has been accomplished, and the amazing people I’ve met while working here. It’s a little bit awe inspiring to look back at the day I started and what the company looked like then vs where we are at now.

On my first day the space at 55 Broadway hadn’t been renovated yet, they had done some demo, but that was about it. It was a puke green color, and there was a lot of work that needed to be done. I think there where 10 people in the office when I started. We lived in one half of the office, while they worked in the other half building things then we would switch sides; it was just a mess of wires strewn across the floor and a bunch of desks clumped together (along with some lucky people that got to share the few offices!)

Now, we are rapidly running out of room at 55 Broadway and have opened an office in Denver, with other great plans to expand and grow.

As far as the sites, we’ve gone from around the 300 mark to flirting with breaking the top 100 on Quantcast since I’ve been here. There have been many fun challenges, and lots of hard work to get us to where we are today.

As I sit here at 10,000 ft. writing this all I can think of is how this is exactly what I envisioned all these years ago when I decided that at least once in my life I wanted to work for a start up. How awesome it is to have a dream come true, and have it be exactly what you thought it would be.

Written by George Beech

September 26th, 2012 at 11:16 pm

Getting the most out of your Sysadmin team

without comments

Disclaimer: Most of my career i’ve been what you would call and “Individual Contributor” these are not observations from a managerial perspective, but from a team member perspective

In the past 10 years or so I’ve worked under pretty much all your basic boss types. I’ve seen how people react to different stimuli and managerial styles. ┬áJust like with our programmer cousins we sysadmins can be a peculiar type of person, who isn’t easily managed in the “traditional” style that is still pervasive though out the business world. We are the type of people that are really good at analyzing a problem, and thinking of – sometime very creative – ways to get around the problem.

When I look back at all of the situations that I have been the most productive in, and have seen others be the most productive in one theme keeps coming up over and over again. That is, let your smart people think and have free reign while NOT overloading them. The latter part is really important, there is a tendency to find one or two really good people, and put a supporting cast around them, thinking that the supporting cast will help out with some of the lower level things. While this seems like a great idea, in practice it never really turns out this way. You end up with only the most minor of duties being taken care of, and you rockstar having to go back and still do a lot of the low level things that take away from what you want them working on. Or, even worse, they have to go back and FIX problems caused by someone who doesn’t know thier limits.

Ideally you want a team of sysadmins that compliment each other, so they can support each other when needed but don’t step all over the other people’s toes. This doesn’t mean you need a team of all star generalists it means you need a team of people that are good at what they do, know their limits and most importantly want to grow as a professional not just stick to the same old things that they know.

So, once you have assembled this team, the question now becomes how to I get the most out of them? How do I motivate them into doing the best that they can do and help them when they are struggling? These are not easy answers at all, and my ideas may not fit all situations they are just some strategies that I have seen work really well in the teams I have been a part of that have worked well together.

Workload

The most dangerous thing for a team is to have people taking on too much work at once. The person in charge of the team – be it a team lead, manager, directory, senior, etc – should be aware of how much people are taking on. Additionally, the rest of the team should make sure that one person isn’t taking on too much. If you see someone else on your team taking on too much grab something from them and help out. A sign that a team working well is when people are helping out and moving workload around as needed, without management intervention. Since the person in charge has a better idea of everything that is going on and how much work people are putting in, they should also keep an eye on the big picture and move work around as needed as well if people aren’t doing it themselves.

Trust them to GTD

After people taking too much work for themselves, the second biggest killer of productivity is constantly having to update someone on how things are going with Project X. Having someone bug you every day doesn’t help you get things done all it does is distract you from actually doing these things. Now, this doesn’t mean that you things should never be checked up on but there should be a set schedule of when everyone checks in and gives status updates. These are the things weekly meetings are good for. Outside of a normal time to check up on how things are going with your team there should be very little “how’s xyz going? Have you done ABC yet?” Basically, constantly asking how things are going outside of a normal time feels like micromanagement, and also possibly like you don’t trust those people. Both of these things are very bad for moral and productivity.

Realistic Priorities

Make sure that the team has a set of realistic priorities. Very rarely are things “not that important, get to them when you can” although they always seem to come up. This just means that they will never get done – there is always something more important that comes up. Setting realistic priorities also allow for your smart guys to prioritize things correctly. Nothing sucks more than coming in first thing and getting jumped on by everyone over a task that had a low priority, but has now shut down the company. Alright, that may be an exaggeration but surprises in priority of things do not help anyone.

Communicate Decisions Permanently

When big(ish) decisions are made, don’t leave it up to ephemeral communication methods – IM, group chat, voice – someone should write them down and send an email to the team with a descriptive title. That way they can ignore it if they want or don’t really care about that piece, but you can always easily go back and do a quick email search to find out why something was done the way it was. This should not be relegated to one person always but to the person actively working on that project. At a minimum these should contain what the decision was, why it was made that way, and who was involved. This helps prevent those “Why did we do this this way, lets change it” and the loops of the same questions coming up over and over annoying everyone.

Listen

Not only should the management of the team be listening and understanding what is going on, but the team should be listening to each other. If there are complaints or points of concern they should be acted on, not just swept under the rug, or shot down out of hand. Remember the team is full of smart people, they don’t have concerns for no reasons. Sometimes they may be misguided or the person has missed something, but that is OK we are human we miss things. To constantly have your concerns disregarded or even worse ignored is not only detrimental to productivity but is down right demoralizing.

Limit Surprises

Everyone on the team should be working to limit the amount of surprises to the rest of the team. You don’t want someone to be surprised by a new setup, that then effects what they are working on. This really boils down to having good communication on the team. Everyone should make sure that as often as they can, other people on the team are not surprised by what they are working on.

Conclusion

Basically, what this really boils down to is there needs to be trust in your team from above, and trust inside of your team between members. Really these are the same tenants that make EVERY team work, not just sysadmin teams. If you want to get the best out of your smart guys, let them do their thing and help them when you see them struggling, listen to their complaints and act on them.

Written by George Beech

June 4th, 2012 at 10:00 am

Levels of Interruption – How Important Is What You are Asking me?

without comments

Communication is an important thing in any team environment, and part of communication is realizing how your chose form of contact interrupts the other people on your team.

Generally we get interrupted multiple times a day via our various means of communication. Generally we are available via: Phone (both land line and cell), IM, Group Chat (IRC/MSN/Skype/etc or some web based chat system), and email. We will get incomming messages via all of these mediums in a day, and need to prioritize, respond (or ignore) and action each one of these requests. Each different way of getting in touch with me brings along an implicit level of importance as well as a speed of action type identifier.

Phone (and text messaging)

What do I think when you call me – or conversely I call you? To me it means that whatever you are calling me about is something that needs immediate action, generally it’s a show stopping problem, or a request for information that is blocking you from working on your assigned task at this time. Phones are the most immediate means of communication, and necessitate that I immediately switch to working on whatever the call is about. Now, if you abuse the phone I will stop picking up for you, and make you leave me a message – or tell you to put in a help desk ticket. You should only be calling me with critical things, that need attention right now.

Instant Messaging and Group Chat

This is slightly less interrupting, Instant Messaging is closer to a phone conversation – it tends to indicate that you need something now, but not right now. I can take my time to respond, but generally it should be within 0-10 minutes. The person I’m responding to has real need of the information, or there is an ongoing conversation happening.

Email

First, I know a lot of people don’t like email. Personally I think it is a great async communication method. It is also the method that conveys the least amount of importance – and yes, using the damned “Important!” flag doesn’t make it more important, if it’s that important use one of the other methods. It’s a great tool for keeping a team informed on what is going on, asking for ideas and feedback, and other things that don’t need immediate attention. Email doesn’t interrupt workflow, and allows you to take time to get back to someone.

When I’m talking to someone I decide what level of interruption I need to keep working well and use that method, and at the same time I judge how important an issue is by the way you contact me. It’s a great way to easily prioritize things, now if only people would get on board with my system!

Written by George Beech

March 22nd, 2012 at 10:23 pm

Posted in Thoughts and Rants

Where are all the SysAdmins?

with 4 comments

I had a great – albeit brief – conversation with Matt Simmons while he was in town on his world tour in NYC recently that got me thinking about bringing the SysAdmin community together. This isn’t the first time that I’ve thought about how bring the SysAdmin community together in a stronger way. And, since I’ve joined Stack Exchange I’ve thought about it more and more since community building is a big part of what we do here.

The working theories are that there are a large portion of our community in the small business sector – places like regional ISPs that have a large need for administrators, but aren’t that big in the grand scheme of things. But it _seems_ that a good deal of system administrators aren’t involved in the community at all – or at least not in a visible way. There are a few questions that come to mind when you start to think about how to grow and solidify a community:

* How do I reach these people
* How do I get them involved

The problem I’m having now, is simply I don’t really have any ideas on how to accomplish these goals, how to reach out and bring people into the wonderful community that is being built.

I shall keep mulling this over, but if you have any ideas – leave them in the comments below.

Written by George Beech

February 14th, 2012 at 5:47 pm