The thoughts of a SysAdmin

Archive for the ‘Teamwork’ Category

Change Control using Git(lab)

without comments

Pre-amble and thoughts on Change Control

If you just want the cool details you can skip to the good stuff

I am a strong believer in change control. It allows for many good things to be done with a well run IT organization. The top three things that come to mind are accountability, reliability and review-ability (I think i’m making that last word up).

There are all good things to have. Many people have come before me to praise change control and that is not what I want this post to be about. I want to talk about the change control process that we are starting to use at Stack Exchange. A process that I believe addresses some of the most common complaints and push back on implementing a change control system I hear.

A good place to start would be to lay out exactly what those common complaints are.

But, Change Control is too complicated!

I hear this a lot, and this particular complaint tends to break down into two different categories. The first is that there is so much process that you can’t get any actual WORK done. The second major complaint in the category is that it slows everything down, making it harder to get awesome things done.

I’m not going to argue with either of the characterizations. In fact the express purpose of a change control system is to put a speed bump in the way. To make you slow down a bit and think through the implications of what you are doing, to have someone else double check your work. We are not infallible, we make mistakes and that is ok. The goal here is to minimize the number of mistakes that make it into production and to minimize the times that they get there at all.

It’s just bureaucracy, busy work, an annoyance

I’ve met a good number of people that have this attitude. They want to be able to just log in as the administrative user and change whatever they want, whenever they want. Personally, I have given up on trying to convince these people that change control is a good thing.

Design a process that is a speed bump

I spent a good deal of time trying to come up with a better change control system. The system should be easy to use, low impact, and accessible to anyone on the team. One of my over-arching design goals is to simply create a speed bump not a road block.

I spent a good deal of time thinking about what tools we use day to day. Looking at the ones that people complained about, the ones people liked, and the ones nobody said anything about. After thinking for a while I realized that the one tool that everyone used, and there were few complaints about was our DVCS (recently moved to git). It’s just about a perfect fit for a light weight change control system.

The workflow

You don’t need anything special to get up and running with version control as the back to your change control system. The glue that brings everything together is a simple python script that does most of the heavy lifting for you. The script – called – takes the name of the system, and the risk of change and from that creates a new merge request from a template. The merge request is pretty light weight. We want all the detail to be in the actual commits and commit messages.

The basic workflow we use is to create a branch, then create a merge request based on that branch, have the change reviewed and then finally have the review merge the branch into master once the change is complete.

That’s it. One long sentence to define out entire change control workflow.

This is a very simple workflow that accomplishes all of the goals I have for a change control system. To make life even easier for people I wrote a small python script that creates the branch, copies some templates into place for you and then creates the merge request. All you have to do at this point is fill out the details and get someone to review it.

The future

This is just v1 of the our change control system. In the future we will be adding web hooks to automatically send out notification, add calendar entries. Basically the goal is to automate all of the boring manual stuff as much as we can.

You can get the code we use on GitHub

Written by George Beech

February 10th, 2016 at 5:50 pm

Getting the most out of your Sysadmin team

without comments

Disclaimer: Most of my career i’ve been what you would call and “Individual Contributor” these are not observations from a managerial perspective, but from a team member perspective

In the past 10 years or so I’ve worked under pretty much all your basic boss types. I’ve seen how people react to different stimuli and managerial styles. ┬áJust like with our programmer cousins we sysadmins can be a peculiar type of person, who isn’t easily managed in the “traditional” style that is still pervasive though out the business world. We are the type of people that are really good at analyzing a problem, and thinking of – sometime very creative – ways to get around the problem.

When I look back at all of the situations that I have been the most productive in, and have seen others be the most productive in one theme keeps coming up over and over again. That is, let your smart people think and have free reign while NOT overloading them. The latter part is really important, there is a tendency to find one or two really good people, and put a supporting cast around them, thinking that the supporting cast will help out with some of the lower level things. While this seems like a great idea, in practice it never really turns out this way. You end up with only the most minor of duties being taken care of, and you rockstar having to go back and still do a lot of the low level things that take away from what you want them working on. Or, even worse, they have to go back and FIX problems caused by someone who doesn’t know thier limits.

Ideally you want a team of sysadmins that compliment each other, so they can support each other when needed but don’t step all over the other people’s toes. This doesn’t mean you need a team of all star generalists it means you need a team of people that are good at what they do, know their limits and most importantly want to grow as a professional not just stick to the same old things that they know.

So, once you have assembled this team, the question now becomes how to I get the most out of them? How do I motivate them into doing the best that they can do and help them when they are struggling? These are not easy answers at all, and my ideas may not fit all situations they are just some strategies that I have seen work really well in the teams I have been a part of that have worked well together.


The most dangerous thing for a team is to have people taking on too much work at once. The person in charge of the team – be it a team lead, manager, directory, senior, etc – should be aware of how much people are taking on. Additionally, the rest of the team should make sure that one person isn’t taking on too much. If you see someone else on your team taking on too much grab something from them and help out. A sign that a team working well is when people are helping out and moving workload around as needed, without management intervention. Since the person in charge has a better idea of everything that is going on and how much work people are putting in, they should also keep an eye on the big picture and move work around as needed as well if people aren’t doing it themselves.

Trust them to GTD

After people taking too much work for themselves, the second biggest killer of productivity is constantly having to update someone on how things are going with Project X. Having someone bug you every day doesn’t help you get things done all it does is distract you from actually doing these things. Now, this doesn’t mean that you things should never be checked up on but there should be a set schedule of when everyone checks in and gives status updates. These are the things weekly meetings are good for. Outside of a normal time to check up on how things are going with your team there should be very little “how’s xyz going? Have you done ABC yet?” Basically, constantly asking how things are going outside of a normal time feels like micromanagement, and also possibly like you don’t trust those people. Both of these things are very bad for moral and productivity.

Realistic Priorities

Make sure that the team has a set of realistic priorities. Very rarely are things “not that important, get to them when you can” although they always seem to come up. This just means that they will never get done – there is always something more important that comes up. Setting realistic priorities also allow for your smart guys to prioritize things correctly. Nothing sucks more than coming in first thing and getting jumped on by everyone over a task that had a low priority, but has now shut down the company. Alright, that may be an exaggeration but surprises in priority of things do not help anyone.

Communicate Decisions Permanently

When big(ish) decisions are made, don’t leave it up to ephemeral communication methods – IM, group chat, voice – someone should write them down and send an email to the team with a descriptive title. That way they can ignore it if they want or don’t really care about that piece, but you can always easily go back and do a quick email search to find out why something was done the way it was. This should not be relegated to one person always but to the person actively working on that project. At a minimum these should contain what the decision was, why it was made that way, and who was involved. This helps prevent those “Why did we do this this way, lets change it” and the loops of the same questions coming up over and over annoying everyone.


Not only should the management of the team be listening and understanding what is going on, but the team should be listening to each other. If there are complaints or points of concern they should be acted on, not just swept under the rug, or shot down out of hand. Remember the team is full of smart people, they don’t have concerns for no reasons. Sometimes they may be misguided or the person has missed something, but that is OK we are human we miss things. To constantly have your concerns disregarded or even worse ignored is not only detrimental to productivity but is down right demoralizing.

Limit Surprises

Everyone on the team should be working to limit the amount of surprises to the rest of the team. You don’t want someone to be surprised by a new setup, that then effects what they are working on. This really boils down to having good communication on the team. Everyone should make sure that as often as they can, other people on the team are not surprised by what they are working on.


Basically, what this really boils down to is there needs to be trust in your team from above, and trust inside of your team between members. Really these are the same tenants that make EVERY team work, not just sysadmin teams. If you want to get the best out of your smart guys, let them do their thing and help them when you see them struggling, listen to their complaints and act on them.

Written by George Beech

June 4th, 2012 at 10:00 am