Brokenhaze

The thoughts of a SysAdmin

Archive for the ‘Thoughts and Rants’ Category

Getting the most out of your Sysadmin team

without comments

Disclaimer: Most of my career i’ve been what you would call and “Individual Contributor” these are not observations from a managerial perspective, but from a team member perspective

In the past 10 years or so I’ve worked under pretty much all your basic boss types. I’ve seen how people react to different stimuli and managerial styles. ┬áJust like with our programmer cousins we sysadmins can be a peculiar type of person, who isn’t easily managed in the “traditional” style that is still pervasive though out the business world. We are the type of people that are really good at analyzing a problem, and thinking of – sometime very creative – ways to get around the problem.

When I look back at all of the situations that I have been the most productive in, and have seen others be the most productive in one theme keeps coming up over and over again. That is, let your smart people think and have free reign while NOT overloading them. The latter part is really important, there is a tendency to find one or two really good people, and put a supporting cast around them, thinking that the supporting cast will help out with some of the lower level things. While this seems like a great idea, in practice it never really turns out this way. You end up with only the most minor of duties being taken care of, and you rockstar having to go back and still do a lot of the low level things that take away from what you want them working on. Or, even worse, they have to go back and FIX problems caused by someone who doesn’t know thier limits.

Ideally you want a team of sysadmins that compliment each other, so they can support each other when needed but don’t step all over the other people’s toes. This doesn’t mean you need a team of all star generalists it means you need a team of people that are good at what they do, know their limits and most importantly want to grow as a professional not just stick to the same old things that they know.

So, once you have assembled this team, the question now becomes how to I get the most out of them? How do I motivate them into doing the best that they can do and help them when they are struggling? These are not easy answers at all, and my ideas may not fit all situations they are just some strategies that I have seen work really well in the teams I have been a part of that have worked well together.

Workload

The most dangerous thing for a team is to have people taking on too much work at once. The person in charge of the team – be it a team lead, manager, directory, senior, etc – should be aware of how much people are taking on. Additionally, the rest of the team should make sure that one person isn’t taking on too much. If you see someone else on your team taking on too much grab something from them and help out. A sign that a team working well is when people are helping out and moving workload around as needed, without management intervention. Since the person in charge has a better idea of everything that is going on and how much work people are putting in, they should also keep an eye on the big picture and move work around as needed as well if people aren’t doing it themselves.

Trust them to GTD

After people taking too much work for themselves, the second biggest killer of productivity is constantly having to update someone on how things are going with Project X. Having someone bug you every day doesn’t help you get things done all it does is distract you from actually doing these things. Now, this doesn’t mean that you things should never be checked up on but there should be a set schedule of when everyone checks in and gives status updates. These are the things weekly meetings are good for. Outside of a normal time to check up on how things are going with your team there should be very little “how’s xyz going? Have you done ABC yet?” Basically, constantly asking how things are going outside of a normal time feels like micromanagement, and also possibly like you don’t trust those people. Both of these things are very bad for moral and productivity.

Realistic Priorities

Make sure that the team has a set of realistic priorities. Very rarely are things “not that important, get to them when you can” although they always seem to come up. This just means that they will never get done – there is always something more important that comes up. Setting realistic priorities also allow for your smart guys to prioritize things correctly. Nothing sucks more than coming in first thing and getting jumped on by everyone over a task that had a low priority, but has now shut down the company. Alright, that may be an exaggeration but surprises in priority of things do not help anyone.

Communicate Decisions Permanently

When big(ish) decisions are made, don’t leave it up to ephemeral communication methods – IM, group chat, voice – someone should write them down and send an email to the team with a descriptive title. That way they can ignore it if they want or don’t really care about that piece, but you can always easily go back and do a quick email search to find out why something was done the way it was. This should not be relegated to one person always but to the person actively working on that project. At a minimum these should contain what the decision was, why it was made that way, and who was involved. This helps prevent those “Why did we do this this way, lets change it” and the loops of the same questions coming up over and over annoying everyone.

Listen

Not only should the management of the team be listening and understanding what is going on, but the team should be listening to each other. If there are complaints or points of concern they should be acted on, not just swept under the rug, or shot down out of hand. Remember the team is full of smart people, they don’t have concerns for no reasons. Sometimes they may be misguided or the person has missed something, but that is OK we are human we miss things. To constantly have your concerns disregarded or even worse ignored is not only detrimental to productivity but is down right demoralizing.

Limit Surprises

Everyone on the team should be working to limit the amount of surprises to the rest of the team. You don’t want someone to be surprised by a new setup, that then effects what they are working on. This really boils down to having good communication on the team. Everyone should make sure that as often as they can, other people on the team are not surprised by what they are working on.

Conclusion

Basically, what this really boils down to is there needs to be trust in your team from above, and trust inside of your team between members. Really these are the same tenants that make EVERY team work, not just sysadmin teams. If you want to get the best out of your smart guys, let them do their thing and help them when you see them struggling, listen to their complaints and act on them.

Written by George Beech

June 4th, 2012 at 10:00 am

Levels of Interruption – How Important Is What You are Asking me?

without comments

Communication is an important thing in any team environment, and part of communication is realizing how your chose form of contact interrupts the other people on your team.

Generally we get interrupted multiple times a day via our various means of communication. Generally we are available via: Phone (both land line and cell), IM, Group Chat (IRC/MSN/Skype/etc or some web based chat system), and email. We will get incomming messages via all of these mediums in a day, and need to prioritize, respond (or ignore) and action each one of these requests. Each different way of getting in touch with me brings along an implicit level of importance as well as a speed of action type identifier.

Phone (and text messaging)

What do I think when you call me – or conversely I call you? To me it means that whatever you are calling me about is something that needs immediate action, generally it’s a show stopping problem, or a request for information that is blocking you from working on your assigned task at this time. Phones are the most immediate means of communication, and necessitate that I immediately switch to working on whatever the call is about. Now, if you abuse the phone I will stop picking up for you, and make you leave me a message – or tell you to put in a help desk ticket. You should only be calling me with critical things, that need attention right now.

Instant Messaging and Group Chat

This is slightly less interrupting, Instant Messaging is closer to a phone conversation – it tends to indicate that you need something now, but not right now. I can take my time to respond, but generally it should be within 0-10 minutes. The person I’m responding to has real need of the information, or there is an ongoing conversation happening.

Email

First, I know a lot of people don’t like email. Personally I think it is a great async communication method. It is also the method that conveys the least amount of importance – and yes, using the damned “Important!” flag doesn’t make it more important, if it’s that important use one of the other methods. It’s a great tool for keeping a team informed on what is going on, asking for ideas and feedback, and other things that don’t need immediate attention. Email doesn’t interrupt workflow, and allows you to take time to get back to someone.

When I’m talking to someone I decide what level of interruption I need to keep working well and use that method, and at the same time I judge how important an issue is by the way you contact me. It’s a great way to easily prioritize things, now if only people would get on board with my system!

Written by George Beech

March 22nd, 2012 at 10:23 pm

Posted in Thoughts and Rants

Where are all the SysAdmins?

with 4 comments

I had a great – albeit brief – conversation with Matt Simmons while he was in town on his world tour in NYC recently that got me thinking about bringing the SysAdmin community together. This isn’t the first time that I’ve thought about how bring the SysAdmin community together in a stronger way. And, since I’ve joined Stack Exchange I’ve thought about it more and more since community building is a big part of what we do here.

The working theories are that there are a large portion of our community in the small business sector – places like regional ISPs that have a large need for administrators, but aren’t that big in the grand scheme of things. But it _seems_ that a good deal of system administrators aren’t involved in the community at all – or at least not in a visible way. There are a few questions that come to mind when you start to think about how to grow and solidify a community:

* How do I reach these people
* How do I get them involved

The problem I’m having now, is simply I don’t really have any ideas on how to accomplish these goals, how to reach out and bring people into the wonderful community that is being built.

I shall keep mulling this over, but if you have any ideas – leave them in the comments below.

Written by George Beech

February 14th, 2012 at 5:47 pm

*Shudder* DevOps

with 4 comments

I think the first time I heard the term DevOps I didn’t think all that much about it, in fact I vaguely remember thinking “oh great another group of developers that think they don’t need sysadmins and they can just code themselves out of any systems problem.” Of course, at the time I was working in a soul crushing job, with incompetent programmers who thought their job was to push out code and as long as it ran on their laptop everything was just fine. So, I may have had a slightly … jaded view of things. At that point I just put my head back down and got back to work.

It’s been about two years now since I first heard the term DevOps, and you know what – i still don’t like it. However, now I don’t like it for completely new reasons.

The first, and I think biggest problem I have with the term DevOps is simply this – it shouldn’t exist. Simply, what people are calling DevOps should be shortened to “SysAdmin.” That’s right every SysAdmin should be working this way – there shouldn’t need to be a new term. Every SysAdmin should have a basic set of skills, a common ground we are first and foremost IT workers – that means we craft raw computing power into usable and complex systems. Those systems are not built by hand after the first time. They are built by automation, automation lets you not have to worry about the details of a solved problem. Automation lets you know that your complex system will be built correctly the second, third, Nth time.

In my opinion every sysadmin should at a minimum say yes to all of these things:

  • You should be able to script in at least two languages
  • You should have a passable command of one compiled language
  • You should be able to look at a piece of code in any language and have an understanding of basically what is going on
  • Why? So you can talk with your devs, that’s why

There are many people that will wave their hands, and shout “But, But that’s not what DevOps is about – DevOps is about bringing your Developers and SysAdmins closer and getting them to work together for a common good.” Ok, that’s a fair point and brings me to the second problem I have with DevOps. That problem is that the real word you are looking for is TEAMWORK. You shouldn’t need to coin a new term that says the IT department should work together – that should already be the goal.

My boss at my last job but it very well:

The guys in charge don’t care about how things get done. They only care that they do get done. All they see when the Dev and SysAdmin teams argue about anything is “The Geeks are fighting again – I don’t know about what and I don’t care, they just need to figure it out and get it done.”

Everyone outside of IT sees us as a collective but, we still bicker between each other like children a lot. We need to start seeing everyone as part of the same team. We should take the ideals of the DevOps movement and repackage them as how things are done every day in the IT department. No need for special labels, no need to make a huge fuss about it. We just need to drop the label and get to work.

For any dev that reads this and goes “what about me” you can just s/SysAdmin/Dev/g and it still applies – for the most part. I’m a SysAdmin so some things may be slanted that way.

Written by George Beech

February 6th, 2012 at 10:32 pm

There is No Such Thing as Too Complex

without comments

This is a crazy idea, but it’s true there really is no such thing as too complex in an IT environment. If you take a moment to think about it, everything we do is incredibly complex. Really, just take a moment to think about it. There is not one thing that you do as a Systems Administrator that isn’t complex.

You still don’t believe me do you? Fine. Lets take a look at a very simple operation. We are going to monitor a single box via SNMP. We are going to assume that you already have a central monitoring box already setup.

  1. You configure your box-to-be-monitored with SNMP, configuring the proper access controls
  2. You add any extra scripts that you need to call via SNMP
  3. You open the firewall on the box to allow SNMP traffic
  4. You configure your monitoring server to query the box-to-b-monitored via SNMP
  5. You check the results

And, THAT is the simplified version of events. In reality there is a lot more that goes into just the simple process of monitoring ONE machine. That really is complex, and it’s not a bad thing.

Now having said this:

There is such a thing as BAD complexity.

Bad complexity is complexity for the sake of making something more complex, or the inverse: making something less complex for the sake of being not complex. What you want to do is create a system that is of the correct complexity. That is you want to create a system that is neither too simple, nor too complex.

The problem that you run into with either end of the complexity spectrum is this: either you have a system that is too basic and cannot be scaled properly as the need arises or you have a system that is unmaintainable, and cannot scale because there are too many pieces meshed together.

When you are designing a system, you really will never achieve the perfect amount of complexity. There will always be trade-offs you have to make, based on past design decisions, future configuration considerations, and application constraints. However what you can do is make every decision in a thoughtful way that tries to strike a balance between the two extremes of complexity. And that, is one of the true zen things when you are a sysadmin. You achieve this beautify nearly perfect level of complexity system, and you sit back and smile. Then you go run to put out the next fire.

Written by George Beech

August 12th, 2011 at 4:07 pm