Coming to the office, today morning I was struck by the picture of one of my colleagues with couple of managers surrounding him. Looking at their faces, it was clear to me, that they stayed up whole night in the office. They were not fighting any fire, just finishing the release (sadly we do releases in the nights in my current work). The release was one that we do yearly, that took the whole night last year as well, and the year before and…
Last year I was driving the release and together with other guys we came to some ideas about things that could have been automated or prepared earlier to save us time and let us all have some sleep and avoid morning debugging (which is no better then debugging while drunk), and most important – shorten the time when site was under maintenance.
But we didn’t get support from managers to spend day or two to automate those little things. Let’s say I understand managers’ point of view. They are chasing due dates, while working with limited resources.
But why don’t they want to go to sleep and leave the job to engineers they trust?
Some other day while I was headed home, on my way between my desk and the office exit, passing near QA desks I was caught into discussing some issue. I have cleared that with them that the issue wasn’t urgent, it was actually there for some time already, and could definitely wait until next day.
A minute after, I was about to leave, my manager stopped while passing by as he noticed the crowd.
He stopped, even though he wanted to catch his train. He stopped not knowing what we were discussing, just out of generic “what’s up” need to know. As we told him the story, I’ve seen his face turning from tired to energized, excited. He would take off his coat and happily assist the investigation and/or fire fighting.
But why didn’t he prefer to leave that with his trusted engineers and go home?
I have actually seen that happening many times. Here’s a meme from devopsreactions tumblr that reminds me of that moments when I’m firefighting some issue while my manager is assisting me.
http://devopsreactions.tumblr.com/post/71190963508/senior-vs-junior-sysadmin-during-an-outage
My experience makes me feel, it’s not at all about seniority.
So what is it?
What makes some engineers energized when things are falling apart, excited with firefighting, more than with building resilient, self healing, automated systems? Why is it the opposite for me and many other engineers?
In my previous work, we used to joke that we don’t sleep, while we hold dresser from falling (context: there was that ridiculous article, that become viral in one Polish tabloid about a guy who claimed that he actually was holding his dresser from falling when trucks passing nearby shook his house)
Our dresser was of course the website. And it was funny because it really wasn’t. We hated those moments when we didn’t have enough time to focus on improving reliability because we were busy firefighting emergencies.
What makes some engineers energized busy holding the dresser from falling, while others get their energy from productive work do on stabilizing the stack to leave the office early and have good sleep?
Or why some managers trust their engineers and leave releasing code to them and some attend releases and hover over every move?
This is the kind of a question and the kind of a discussion that go insanely well with beer. Very thought-provoking, yet the outcome is always the same: twelve beers downed, the question still unanswered 🙂
I do not believe in superstars and prime donne. They do not make teams. As a matter of fact, a team is only as strong as its weakest member. When it comes to assembling a team, the quality of the weakest members is the most important thing. It is the weakest who focus on keeping the things going forward. It is the strongest who focus on keeping the things improving. It is not working hard versus working smart. Knowledge must be applied in order to create space for further learning.
The whole hiring thing seems superstar-oriented. We advertise ourselves as creative problem-solving enthusiasts and seek for others who do so. We talk about dynamic environment and instant results. A lot. Too much. To the point that it sounds cheap. Like a standard disclaimer or a license nobody ever reads. Dynamic environments are not that dynamic. And it is not because my heart beats four times a day. Or six, if I am on a triple espresso. It is because most of the dynamics is plain chaos. Where does the chaos come from? When talking about instant results, we skip one important question: how long will they last. When talking about experiments and creativity, we never discuss how to convert them into best practices and common knowledge. When it comes to knowledge, we only talk about having and applying it. We never ask how to abstract it, solidify and share. The hiring mentality in the industry is very much focused on what an employee brings to the organization. Perhaps we should focus on how much of it will stay when the employee is gone?
I’ve had a chance to discuss pretty much the same problem, with two of my former team mates. One of them was nostalgic about the times when work hooked him up so much that he forgot to eat and drink. It happened to me as well, and it’s still happening every now and then. But, hey, it’s not when SEV-1-ish / SEV-2 issue hits the fan when I get into this state. Especially not after 7h50min of work. Especially not when we’re dealing with XX century problems, that I was crying for fixing many times before to avoid the issue. Especially when runbook requires me to execute actions that will surely not prevent the issue from coming back. It depletes my willpower and makes me hungry and craving coffee, or maybe beer, or twelve beers 😉
I understand the point one of my colleagues made that what energized him was trying to find the the missing bits of the puzzle, approaching it from all sort of angles, trying to connect the pieces. It energizes me as well. Except for the case, when if I come up with a solution after the puzzle is solved, there is no chance of implementing it straight away. Imagine figuring out that changing one line in apache config fixes the problem, after hours of debugging, being are on fire to implement it straight away, being asked for adding a cronjob to restart apache instead, and going through 3-4 weeks long process to implement the actual oneline fix.