On May 14 2013 I’ve spoken at Atmosphere Conference 2013 in Poznan – first (to my knowledge) polish conference dedicated to web performance and scalability and DevOps culture. Yay! This year Atmosphere Conference 2014 is going to take place in Warsaw, on 19-20 May. Check it out if you haven’t yet.

Topic wise it seemed to me that they wanted it to be something similar to Velocity Conference. That’s surely direction I like. Topics in Atmosphere’s schedule were mixed from quite different worlds, so that only some talks were about performance, scalability, monitoring and such. The conference was worth going to anyway, I’ve enjoyed at least couple of talks. From the talks I remember I liked the most, I’d like to recommend watching “Monitoring at scale” from Lorenzo Alberton.

Organization was brilliant. Food was the best I ever had on event like this. There’s also interesting review of the conference on Shelly Cloud Blog.

Two things I didn’t like was that Polish speakers (with 2 or 3 exceptions) gave their talks in Polish (there were two tracks and often both was in polish – imagine you came to attend from UK or Germany and you naturally don’t speak polish) and that 2/3 of speakers were Allegro employees. I felt kinda like if I was attending Allegro meeting.

What this post is about is something I realized attending foreign speakers talks siting within mainly polish group of listeners. I’ve seen people on stage looking kinda puzzled by reaction different from what they expected. Brian McCallister for example, in his second talk, demoing Docker, surprised by almost no reaction to some simple questions he went asking (like “raise your hands who is familiar with chroot”) he tried something like “raise your hands who of you is human”. Did he saw the audience laughing? Nope, this is Poland.

So what you can expect from Polish audience. What is so different about us?

1. Poles don’t smile. Period. Or at least they don’t smile for the same reasons Americans do. Yes we know it’s not good, we’re trying to change it. So don’t be surprise polish audience don’t laugh when you make a joke. Unless the audience is drunk, but that’s another story. Or you can try looking up some hints on what makes poles laugh.

2. Poles are taught at school to sit down and listen. We are not encouraged to interact with a teacher. Conference presentation, for some, reminds the class. Asking questions is asking for trouble. Again, it starts in school. Asking question means you were not paying attention to what teacher was saying. Bad. That’s why you may be surprised by how short the Q&A part after your talk was.

3. English is not Poles mothers tongue, so it’s possible that polish audience will not follow to all of details of your talk. And we try to avoid asking questions, when it’s uncertain for us, if the question wasn’t answered before (see point 2).

4. Poles are shy (unless we’re drunk). Shy and uncertain of their language skills. This is why you should expect less interactions. Take Filip Barański’s advice and make some fun of yourself as an icebreaker.

5. The other gap you should mind is money. No, we don’t earn $100k as Paul Hammond may think Polish developers do (see his Infrastructure for Startups talk).

Thoughts on modularity while refactoring Chef cookbooks. The dilemma of reusability and complexity and maintainability and… It’s never just black and white.

Finer grain recipes are easier to maintain and to understand. Roles containing lightweight recipes provide high level overview helping to understand the system architecture. Recipes that are too lightweight, however, may provide minimal value and may require combining many recipes into single useful role.

Coarse grained recipes make it easier to keep changes in one place – you edit one file. Recipes that are too heavyweight, however, may become difficult to reuse because they provide more than needed for different use cases.

As system grows in size and complexity, it’s important that we design our automation for easy reuse and optimal change isolation.

I find pidstat useful tool in troubleshooting system performance. Let me share with you some examples of the ways I use it.

1. Checking CPU consumption per process.

This oneliner will run continuously every 1s showing output lines only for processes consuming more than 20% of CPU:

# pidstat -l 1|perl -lane 'print if @F[5] =~ /([2-9]\d|\d{3,})\./'
12:58:50         1484   44.55    5.94    0.00   50.50     1  /usr/sbin/apache2 -k start
12:58:50         2990   46.53    4.95    0.00   51.49    13  /usr/sbin/apache2 -k start
12:58:50         2999   30.69    2.97    0.00   33.66     4  /usr/sbin/apache2 -k start
12:58:50         8976    0.00   32.67    0.00   32.67     9  flush-0:21
12:58:50        11937   54.46    4.95    0.00   59.41     2  /usr/sbin/apache2 -k start

Same, but including threads:

# pidstat -lt 1|perl -lane 'print if @F[6] =~ /([2-9]\d|\d{3,})\./'

Note that the column number differs when watching threads. It may also differ depending on how your OS diplays time (AM/PM adds one column).

Watch single PID’s CPU usage:

$ pidstat -l 1 -p 5181
 
03:11:16 PM       PID    %usr %system  %guest    %CPU   CPU  Command
03:11:17 PM      5181   22.00    2.00    0.00   24.00     1  /usr/lib/firefox/firefox
03:11:18 PM      5181   23.00    2.00    0.00   25.00     1  /usr/lib/firefox/firefox

2. What process in making most context switches.

I graph context switches in Ganglia (with sflow) and I once saw this kind of graph:

context switches graph in ganglia

I found out that pidstat could tell me what process was making these spikes. Let’s look for processes making more than 100 non voluntary context switches per second:

# pidstat -wl 1|perl -lane 'print if @F[3] =~ /\d{3,}\./'
 
13:18:40        32579   4408.00   1262.00  /usr/bin/plackup
13:18:40        32588    177.00    134.00  /usr/bin/plackup

3. Checking what process is using disk the most.

If you wanted to know which processes are writing more than 100 kB/s to disk:

# pidstat -dl 1|perl -lane 'print if @F[3] =~ /\d{3,}\./'
13:24:40          382      0.00    172.00      0.00  jbd2/sda3-8
13:24:48         1406      0.00    160.00    160.00  /usr/bin/perl /etc/rc2.d/S20varnishgmetric start
13:24:54         1981      0.00    212.00      0.00  /usr/bin/plackup
13:24:54        24520     56.00   1912.00   1080.00  /usr/sbin/apache2 -k start

Bonus note on KVM and iostat: On KVM host if you run iostat -dx 1 it will show you I/O consumption per drive. Now, how to tell which of dm-N devices belong to what VM? Let’s say the most I/O heavy drive was dm-15. Here’s how:

# dmsetup ls|grep 15
guests-vm_i1_root    (252, 15)

4. Looking at RAM hungry processes second by second.

This oneliner will show processes that hold more than 200kB of their RSS in RAM.

# pidstat -rl 1|perl -lane 'print if  @F[5] =~ /([2-9]\d{5,}|\d{7,})/'

The way attributes get merged during the chef client run may appear troublesome in a situation when you need to overwrite an array of attributes.

Imagine you want to store an array of, say, memcache pool members in attributes. You’ve chosen to do it with array, so it looks like this (attributes file notation):

default['fooapp']['memcache'] = [ "memcache-a1:11211", "memcache-a2:11211" ]

Now you want to configure different memcache pools, for different datacenters. You want to overwrite default attributes from attributes file, you also want to be able to have different number of pool members (array elements) in each datacenter. Let’s try with a roles specific to each datacenter, like this:

"fooapp": {
    "memcache": [ "memcache-b1:11211", "memcache-b2:11211", "memcache-b3:11211" ]
}

What you’ll getting on a node in datacenter “b” is:

$knife node show somenode-b1 -a fooapp
fooapp:
  memcache:
    memcache-a1:11211
    memcache-a2:11211
    memcache-b1:11211
    memcache-b2:11211
    memcache-b3:11211

Oh, of course it’s merged! You didn’t want app from colo “b” to use memcache in colo “a”!

What you could do is remove default elements using special :knockout_prefix attribute allowing for subtractive merge. Elements prefixed with this custom prefix would get removed during merge. This is however not available since Chef 11 so I’ll skip discussing it.

The idea I got is to workaround it by keeping the elements in string instead of an array, like this:

default['fooapp']['memcache'] = "memcache-a1:11211,memcache-a2:11211"

and in role:

"fooapp": {
    "memcache":  "memcache-b1:11211,memcache-b2:11211,memcache-b3:11211"
},

And then split them in a template:

<% node[:fooapp][:memcache].split(",").each do |m| %>

It’s less elegant than using array but it works.

I love the way they bring fun to tech events. Especially if it has something to do with music. Last years DevOpsDays in Mountain View, it started with screening AC/DC’s “Let There Be Rock” video with vocals replaced to some custom lyrics with “Let there be DevOps” as a chorus. Music was leitmotif of the whole event. Crew wore Blues Brothers costumes. Later on, in the evening, there was jam session as a part of the event.

Great postscript to that is David Lutz singing “Packet Loss Blues” in his car. 🙂

Yeah, actually why not sing out your tech job worries?

Another frustrated admin helping himself singing it out – “Kill Dash Nine” by Monzy:

I have recently visited fun sysadmin quiz intended as a hiring tool. The toy was so popular that they’ve started second part just for fun. Cool! I hope they are not serious about hiring people this way though. This reminded me of some questions from Google SRE hiring process, like “What is the difference between hub and switch”. Good if you are looking for IT archeologist maybe. Another interesting approach to recruiting I remember is Matt Biddulph’s idea of Algorithmic recruitment with GitHub which I think is a masterpiece but still, no more than interesting code experiment or source of contacts for head hunters.

Both tricks mentioned above may help you find some skilled and/or experienced engineers. Will they help you hunting top talents? Will they tell you anything of the candidate personality? You still need a way to filter out an asshole from joining your team, right? Here’s why.

So use toys such as quiz or algorithmic searches for fun and as a additional source for your candidates base. Then, add following rules to your current hiring process methodology:

1. Don’t test technical skills in academic like tests. At least don’t rely on that tests. Try rather to challenge ones intellect.

2. Don’t rely too much on ones public activity (GitHub, Twitter, StackOverload etc.) of which I wrote recently. Imagine there are people who are truly engaged at work, not doing anything external to their work (you don’t want to hire person writing private blogs and developing their own apps when at office, instead of working, do you?). So should they blog, twit, code in their free time. Right. Except they may have different reasons to be detached from the computer, when home, for example they may have kids, or hobby, whatever.

There may be hidden unicorns!

3. Trust you gut feeling. If possible let the candidate be interviewed only be people entirely convinced to your team values, ones that contribute good to the workplace atmosphere, ones that can smell the unicorn. Attract top talents by letting them meet cool people from your company.

If you think you may not be able to smell the unicorn, invite someone, who you think is, to the help you with the interview process.

4. Value person’s attitude and passion. Some google’s managers say humility is what is essential,  “For one startup ascertaining humility is so important, it is the first filter in the interview process” says Tomasz Tunguz.

5. I like the tip of Rajat Taneja, to Hire for UNCOMMON strength. He says “Focusing on finding candidates who have the least amount of weaknesses will lead to mediocrity. Hire for uncommon strength and then put the person in a role that uses that muscle.”

Until recently I was considering writing blogs a waste of engineers time. But then I realized that because I’m benefiting from reading other peoples blogs, writing my tech blog is then, in a way, paying back to the community, same way like contributing to open source projects is.

One disagreement about blogging I had was: why all authors writing on the same subject don’t join their forces and contribute to the wiki on the subject? Well, apparently recognition is a human basic need. I just needed to grow up to accept that fact.