Senior Developer Assessment: Re-aligning Expectations
Today we assessed our 20th candidate for position of senior developer, using a small test that I drafted some time ago. Since then I’ve spoken about how the test was dumbed down since one of the questions was perceived as being too difficult. The assessment consists of three questions, which I’ve dubbed Fibonacci, Quicksort and University. I’ve already discussed two of the questions, namely Quicksort and University.
Although not an earth-shattering sample, 20 is a number that we can start to draw pretty graphs with, so I’ve included a summary of the results alongside.
We can speculate about what it tells us:
- There’s something wrong with University; nobody has come up with a satisfactory answer;
- In the case of Fibonacci, there’s something wrong with the candidates; only five people have come up with a satisfactory answer.
Ok, my second conclusion above is possibly a little harsh. I think it’s a little harsh, because upon inspection by others, it’s elicited use of some power words like “mathlete” and “math-wiz”.
Now Fibonacci is the only question that I haven’t posted for all to see, mostly because I thought it was too simplistic and therefore quite uninteresting. Given these results though, I’ve been prompted to reword it completely. I mean, if only math-wiz’s have what it takes to do it then I can’t make any assumptions about the background that a candidate may have in the arithmeticmaths department.
What follows is the version of Fibonacci as it was up until now:
I’ve taken great pains to remove anything that may constitute an implicit (read “unfair”) assumption about mathematical background and I’ve reworded the question to provide a lot more hand-holding for the candidate. This should also help in offsetting the nervousness factor.
Here’s my updated version:
Your “updated version” is hilarious :-). I really hope that was just a parody because I can’t imagine there are really people that call themselves computer programmers that need such detailed explanation of Fib to be able to print first 30 terms of it…..
Ivan – dunno, thinking about using the updated version for real – do you think candidates would find it insulting?
So far the only one that could feel insulted is you, insulted by their answers.
What a scream! Thank you for a very entertaining post.
Hi Eric!
First a disclaimer – I am writing this after 2 beers so please do not take this too serious and/or personal :-).
First – I am reading your blog for last 2 months and I just can’t be bothered to seek through archive to possibly find out which company you recruit for (if that is mentioned at all) so this answer could be totally off. The problem you are facing about finding appropriate senior developer seems it is not in questions itself because they are pretty clear in their prime form (except University – that one could have been defined a little better) but I see this as a problem attracting right candidates. A while ago I wanted to compete for job in Google or Amazon so I went through a lot of “My interview at…” web blogs which showed me the level of knowledge required to pass recruitment process and I have to tell you that all of them have a lot harder problems for candidate to solve than ones you posted.
Since I wanted to compete for mentioned jobs I went through algorithms and data structures books, object oriented design etc and after that I can tell you that I was able to solve all 3 presented problems in less than 45 minutes even though I would never think of myself as a senior developer. Well to be fair – I don’t work (nor have professional experience) as software developer at all. I have a lot of background in networking and VoIP – I just do programming as a hobby and lately thinking of going SW developer route professionally if I find a first developer job :-).
So, in a nutshell – I think you need to work on how to attract quality people by your company – not lower the standards of recruitment questions.
Regards,
Ivan
P.S. If you have open SW developer position (not senior) we can talk… :P.
I think a candidate that has been approved on this process is not necessarily a senior developer. It all comes down to what you expect him/her to do daily.
Is he going to spend his day to day routine writing algorithms to solve math equations?
Or is he going to spend his day writing rock solid object-oriented systems – or concurrent systems or…?
No doubt you need to have a good understanding of algorithms to do the latter, which is well assessed in your interview questions.
But those fail to address what “rock solid object-oriented systems et. al.” is. For instance, don’t you think it would be useful to have your candidate design and implement a full – yet small – project like the “Mars rovers”? That way, you’ll also be able to assess his object-oriented skills and TDD experience – assuming these are important for the position you’re hiring for.
Of course if you’re hiring someone to then provide training, it’s a totally different story and your questions would work really well. But then, I don’t think that would be the definition of a senior developer. ;)
Nevertheless, great post!
Hi Eric,
Your test seems very “domain specific”. Are you looking for a mathematician? Are you hiring for a maths project?
I’m not sure the type of work you do, but if it is the typical type of web application, I would suggest that your test is unrepresentative of the type of work your candidates will end up doing on the job.
I like maths, and my degree is in Electronics, but I have also know of potentially potentially good engineers that are intimidated by mathematical formulae. Put it this way. I would feel slightly intimidated at your interview and I have 20 years programming experience.
I don’t know, but your approach seems slightly lazy to me. Why not pair program with candidates on a “real world” problem and get to find out how they actually think? rather then whether they have the prior context needed to pull out a by rote answer to some arbitrary mathematical test?
Just a thought.
BTW. The biggest skill I see missing amongst candidates is a firm grasp of OO design. Fibonacci may tell me something about a candidates functional programming skills, but it won’t tell me anything about their grasp of OO.
Paul.
Hm… I don’t get how “Fibonacci” is a math-specific question. You don’t have to derive functions or anything, you have algorithm in front of you – you just have to implement it. It shouldn’t take more than 10 minutes even if you never heard about fibonacci before. Except that – fib is the most common programmers job interview question. Seems like Eric is trying to get multi-purpose programmers that can adjust to requirements quickly.
Hi Eric,
And just to prove I’m not a complete “an idiot” :) The answer is that the first two numbers in the sequence are special cases, and the rest (the other 28) can be generated by recursively calling a function.
If I was in an interview I’d be racking my brain to work out exactly what the function should be, on the job I’d Google it in a second :)
And if I really wanted to impress, I’d download an Haskell compiler and show off the three line Haskell solution.
Now that’s the kind of candidate that would impress me! :-)
Paul.
Very amusing… and insulting. My annoyance peaked at the definition of the plus operator with the helpful picture of apples. If I were a interview candidate that would have been my WTF moment; as in WTF is this company’s problem? WTF am I doing here / Why the fuck would I want to work here?
@Paul, @Leonardo – the test is there to filter out the sort of undesirables that Jeff Atwood talks about in “The Non-programming Programmer”. Re the OO question – have a look at “University”.
@Ivan – if you managed to do those three questions in 45 minutes then you’re in the top 1%! Just don’t know if you’re up to living and working in the southern tip of Africa ;)
Although I was briefly tempted to actually use that brazenly rude question, I wouldn’t subject anyone to it, no matter how math-challenged they might be. It was drafted and posted in a fit of frustration due to being bombarded with complete rubbish.
I think Ivan is correct. The frustration you experienced occurs when you are interviewing from an insufficiently skilled pool of developers. I have found public job ads to be a complete waste of time for this reason. These days I only consider people I know or who are referred to me.
Eric,
I took a look at University, and yes its a nice OO problem. I’ve thought a bit more about Fibonacci and I agree with Ivan, the formula is given to you, so even if the candidate doesn’t come up with the canonical soln, they should be able to come up with ‘a’ solution.
I then thought to myself, well you should be looking for potential rather then the finished article, then I thought, no you’re looking for a “senior” developer so they should know this stuff.
Sadly I tend to agree with Liam. There isn’t a lot of good people out there. No scrub that, there isn’t many competent people out there period.
After being in the game awhile, what I notice is that everyone wants “good people”, yet no one is willing to invest in said people. We can blame the candidates or we can choose to blame an industry that treats individuals in a very challenging profession like hired labourers.
You clearly know your stuff. Isn’t there anyone you could “groom” internally into the senior role?
(I’ve come across people with 10 years+ experience who just don’t grasp the basics and have picked up a load of bad habits. I’ve also mentored good graduates who were well on their way to being good developers after only a year or so out of school. Most of what I know is self taught in my ‘spare’ time. I guess my point is we as an industry tend to get out what we put in)
Paul.
Paul, I’ve had the same experiences as you have – some people seem to have a ceiling to their abilities, and others just have it all come naturally.
First prize is to “groom” internally as you put it. What I’m encountering is a disproportionately small number of “groomables”. So, the natural response is to raise the bar when it comes to hiring policy.
Hi Eric,
Understood. I’ve gone back and looked at your series of post on the theme of what is a “Senior Engineer”, and I must admit it’s got me thinking. Some of the links to posts that refer to programmers that can’t even program were a real eye opener :).
I’m curious to know whether you’re an ‘Agile” shop? There seems to be a cultural zoo out there when it comes to software and programmers. With the exception of University, I think your tests will appeal to the “macho” developer (not good, or bad, just saying). I quite liked Solver and I thought it was a good application of numerical analysis. I didn’t care much for Sorter, because it caused me to gleam back to the binary sort exercise I did at university over 20 years ago and have never used since (on an Electronics Engineering degree I hasten to add :)).
University came closest to what I would call a real world problem. I think it would appeal to the “business savvy” type. It doesn’t go deep enough into OO for my liking, and the diamond inheritence problem is a bit “C++” text bookish (I should add that I’m a Smalltalk zealot :))
I work as an Agile Coach, and there is a whole gamet of people skills that developers need to have too in order to work effectively with… well other people :) I’m sure that you are considering these also :)
Thanks for the posts. It got me thinking. One last thing. I gather that you are a C# shop. I’ve noticed that certain cultural traits tend to attach themselves to differing programming communities. I’ve been doing .NET for about a year now, and the “charlatan programmers” you speak of seem to be over represented in that community. Java guys seem generally to be better (but not by much). And if you are looking for real “OO/business” savvy, have you considered (ex) Smalltalk developers? And of course, there are the old school C++ers, many of whom are macho, but there are some good OO people amongst them also.
Picking up C# shouldn’t be an issue for a ‘Senior’ person :)
Paul.
Eric,
One last thing. Be sure to make a post when you do finally find someone worthy. I’m curious to see who you end up with. With their permission it would be nice to see the winning answers to your test :)
All the best and good luck.
Paul.
@Paul – re your questions, no we’re not an agile shop per se, although I have borrowed various agile practices that I think add value, including unit testing (not necessarily TDD though), CI, three-week iterations, burndown charts and sprint and product backlogs.
Unfortunately, IMO agile is more cultural than anything else (at least if you actually want it to work!), and when you’re confronted with a “non-grokking” set of business users it becomes an effort in mindset change as opposed to practice implementation. Needless to say, I haven’t even tried to address that quagmire, so it’s the 80/20 rule here when it comes to agile.
Re the whole C#/Java thing. I agree with you on this one; I’ve had the benefit of having developed in the Java world as well as the .NET world so I can attest to Java developers in general being of a slightly higher standard. I do, however, think that this trend is in the process of attenuating, and that’s largely as a result of a gradual culture shift—things like Alt.NET are having a positive influence here; that combined with MS’ recently more-accepting attitude towards OSS is forcing a typical .NET developer to be a polyglot when it comes to technologies (with a commensurate increase in standards).
I may ruffle some feathers in saying this, but IMO Smalltalk has always largely been the domain of academics. I have never met someone who has done any serious commercial Smalltalk development, but something tells me that they would know their stuff :) I’m only too happy to meet a developer who doesn’t think that Smalltalk is business-speak for gossip.
We’re actually struggling to find good people, as evidenced by the graph that I posted, so no doubt there will be more posts on this subject :)
Thanks for all the feedback!
Eric,
If you are in the web-apps data-base biz, I really don’t understand why you don’t have a programming problem from that domain. Constructing an SQL query (or whatever other query language) you want to use can be a very easy kick-off test and with some joins you can turn it into a more advanced logical thinking test on the spot!
This would have the benefit that the real seniors would recognize the problem domain and immediately be more at ease. Why not just take some typical coding problems from your normal work life as tests? If you want to be more demanding you can take one of the more complicated problems from work (say one which your non-groomable coworkes couldn’t solve well) and boil it down to it’s essentials to use as a test.
You can even accommodate for senior programmers who are switching fields (say, from Java to C#, or from non-web apps to web-apps) by asking them to describe some interesting coding problem from their past job and how they solved it.
Of course, before you accommodate individual candidates, you must screen them with a simple pass-or-out test on the phone or as an assignment. I also like the method proposed there: http://www.jbaltz.com/weblog/2006/04/hiring_geeks.html
You make a good argument, Jack. Why don’t I have any database “stuff”? I guess it really comes down to what I’m trying to test. I’d like to know that the candidate is able to adjust to a domain with-which she is unfamiliar (a senior trait). I’d also like to know that the candidate can solve problems. I wanted to make the test as technology/domain agnostic as possible. Arguably, at least with Fibonacci, I’ve introduced a touch of “math domain” bias (I’ve actually updated this question to correct this, which I’ll blog about soon). As it so happens our particular domain does involve typical business-y database stuff, but there’s also a significant amount of domain complexity to deal with (much more than your typical corporate system), so I’m more interested in raw grey matter here than ability to construct a relational query.
Thanks, Eric. I’d even say that the concept of “raw grey matter” is scientifically disputed. Taking myself as an example, I am brilliant at some kind of example problems, but suck at others. (Incidentally, I love puzzles, algorithms, and the like, but I have severe problems with normal business-type code. Currently this is leading me to try find an IT job that’s not in programming.)
Remember that your interview programming problems are also a very effective way to let the candidate know what kind of work they will do on that job. And besides filtering for the right people, you want to attract the right people, do you? So the question is, after a dead-simple programming question to weed out clowns, can you present some boiled down domain complexity that tests the candidates aptitude and spurs on their imagination?
Also, I’d suggest to mention some of the domain complexity by name in the job ad. Not just to find people with experience in that domain, but also to find people who are intrigued by domain complexity in general (as opposed to simple technical coding).
HTH, Jack
I think the first question was much easier to understand. The “dumbed down” version is far too verbose.
Here’s my solution in Ruby. Done in about 5 minutes.
1 def fib
2 fibs = [0, 1]
3 (2..30).each do |n|
4 f1 = fibs[ n-2 ] || 0
5 f2 = fibs[ n-1 ] || 0
6 fibs.push( f1 + f2 )
7 end
8 puts “Sequence is #{fibs.inspect}”
9 puts “Press enter to quit.”
10 gets
11 end
12
13 fib
That’s because the dumbed-down version is tongue-in-cheek.