Senior Developer Assessment Revisited
This is really part two of the article I wrote “What is a Senior Developer?”. I’ve received some shrill feedback on my choice of assessment problem:
- Too math’y!
- Standards too exacting!
- A bit much to ask of your typical commercial developer.
So I’ve taken this all to heart and decided to revamp the Solver question. Actually, I’ve decided to drop it completely and replace it with something a lot less “math’y” but possibly no more representative of real-world requirements.
A suggestion that was given me: “get them to sort stuff”. Ok, so what would making someone jump through “sort algorithm” hoops prove? After all, these days, sorting things amounts to a call to List<T>.Sort—I mean honestly, who ever needs to resort to first principles when sorting these days? I’m willing to take a different tack—if I’m testing something slightly different, that is, not knowledge, but the ability to absorb and apply … well then that’s slightly different. Besides … dealing with pointers is generally seen as unnecessary masochism, but some people still regard it as crucial background to being a good developer.
So this time, I’ve actually taken the time to capture the requirements in detail; this amounts to softening things up a little since the general consensus seems to be that the original assessment was too demanding (at least, the Solver question was).
Quicksort, above, doesn’t test ability to perform research independently, and offers a lot of hand-holding, but it is somewhat less daunting than Solver. I can’t help thinking though that things are being dumbed-down a little too much.
The dumbing-down:
- This is limited to System.String, but could easily be extended to be generic (bonus points if the candidate takes the initiative to do this!);
- I haven’t specified any constraints in terms of efficiency issues (the naïve implementation is, of course, a horrible memory hog);
- I don’t know if I could provide any more hand-holding than this, it’s practically paint-by-numbers.
I have been somewhat vague about one thing, namely choice of pivot. I have arguably been a little tricky in this question because the example isn’t consistent in how the pivot is chosen. The astute candidate will quickly realise that choice of pivot isn’t crucial.
Here’s my solution, coded up in approximately 20 minutes:
public static class QuickSorter
{
public static IEnumerable<string> QuickSort(IEnumerable<string> jumbled)
{
if (jumbled.Count() < 2)
return jumbled;
else
{
return
QuickSort(AllLessThan(jumbled.ElementAt(0), jumbled.Skip(1)))
.Concat(jumbled.Take(1))
.Concat(
QuickSort(AllGreaterThan(jumbled.ElementAt(0), jumbled.Skip(1))));
}
}
private static IEnumerable<string> AllLessThan(string value, IEnumerable<string> others)
{
return AllSatisfying(others, s => String.Compare(value, s) > 0);
}
private static IEnumerable<string> AllGreaterThan(string value, IEnumerable<string> others)
{
return AllSatisfying(others, s => String.Compare(value, s) <= 0);
}
private static IEnumerable<string> AllSatisfying(IEnumerable<string> others, Predicate<string> predicate)
{
return others.Where(s => predicate(s));
}
}
Things to notice about my implementation:
- It’s pretty much declarative, thanks to Linq, of course;
- It’s not very efficient … no in-place swapping; that’s what you get in twenty minutes.
I think that this provides a less jarring assessment experience for a would-be candidate than Solver, especially if our candidate isn’t a math-wiz.
This question is too easy, the Solver one was just about right.
For me, a programming problem during an interview is more a test of my level of nervousness than anything else.
Grant Austin
February 22, 2010 at 6:29 PM
You might think so, but honestly, no question is too easy for large numbers of people to get wrong. At least that’s been my experience. :)
See also: FizzBuzz.
Blake Winton
February 23, 2010 at 3:57 AM
@Blake, I purposely avoided FizzBuzz due to its “fame”. But thinking about it again, I realise that the sort of person that this type of assessment is best at “filtering out” wouldn’t have known about FizzBuzz anyway :)
Eric Smith
February 23, 2010 at 5:21 AM
Good to know that my intuition is right and that my title as “Senior Software Engineer” is likely undeserved. Though, I can write some mean “put-stuff-in-the-database-and-take-it-out-again” code and I have managed to deliver some pretty decent software. :)
I can do this but I couldn’t do the Solver one without some significant sitting down and remembering my algebra.
Drew
February 23, 2010 at 5:45 AM
As a recent grad looking for my first “real” position it is interesting to see that so many people are unable to do this. Though, I’m not sure how valuable it is to you. I think the ‘Solver’ question is much more reasonable for many of the reasons you mention, especially that many people haven’t thought about sorting in a long time (Freshman year). In addition it provides you more info about the candidate which is the real end-goal of the interview. Like you noted: if they are willing to ask/search for the algorithm it tells you if they are willing to use all resources available even if it means they may look “dumb” for not knowing.
On a side note: the hardest part for me, and my friends who are in the same boat, is dealing with time constraints/ atmosphere of the interview. Basicily, we’re perfectly capable of doing it at home, but crack under pressure when it counts.
Ian
February 23, 2010 at 8:21 PM
@Ian – on your last note, just don’t know how to address that other than to say that I am aware of it, and deliberately soften the questions to offset the nervousness factor.
The test has to be administered in a controlled environment, so no “take home”. I’ve already decided to remove Internet access because we’ve now had two instances of guys copying code directly off the ‘net without even bothering to try to cover their tracks. In this case, the test has proven worthy in revealing candidate character.
Eric Smith
February 26, 2010 at 5:02 AM
[...] leave a comment » Recently I chatted about appropriate coding assessment questions for senior developers, and came to the conclusion that Solver was a little too demanding for someone to do in around twenty minutes (under pressure), so I replaced it with Quicksort. [...]
Desperately Seeking Senior « The Limber Lambda
February 27, 2010 at 5:01 PM
[...] Senior Developer Assessment Revisited – Eric Smith continues his assessment of the characteristics of a Senior Developer. [...]
Weekly Link Post 134 « Rhonda Tipton’s WebLog
March 1, 2010 at 2:10 AM
The best interview test I ever took was a 48 hour take-home. It was virtually impossible to complete in 48 hours, so the instructions were to complete as much as possible, using any available resources. It touched on numerous disciplines: from basic coding, requirements analysis, database design, etc. During the subsequent face-to-face interview, you had to explain and defend (and understand!) your answers. The two key advantages IMO are 1) it weeds out any casual applicant not willing to devote 48 hours, and 2) it focuses the interview time on exploration of the thought process, not panicked coding. The longer time span allows for demonstration of real problem solving skills. On-the-spot recall of the mod operator or recursion are going to be very hit or miss (mostly miss), and don’t accurately evaluate a programmer’s day to day activity. I expect a “senior” programmer to be working much higher-level problems, like how (and when) to apply MVC patterns, for example.
Chris Lininger
March 1, 2010 at 5:09 PM
@Chris – this is a ‘screener’; the MVC pattern application questions happen in the (subsequent) interview.
IMO a senior should be working at *all* levels, but should eat this up for breakfast.
Eric Smith
March 2, 2010 at 5:25 AM
Test, schmest. A math wizard with zero applicable intuitive skill can solve any test like this.
You’re looking for an experienced candidate, meaning someone who can resolve high level problems. And if you’re the interviewer, it’s likely you, yourself, have some experience.
Therefore, pull from your experience. Think back to your most challenging on-the-job problems, and recall how you resolved them and the process you undertook.
Then, reiterate the problem to the interviewee in a manner in which he or she can undertake it. Ask them how they’d approach the problem, the sorts of resources they’d use, and from their experience, what it might possibly be. Handle it like a Zork text adventure. Allow them to have any approach open, and resolve from any angle that appears applicable.
This is a test to figure out how sharp your candidate is and how well they can think out of the box.
XuaXua
March 2, 2010 at 2:39 AM
“Math’y” ???
A new and exciting (mis)use of the apostrophe.
Please follow these guidelines for the use of the Apostrophe.
http://theoatmeal.com/comics/apostrophe
The Oatmeal knows.
Try a hyphen there. In English, the apostrophe generally indicates possesion, or a word contraction (letters were dropped, and where dropped)
John H.
March 2, 2010 at 3:06 AM
Dang nang. Muphry’s Law struck again.
I meant “possession”
John H.
March 2, 2010 at 3:07 AM
[...] using a small test that I drafted some time ago. Since then I’ve spoken about how the test was dumbed down since one of the questions was perceived as being too difficult. The assessment consists of three [...]
Senior Developer Assessment: Re-aligning Expectations « The Limber Lambda
March 7, 2010 at 8:21 PM
Eric I think the Solver was just the right level, I think you may want to have a series of tests rather than just one.
For example FizzBuzz as the ‘clown filter’, get rid of the clowns :)
Then QuickSort to get the Good developers, and finally those that at least make half attempt at the solver as the better Candidates.
Anhar
Anhar
March 11, 2010 at 10:05 PM
@XuaXua
You are bang on. :-)
@Eric
I think removing internet access would be a mistake. Let them cheat. Why block a perfectly good character test (analyse their internet tracks after and see if they copied the code)? Just make sure you explicitly state that you want them to write it from scratch and not copy it.
Honestly, I like both XuaXua and Eric’s approaches. I don’t think they’re mutually exclusive. However, I think that you can read a good developer just by talking to them, if you’re a developer. Only past or current developers should be hiring and managing developers. The single biggest problem with these “non-coders” flying under the radar is that the hiring or managing person doesn’t have the necessary specific hands-on skills to be in that position.
If you’re in such a position: pause, take a deep breath, park your ego and take a hint that you shouldn’t be managing these people or software development at all if you don’t understand what’s happening under the hood. Hire a good past developer combined software project manager and delegate the management to them.
Steve Sperandeo
March 12, 2010 at 12:30 PM
PS.
How you know you’re in such a position: you fail the clown test (FizzBuzz). :-)
Steve Sperandeo
March 12, 2010 at 12:44 PM
Steve – I actually think you’re right about your suggestion, so I’ll re-instate the Internet access and make it explicit in the assessment that you’re required to do the problem yourself. Really clever individuals will, of course, cover their tracks though :)
Eric Smith
March 13, 2010 at 8:05 AM
Hi Eric,
You just have to be one step ahead of them. If you log the requests in the firewall and not on the provided computer, they won’t be able to erase the log. Also, limiting access to only outbound ports 53 and 80 (and arguably 443) is a cheap way to limit their connections to ssh tunnels and proxies. I’d still log requests to the other ports (so you know if they attempted to connect to ssh).
Another option would be to not block the requests, but instead just log them. You’ll be able to see the context of what they’re doing, which is sometimes good enough. Seriously, the only reason people would use ssh during an interview would be to copy or review some old source code that they had written in the past. That brings up a very interesting point.
Is looking at your old source code considered cheating? Old source code is something akin to looking at other people’s websites and code snippets on the internet. Arguably, it doesn’t violate any licenses. In my humble opinion, the best developers are extremely resourceful developers; so, couldn’t old source be considered just another resource?
Comments?
Steve Sperandeo
March 13, 2010 at 5:08 PM
Steve – I don’t think it’s worth blocking ports at all, and logging activity is definitely the way to go.
To be honest, given the time constraint, I wouldn’t expect a particularly sophisticated attempt to cover tracks. What’s more is, the more sophisticated the attempt to cover tracks, the less likely they’ll actually need to go to those lengths to do so (i.e., they’ll probably be the sort of person who will nail the test *anyway*).
My concern really is use of InPrivate or clearing the browser cache because it’s just so darned easy. Yes, looking at firewall logs gets around that, and I even considered setting up a proxy server on my own PC that the browser will point at (with suitable rules to prevent access any other way). In all honesty though, I don’t think it’ll be needed.
I do not believe that looking at source code is cheating at all—however, copying source code *verbatim* without understanding anything about it *is* cheating.
Eric Smith
March 15, 2010 at 5:37 AM
I’d say these assignments unfairly favour parts of the crowd.
At some universities, quick sort is one of the most used examples, so many of those who’ve been there probably know it by heart, and are extremely familiar with it.
At some other university, quick sort are only mentioned at a glance and other examples are used.
So if I were you, I would use a real life example instead, more geared towards structure than algoritm.
Structures are hard to fix if someone have made bad ones, algorithms aren’t. Usually.
None Required
March 29, 2010 at 11:36 AM
Funny, I was thinking the same thing about the solver problem. I have a degree in Math, straight out of university I could have coded the solution to that no problem. Now I would really have to think about it and go through the links provided.
The problem with these tests is that it’s treating the process backwards. If you want someone who is really good at what they do, no matter what discipline you’re hiring, they have to choose to work for you. You want it to be as easy as possible for them to find you and get through the interview process while still filtering out the people you don’t want. Good people do not need a job, they already have one. The more hoops you make them jump through the less likely they’ll bother to keep going. This is why tests like ‘reverse a string’ or fizzbuzz are great. They’re easy to understand, fast to implement, and shouldn’t require any outside resources if you allow the interviewee to use whatever language they want. Then you give them a complex problem and have them work through some general design and implementation ideas without talking about code to make sure they can manage some high-level design. Between these 2 types of tests you should be able to filter out the non-programmers while not putting up too many roadblocks to deter someone who needs to choose to work for you.
Tim
July 2, 2010 at 5:25 PM
I don’t know about you guys. But I found the Solver problem to be easier than this Quick Sort. Maybe I haven’t code recursion that much or that I don’t have a CS degree. ( I can code simple recursions like fib or string reverse ). Or maybe I’m overthinking this problem..and this is in the comfort of my home!
bwc
May 23, 2010 at 4:26 PM
Actually, I realized the problem is this. It’s because I don’t know much about the details of Quicksort (although heard of it and it’s fast) and therefore just using the problem explanation and example above, it wasn’t as clear as I want it to be.
I was thinking what happens when you have Aaron on the left of Carol (I mean everything in the example seems so orderly and simple without loopholes). A better example detail explanation for me at least is the bigger set of array of numbers and not as orderly initial set like the one found in
http://www.algolist.net/Algorithms/Sorting/Quicksort
It’s the partitioning that I was confuse about.
bwc
May 23, 2010 at 5:00 PM