Senior Developer Assessment Revisited

This is really part two of the article I wrote “What is a Senior Developer?”.  I’ve received some shrill feedback on my choice of assessment problem:

  • Too math-y!
  • Standards too exacting!
  • A bit much to ask of your typical commercial developer.

So I’ve taken this all to heart and decided to revamp the Solver question.  Actually, I’ve decided to drop it completely and replace it with something a lot less “math’y” but possibly no more representative of real-world requirements.

A suggestion that was given me: “get them to sort stuff”.  Ok, so what would making someone jump through “sort algorithm” hoops prove?  After all, these days, sorting things amounts to a call to List<T>.Sort—I mean honestly, who ever needs to resort to first principles when sorting these days?  I’m willing to take a different tack—if I’m testing something slightly different, that is, not knowledge, but the ability to absorb and apply … well then that’s slightly different.  Besides … dealing with pointers is generally seen as unnecessary masochism, but some people still regard it as crucial background to being a good developer.

So this time, I’ve actually taken the time to capture the requirements in detail; this amounts to softening things up a little since the general consensus seems to be that the original assessment was too demanding (at least, the Solver question was).

image

Quicksort, above, doesn’t test ability to perform research independently, and offers a lot of hand-holding, but it is somewhat less daunting than Solver.  I can’t help thinking though that things are being dumbed-down a little too much.

The dumbing-down:

  • This is limited to System.String, but could easily be extended to be generic (bonus points if the candidate takes the initiative to do this!);
  • I haven’t specified any constraints in terms of efficiency issues (the naïve implementation is, of course, a horrible memory hog);
  • I don’t know if I could provide any more hand-holding than this, it’s practically paint-by-numbers.

I have been somewhat vague about one thing, namely choice of pivot.  I have arguably been a little tricky in this question because the example isn’t consistent in how the pivot is chosen.  The astute candidate will quickly realise that choice of pivot isn’t crucial.

Here’s my solution, coded up in approximately 20 minutes:


	public static class QuickSorter
	{
		public static IEnumerable&lt;string&gt; QuickSort(IEnumerable&lt;string&gt; jumbled)
		{
			if (jumbled.Count() &lt; 2)
				return jumbled;
			else
			{
				return
					QuickSort(AllLessThan(jumbled.ElementAt(0), jumbled.Skip(1)))
					.Concat(jumbled.Take(1))
					.Concat(
					QuickSort(AllGreaterThan(jumbled.ElementAt(0), jumbled.Skip(1))));
			}
		}

		private static IEnumerable&lt;string&gt; AllLessThan(string value, IEnumerable&lt;string&gt; others)
		{
			return AllSatisfying(others, s =&gt; String.Compare(value, s) &gt; 0);
		}

		private static IEnumerable&lt;string&gt; AllGreaterThan(string value, IEnumerable&lt;string&gt; others)
		{
			return AllSatisfying(others, s =&gt; String.Compare(value, s) &lt;= 0);
		}

		private static IEnumerable&lt;string&gt; AllSatisfying(IEnumerable&lt;string&gt; others, Predicate&lt;string&gt; predicate)
		{
			return others.Where(s =&gt; predicate(s));
		}
	}

Things to notice about my implementation:

  • It’s pretty much declarative, thanks to Linq, of course;
  • It’s not very efficient … no in-place swapping; that’s what you get in twenty minutes.

I think that this provides a less jarring assessment experience for a would-be candidate than Solver, especially if our candidate isn’t a math-wiz.

26 thoughts on “Senior Developer Assessment Revisited

  1. Grant Austin

    This question is too easy, the Solver one was just about right.

    For me, a programming problem during an interview is more a test of my level of nervousness than anything else.

    Reply
  2. Eric Smith

    @Blake, I purposely avoided FizzBuzz due to its “fame”. But thinking about it again, I realise that the sort of person that this type of assessment is best at “filtering out” wouldn’t have known about FizzBuzz anyway :)

    Reply
  3. Drew

    Good to know that my intuition is right and that my title as “Senior Software Engineer” is likely undeserved. Though, I can write some mean “put-stuff-in-the-database-and-take-it-out-again” code and I have managed to deliver some pretty decent software. :)

    I can do this but I couldn’t do the Solver one without some significant sitting down and remembering my algebra.

    Reply
  4. Ian

    As a recent grad looking for my first “real” position it is interesting to see that so many people are unable to do this. Though, I’m not sure how valuable it is to you. I think the ‘Solver’ question is much more reasonable for many of the reasons you mention, especially that many people haven’t thought about sorting in a long time (Freshman year). In addition it provides you more info about the candidate which is the real end-goal of the interview. Like you noted: if they are willing to ask/search for the algorithm it tells you if they are willing to use all resources available even if it means they may look “dumb” for not knowing.

    On a side note: the hardest part for me, and my friends who are in the same boat, is dealing with time constraints/ atmosphere of the interview. Basicily, we’re perfectly capable of doing it at home, but crack under pressure when it counts.

    Reply
  5. Eric Smith

    @Ian – on your last note, just don’t know how to address that other than to say that I am aware of it, and deliberately soften the questions to offset the nervousness factor.

    The test has to be administered in a controlled environment, so no “take home”. I’ve already decided to remove Internet access because we’ve now had two instances of guys copying code directly off the ‘net without even bothering to try to cover their tracks. In this case, the test has proven worthy in revealing candidate character.

    Reply
  6. Pingback: Desperately Seeking Senior « The Limber Lambda

  7. Pingback: Weekly Link Post 134 « Rhonda Tipton’s WebLog

  8. Chris Lininger

    The best interview test I ever took was a 48 hour take-home. It was virtually impossible to complete in 48 hours, so the instructions were to complete as much as possible, using any available resources. It touched on numerous disciplines: from basic coding, requirements analysis, database design, etc. During the subsequent face-to-face interview, you had to explain and defend (and understand!) your answers. The two key advantages IMO are 1) it weeds out any casual applicant not willing to devote 48 hours, and 2) it focuses the interview time on exploration of the thought process, not panicked coding. The longer time span allows for demonstration of real problem solving skills. On-the-spot recall of the mod operator or recursion are going to be very hit or miss (mostly miss), and don’t accurately evaluate a programmer’s day to day activity. I expect a “senior” programmer to be working much higher-level problems, like how (and when) to apply MVC patterns, for example.

    Reply
    1. Eric Smith

      @Chris – this is a ‘screener'; the MVC pattern application questions happen in the (subsequent) interview.

      IMO a senior should be working at *all* levels, but should eat this up for breakfast.

      Reply
  9. XuaXua

    Test, schmest. A math wizard with zero applicable intuitive skill can solve any test like this.

    You’re looking for an experienced candidate, meaning someone who can resolve high level problems. And if you’re the interviewer, it’s likely you, yourself, have some experience.

    Therefore, pull from your experience. Think back to your most challenging on-the-job problems, and recall how you resolved them and the process you undertook.

    Then, reiterate the problem to the interviewee in a manner in which he or she can undertake it. Ask them how they’d approach the problem, the sorts of resources they’d use, and from their experience, what it might possibly be. Handle it like a Zork text adventure. Allow them to have any approach open, and resolve from any angle that appears applicable.

    This is a test to figure out how sharp your candidate is and how well they can think out of the box.

    Reply
  10. John H.

    “Math’y” ???

    A new and exciting (mis)use of the apostrophe.

    Please follow these guidelines for the use of the Apostrophe.

    http://theoatmeal.com/comics/apostrophe

    The Oatmeal knows.

    Try a hyphen there. In English, the apostrophe generally indicates possesion, or a word contraction (letters were dropped, and where dropped)

    Reply
  11. Pingback: Senior Developer Assessment: Re-aligning Expectations « The Limber Lambda

  12. Anhar

    Eric I think the Solver was just the right level, I think you may want to have a series of tests rather than just one.

    For example FizzBuzz as the ‘clown filter’, get rid of the clowns :)

    Then QuickSort to get the Good developers, and finally those that at least make half attempt at the solver as the better Candidates.

    Anhar

    Reply
  13. Steve Sperandeo

    @XuaXua
    You are bang on. :-)

    @Eric
    I think removing internet access would be a mistake. Let them cheat. Why block a perfectly good character test (analyse their internet tracks after and see if they copied the code)? Just make sure you explicitly state that you want them to write it from scratch and not copy it.

    Honestly, I like both XuaXua and Eric’s approaches. I don’t think they’re mutually exclusive. However, I think that you can read a good developer just by talking to them, if you’re a developer. Only past or current developers should be hiring and managing developers. The single biggest problem with these “non-coders” flying under the radar is that the hiring or managing person doesn’t have the necessary specific hands-on skills to be in that position.

    If you’re in such a position: pause, take a deep breath, park your ego and take a hint that you shouldn’t be managing these people or software development at all if you don’t understand what’s happening under the hood. Hire a good past developer combined software project manager and delegate the management to them.

    Reply
    1. Eric Smith

      Steve – I actually think you’re right about your suggestion, so I’ll re-instate the Internet access and make it explicit in the assessment that you’re required to do the problem yourself. Really clever individuals will, of course, cover their tracks though :)

      Reply
      1. Steve Sperandeo

        Hi Eric,

        You just have to be one step ahead of them. If you log the requests in the firewall and not on the provided computer, they won’t be able to erase the log. Also, limiting access to only outbound ports 53 and 80 (and arguably 443) is a cheap way to limit their connections to ssh tunnels and proxies. I’d still log requests to the other ports (so you know if they attempted to connect to ssh).

        Another option would be to not block the requests, but instead just log them. You’ll be able to see the context of what they’re doing, which is sometimes good enough. Seriously, the only reason people would use ssh during an interview would be to copy or review some old source code that they had written in the past. That brings up a very interesting point.

        Is looking at your old source code considered cheating? Old source code is something akin to looking at other people’s websites and code snippets on the internet. Arguably, it doesn’t violate any licenses. In my humble opinion, the best developers are extremely resourceful developers; so, couldn’t old source be considered just another resource?

        Comments?

        Reply
  14. Eric Smith

    Steve – I don’t think it’s worth blocking ports at all, and logging activity is definitely the way to go.

    To be honest, given the time constraint, I wouldn’t expect a particularly sophisticated attempt to cover tracks. What’s more is, the more sophisticated the attempt to cover tracks, the less likely they’ll actually need to go to those lengths to do so (i.e., they’ll probably be the sort of person who will nail the test *anyway*).

    My concern really is use of InPrivate or clearing the browser cache because it’s just so darned easy. Yes, looking at firewall logs gets around that, and I even considered setting up a proxy server on my own PC that the browser will point at (with suitable rules to prevent access any other way). In all honesty though, I don’t think it’ll be needed.

    I do not believe that looking at source code is cheating at all—however, copying source code *verbatim* without understanding anything about it *is* cheating.

    Reply
  15. None Required

    I’d say these assignments unfairly favour parts of the crowd.
    At some universities, quick sort is one of the most used examples, so many of those who’ve been there probably know it by heart, and are extremely familiar with it.
    At some other university, quick sort are only mentioned at a glance and other examples are used.

    So if I were you, I would use a real life example instead, more geared towards structure than algoritm.
    Structures are hard to fix if someone have made bad ones, algorithms aren’t. Usually.

    Reply
  16. bwc

    I don’t know about you guys. But I found the Solver problem to be easier than this Quick Sort. Maybe I haven’t code recursion that much or that I don’t have a CS degree. ( I can code simple recursions like fib or string reverse ). Or maybe I’m overthinking this problem..and this is in the comfort of my home!

    Reply
  17. bwc

    Actually, I realized the problem is this. It’s because I don’t know much about the details of Quicksort (although heard of it and it’s fast) and therefore just using the problem explanation and example above, it wasn’t as clear as I want it to be.

    I was thinking what happens when you have Aaron on the left of Carol (I mean everything in the example seems so orderly and simple without loopholes). A better example detail explanation for me at least is the bigger set of array of numbers and not as orderly initial set like the one found in

    http://www.algolist.net/Algorithms/Sorting/Quicksort

    It’s the partitioning that I was confuse about.

    Reply
  18. N Ven

    I think everything / everyone above is missing the point. Why do we hire? Intellectual and emotional capital. Our abilities to memorize, reason and solve problems vary individual to individual. If we focus too intently on the specific ability to recall very specific details about certain problems, we lose focus on the fact that we are hiring to increase intellectual and emotional quota within the organization.

    If you really want to weed out the inappropriate candidates, ask a range of basic questions to gauge their range of knowledge. From the more old school questions ‘how is a string stored in memory’, ‘what is a pointer’, to much more new school questions ‘what is a pattern’, ‘what are your thoughts on using 3rd party extensions vs coding solutions in house’, ‘when should you unit test’. Nobody will have perfect answers for all questions, but understanding where the candidate has focused their knowledge development will help you gauge how much to expect from the candidate. A truly passionate developer will A) Know of, or have some experience with the answer or B) Have the balls to say ‘I don’t know the specifics, but I’d really like to find out… can we discuss it for a moment?’.

    They only take a few minutes to ask, and if someone can answer some of these questions with honest thought and understanding, you’re way beyond the ‘write some code you code-monkeying code monkey’. No screwing around with net meetings and code review and all that bs.

    You want intelligent and relevant candidates? Ask intelligent and relevant questions.

    Reply
    1. Eric Smith

      Agree wholeheartedly. What you’re talking about though is what we’d do in the interview. This is pre-interview screening stuff–the answer to the question: should we be investing time to ask the important questions?

      Reply
  19. Zbigniew Tomczak

    Man, I totally agree with you. I’m a developer. I work hard. Any recruitment process is very time consuming if you are busy. And, I noticed that, those which are the best are also most busy. For example, do anyone know skilled, hard working professional who possibly has also family or his own life to sit consecutive 48 hours to solve some recruitment tasks?

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Current day month ye@r *