« Why did Google create their own browser? | Main | Thanks to Joshua Cyr from the Boston CFUG »

Is it fair to test a candidate during an interview?

A recent posting on Slashdot titled Testing IT Professionals on Job Interviews? was similar to a question posed to me by my manager. After I had forwarded him the test I was planning to use for candidates for our web developer position, he cautioned me that he thought it was possible that someone with years of experience might just be a little insulted at having to take a skills test. I understand that thought completely, and I would never want to insult someone in that manner-- but I just don't think that a test would or should be a problem for a candidate. From my perspective, any smart employer will want to measure just how good of a developer you are, and I would really hope that they've applied the same care in selecting the other developers I'd be working with.

So I want to pose a question to the rest of the community: do you think it's fair to test a candidate during an interview? Oh, and here's the manner in which I apply the test: I always tell the candidate before they come in that I'm going to be testing them; and, I let them browse the web during the test for solutions as long as they're not copying-and-pasting the answers-- after all, I look up other people's tips and solutions myself several times a day. As long as a developer can produce good code in good time during the test, I don't necessarily care whether they wrote it all from scratch or had every answer in their head.

What do you think?

(Edit: Someone made the good suggestion that I post the skills test that I use. Here's the web developer skills test that I use.)

Comments (25)

Here's what it comes down to: attitude.

The idea of insulting an interviewee is simply insane. You want an employee who is going to want to do his best at work. As such, if you give him/her a test during the interview, the mindset should be:

"A test - great! Here's a chance for me to demonstrate how great I am and how much this company is going to like working with me!"

Anything less than that attitude is probably going to be a reflection of the work that they will produce. It they begrudge having to demonstrate their abilities to an employer, it makes you wonder how they will perform when writing code for a client who will never look at the code??


Now with that being said, I think the test style matters the most. I've taken tests in a recruiters office that were insulting, not because of the skill level but because it was multiple choice. I don't think these types of tests work the best. I think the best testing style for candidates is free response. As an employer you can see how they problem solve under pressure.

The fact that employers are complaining that applicants were failing the FizzBuzz problem scares me.

@Ben, I agree the applicant should be excited to show you how far above the competition s/he is.

I agree with Ben. Also, providing the internet during testing is great, because what developer really works in a vacuum?

As an aside, care to post your test? I'm curious ;)

@Justice: Good suggestion. I've created a link to my skills test in the post above.

Actually I do feel a little insulted in general at the idea of testing during interviews, but mostly it depends on the type of test. However I did do one test in Java, and didn't feel insulted about having to take one.

@Chris: what makes a test insulting or not? I'm curious about what you react to in a test or interview.

I've never been offended by a test.

I do feel that my accomplishments rather speak for themselves. Anyone can read up on the articles I wrote for CFDJ, I was twice certified advanced for ColdFusion and for a while was a member of either Team Macromedia or the Adobe Community Experts, both of which are by invitation only. And when Judith is short on content for FAQU, I'm on her short list of people she'll ask to write for her.

However, just because a person *can* go find all that information and read those articles doesn't mean they want to or that they'll do it quickly. For a manager looking to hire, I'm sure an in-house test is a "more efficient" way to verify the person's skill (or at least it would seem to be) since they're not doing something "special" for one candidate compared to the others (most of whom probably don't have a handy collection of magazine articles linked in their resume).

On the other hand I'm also autistic and frequently don't understand what offends other people (or why). It's an unfortunate side-effect of the condition... but it also means that in part I'm not easily offended by others making those same kind of faux pas. Or rather the reverse -- I have a difficult time understanding why other people are offended by things that don't offend me and since there are few things that offend me I find myself making a lot of those social blunders.

And like most autistics, I rarely score below 90% on written tests and have always been in the top 3 or so on interview exams. So to echo Ben's comment at least for me the exam is good because it always shows that I'm being honest when I describe myself as an expert.

Plus I think you have a very pragmatic approach to the exam. I might be a bit annoyed if someone gave me an exam and it was chocked full of obscure theory and they stuck me in an empty cubicle with no computer and an egg timer. Actually come to think of it, that happened to me once. I didn't get the job. :P (Though I don't think it was because of the test.)

So there's my 2c for what it's worth ... I've never been offended, but I don't know if you can really consider my opinion to be "average".

Maybe it depends on how the test was presented but I've walked out of interviews that have forced tests on me.

Test environments do not reflect real world situations. In the real world, programmers have an IDE and code hinting and access to all the documentation.

I do not make candidates sit through tests because I don't believe tests are useful - and I feel the same about certification exams. I ask candidates to tell me about projects that succeeded, projects that failed, what they enjoyed, what they didn't enjoy. I want a candidate to be relaxed, open and honest. A test doesn't tell me what I need to know about a candidate's skills.

Oh, and if you still think testing candidates is worthwhile, make sure you make your entire existing team take the test first and see what results you get under "test conditions". That may change your mind about the value of interview tests too.

Having read your skills test I can see you're verifying some really low-level stuff there. What I'm looking for in a candidate is much higher-level in terms of problem solving. I guess if I was looking for a really junior web developer, I might want to ask them those sorts of questions but I'd still prefer to get them talking about the problem and *communicating* rather than just writing low-level code.

@Jon Dowdle - I guess I don't understand why the fizz-buzz test would be a problem... I did it in about 20 seconds, mostly because I couldn't type any faster. He said on paper (which would slow me down some, because I know I can type faster than I write by hand), but I didn't have to correct any mistakes...

If someone had given that to me as a test question, without having been told about it before hand, I'd have probably guessed that it was expected to be an "easy question".

I'm actually rather shocked that he says most comp-sci graduates can't. I fouled up my opportunity to get a degree when I was younger. And as a result have been denied the opportunity to apply for a lot of jobs merely because I don't have the paper. In retrospect I don't know if it would have helped much because the autism has so badly deteriorated my "people skills" that I've not been able to hold on to the jobs I've had. And the fact that I'm good at logical puzzles like that doesn't help actually I think its made things worse, because they assume that since I'm good at those kinds of puzzles that I therefore must be smart about everything. But that's not the case -- I got a boost in one area and a deficit in the other and frequently things that "everyone knows", I don't know. It's like the "idiot savant" problem only not to that extreme.

I'm anti test. If the dev has made it to the interview, you've already researched this person, googled them etc. They're reputation and work should speak for itself.

I won't interview if there's a test. I don't code in a vacuum, so the test is totally not real world. I sit at a desk, covered in books, with my own web browser and book marks.

I like tests to pose a problem and then ask the candidate how they would solve it. I find good candidates will often give more than one solution or ask questions about the environment. I gave multiple solutions to my current employer and have been told that they really liked that - they gave me the job after all! We also give them a challenge to complete at home as we want to know that they can solve a problem rather than testing their knowledge of tags and attributes which can be looked up. The other thing I like to do is ask questions that don't have a right or wrong answer. This gives me a much better understanding of how they think and will fit into the team.

@Sean: Thanks for your comments. I should have specified that my test is applied on a computer, where the user can access whatever IDE we have installed-- and if they want to work in our office, it's fair to expect them to get used to coding in our environment.

And you're right-- my test does look for low-level stuff that I'd only worry about in a low-level position. And I agree that nothing replaces a good conversation during an interview about what professional challenges a developer has faces and how they've found a solution to them.

I agree with Sean that the test seems to cover only basic stuff, and I suppose that is fine for a low-level entry position. I don't get offended by tests but I also add that to the information I use to evaluate the place and determine if its the kind of place I would like to work. Most of the time, the places that tested me, were not a good fit because it reflected a rigidity on their part and, as noted, doesn't do very well at evaluating critical thinking skills.

I tended to like to ask about hot topics in the industry and find out what books, articles, blogs the person is reading. To me, it isn't what you know but your inquisitiveness and potential to learn. I can teach you ColdFusion easily but I can't easily teach you to become a good problem solver and critical thinker if that potential is not already there.

For those of you who think that a test shouldn't be needed because you can Google for a developer's reputation and code, consider that not every developer has published their own code online. As a matter of fact, I'd go out on a limb with the guess that only a small minority of CF developers are blogging and/or publishing their code.

Yes you should give a test. The test should reflect the level of code you expect from the position.

There are a lot of "advanced" developers out there that can pass a phone interview. Many years with a language != advanced developer in that language.

I asked a 6 year CF developer how to code n! where n = 4 and he didn't even know what n! meant.

A 10 year CF developer had never seen the "group" attribute in <cfoutput>.

My last employer gave a basic CF, SQL and HTML test to everyone we interviewed and one guy cried.

Yes, you should give a test.

The interview itself is a test. However, it is not just the interviewee who is being tested, but rather the interviewer is being tested as well. When I go into an interview nine time out of ten I ask more questions than the interviewer. I am interviewing my potential client to get a feel for their project, to evaluate whether its something of interest to me, and to understand the business problem of the client fully.

However, I think what you are referring to in your post is an actual written test. In that sense, I don't just find them offensive, I find them to be a poor attempt to make up for poor interviewing skills. Unless the role you are trying to fulfill is that of a professional test taker, what does giving a written test really prove? Will the candidate need remember everything from the top of their head, or be able to correctly answer test questions as part of their job? Point being, a written test does not give you a feel for how the candidate will actually perform on the job. On one hand, some folks just don't test well, and a written test, usually a surprise in the interview, may make such folks even more nervous. They may do lousy on the test, but in actuality be star performers in the real work environment, and you will have just passed up on them. On the other hand, some folks are great test takers but can't think creatively at all. I have met tons of Java programmers, for example, who can't do much more than follow other peoples examples, copy paste code and make small changes here and there. Folks like that just can't make a transition to a technology like Adobe Flex, for example, because they are required to think creatively and out of the box. How do you test creativity and out of the box thinking on a written exam?

I once intervied for a contract at AOL. The interviewer gave me a 10 question written test on Java. I hadn't written a line of Java code in over five years, and I had made that clear to the interviewer already. I completed the test, and to my surprise they extended an offer - which I promptly turned down. They had failed my test...

Yes, with one caveat: Don't make it the most critical part of the interview.

Strike up a meaningful conversation about what they like to do, how they work, what tools to they use, what resources are invaluable to them, how they've solved difficult challenges.

You learn more from a candidate when they are in a relaxed, open conversation than from a written test.

What I've found to be much more useful than a "test" is to ask the person to submit what they consider to be a reasonably complex and well-written code sample. It doesn't have to be huge; whatever they think is enough to demonstrate their skill.

Then, and this is the critical part, during the interview, HAVE THEM TALK YOU THROUGH THE CODE. What design forces were at play? Why did they decide to take one approach over other possible approaches? I want to determine two things:

One, did they actually write the code they submitted? (Yes, I have actually had a person submit code that they didn't write, and at the time I didn't realize such a thing was really possible and hired them. Within a short time it was instantly obvious that they had lied baldface.)

Two, what are their problem solving skills and level of design like? I want to know WHY they wrote the code the way they did. I want to know how deeply they considered other possible solutions which were rejected. Etc.

Personally I find this to be a very good compromise. You get the information you need. They get to show off some of their good code. Most hard-core developers I know could talk about their code and their thought process for HOURS. Most of them actually love talking about it. It isn't insulting at all; quite the contrary, I would think the code and their discussion of it would be a welcome opportunity to show off one of their most important skills.

Having just taken the BrainBench test for CF last Friday, I can confidently say that you should not give THAT test! Sixteen multiple choice questions, ALL of them dealing with syntax. Six of the sixteen where about the attributes of the different CFFORM tags. Does any serious CF developer use CFFORM in this world of Ajax and Web 2.0? Two more were about CFABORT withe the SHOWERROR attribute. Really?

I think that tests about concepts are great -- if you want to test a candidate, give him/her a short project to code, or at least some existing code to analyze. Don't simply test someone on his/her memorization of syntax and attributes...that is why we have all these fancy IDEs and great documentation. Smart developers save their brain-cycles for designing elegant solutions!

@ Ben Nadel - I completely agree about your "attitude" statements. We want people who at least don't mind playing along - people who refuse to take any type of test aren't the type that are usually easy to work with/for.

I also agree with the folks who say that the test should be properly designed for the position (junior-level for jr. programmers, and architect-level for architects, etc.)

I don't think that web research can tell you much about an individual's skills problem-solving, fixing bugs, creative thinking, etc. (unless they have a Nadel-like blog that walks you through their exact thought during each step along the way).

I do like the give a test, but the "grade" they get on the technical aspects is not, and this is a secret (shhhhh), the primary thing I'm looking at. I'm primarily looking to see what their reaction is and attitude towards being given difficult technical problems to solve. My test is not written, but rather I have some pre-typed questions that I look ask them verbally.

I want you to be honest and tell me when you don't know something. I want you to try to turn it around, and try to solve something, or where you would go (books, google, forums, a mentor) to figure it out. And please don't make up an answer, or answer *as if* you know when in your heart, you know you're grasping at straws.

@Steve - I agree about striking up a conversation, too. Rapport and personality is essential for long-term employees (it doesn't matter too much for contractors), and I'd hire the candidate with the better attitude and personality over the better test taker every time.

And if time permits, for an intermediate developer and higher, it's a great idea to design a small project for them to code within about an hour. To prep, allow them to bring a thumb drive, or even they're own laptop (so they're resources, code sample, and IDE settings are readily available); if the job will involve lots of maintenance, have them fix some bugs, and make some minor changes in an existing app. If they will be doing lot's of new projects, have them start some base code (CFCs and maybe some simple view code) for a new project. You'll see if they do TDD, unit testing, and architectural design (or just dive in), and other attributes of how they code. And of course you'll get a good idea of how efficient they are under pressure. ;-)

I agree with tests but only in this respect: Judge the approach that the person takes to answering the questions, and not the actual answer itself.

Personally, I don't expect developers to remember syntax, so if someone pseudo codes instead, then thats fine. Don't know the answer? Fine, but don't just shake your head and say "pass" - give as much information as you can, and start a conversation.

The questions I ask are not syntax specific than yours, but would be something like "Explain the difference between an inner and outer join" (I've had about a 20% pass rate on that one!). Sometimes I'll expand after that, and push things as far as I can (maybe ending in some code), but again, I'm not worrying about a person coming up with the syntax on the spot, I'm observing their approach to the question/problem. Everything - their willingness to answer, demeanor, body language, communicating the answer, seeking clarification, etc.

On occasion, if someone didn't do well on a test, I've had someone else give it again when they come back for a 2nd interview. If your primary qualification is "I learn quickly" you better perform better the second time around.

I think the most important part is not to make this a "gotcha" situation. THAT would reflect badly on you, and would probably turn the candidate off, more than anything else.



@David - I guess this is kind of off-topic, but you mentioned asking about the difference between inner and outer joins. I actually find that I've never needed a right join and have never met anyone who ever did... and so I found the semantic of "inner" and "outer" to be overcomplicated really for their purpose.

So when I created the DataFaucet ORM, I changed the semantic. So there aren't "inner" or "outer" joins or "left" or "right" joins, there are just "required" and "optional" joins. I think this is a good simplification of the language. Granted that I have no problem working with the semantics of inner/outer and have never had a problem answering that question in an interview. But just because it's easy for me to do something doesn't mean I don't appreciate making it easier. :)

I do however find it surprising (much like the FizzBuzz thing) that you say you get a 20% pass rate on that question in particular. It's so basic.

Hi ike - yeah, and I use that term because thats how it is presented in documentation and I don't want to get into a "at my company we call it.." kind of situation.

Only recently, a candidate for a position (not a development position on my team, but one that still required SQL skills) gave me a text book answer to the question.....too text book! So I asked a follow up question, which led into a quick example on the whiteboard, which led to the realization that this person had most likely never coded a join of any type, and most likely had never done anything more complicated than a "select * from".

Yeah, I agree, I'm baffled. Its not like I'm giving trick questions or anything, I go out of my way to practically GIVE the answer to the candidate. But like I said, I'm more interested in the candidates approach to the problem/question than the actual answer.



But there are a few tips in order to get that ideal crunch and taste.

Then you can dip them into liquid batter and fry them. In a
casserole, bring to boil 2 cup of water with chicken feet, put fire into low and simmer
10 minute.