Should you use work samples in design interviews?

Employing work samples during your hiring process has long been a source of debate with designers. On one hand it’s wrong to take advantage of someone and get them to do free work but on the other hand, how do you know a designer is as good as they say they are?

A designer may have a really outstanding portfolio but I find I’m often left questioning the role someone played in the work captured in the portfolio. Were they the only designer, is this 100% their work or were they part of a team and if so how much of what is captured is their work. I actually prefer to use work samples as I find it’s the only really reliable way of seeing how someone works. 

A few years back our company used a fairly typical work sample where we would ask a designer to design a flow or page and give us a presentation on their work. I’m glad to say that this kind of test has very much gone out of fashion and good riddance! The test is unrealistic, in the real world we don’t want designers working away by themselves. We have a very social and open design practice, our designers need to collaborate and work with a diverse team of stakeholders and peers to accomplish their design goals. We wanted to ensure designers who join our team would thrive in such an environment so set out to create a challenge that helped us test them in a realistic simulation.

What problems are we trying to solve?

• We needed a test that was fair to the candidate
  • Fair in terms of effort required
  • Fair in that it didn’t require any specialist knowledge
    
    
• We wanted a realistic test that would help give us confidence that a candidate would be a good fit for the team.

 
We didn’t want to take away days of a candidate’s free time as this is unfair and could also introduce bias. What if I’m a busy parent and can’t dedicate as much effort as someone without children. Or what if my existing role is demanding and leaves little spare time? We could do a purely reactive test where the candidate is given the test cold though we also wanted to give candidates thinking time.

We also thought that the test should reflect life as a designer at hipages. This would make the test realistic and hopefully give us more confidence about the suitability and fit of a candidate. But how do you make a short one hour interview a realistic simulation of something which normally takes days or weeks or months?

What makes a good designer?

This is the question we set out to understand and answer as once we had a model for what a good designer looks like, we could ten create tests to look for people who match the model. I ran a workshop with members of our design and product team to build an answer to this question.

I worked with the group to identify the qualities that we value in our colleagues who are designers. I then asked the group to consider what are the factors in our work environment and culture that make it hard to be a successful designer. Our workshop produced these two lists:

Qualities we value

• Collaboration
• Communication
• Design leadership
• Ability to determine what is the right problem to solve
• Flexibility

What makes it hard to be a good designer?

• Lack of time to do a good job
• Sometimes it’s hard to learn necessary context
• Delivery constraints often mean only simplest design is released

So… if we could test that designers could exhibit the qualities we were looking for whilst overcoming some of the issues we encounter every day then the candidates who do well in the test should do well in the actual role.

Crafting the challenge

In the workshop where we examined what a good designer is, the quality that came through loudest was the ability to collaborate so we knew that we would need to design an experience where candidates worked with us to accomplish some goal but we also wanted to test their ability to get to the heart of a problem AND work under a bit of pressure. Our solution was to ask a candidate to run a mini fact finding / context learning workshop with their prospective team members? This would allow us to see how they take a problem apart and give the team an opportunity to test drive working with the candidate. 

We also wanted to see how they analyse and synthesise information then use this in their design process so we also wanted them to do some whiteboarding with us show their thinking and collaborate on some ideas. And… finally, we wanted it to take no more than sixty minutes. To this end we designed a simple process of:

• 20 minutes workshop
• 20 minutes analysis and synthesis
• 20 minutes whiteboarding  

OK, we had a prototype, we now needed a problem to solve. I was keen to avoid real problems that we had encountered as I have had issues in the past with interviewers forgetting that candidates don’t know our business and dismissing their ideas as invalid based on either our research or our experiments. To this end I created an entirely fictional problem about a queue at an ice cream shop.

As it turns out, we’re really bad actors! When we ran the exercise with candidates we were inconsistent with our input and our data as we were making it up. This meant that different candidates each got slightly different problems so it was hard to compare.

I found the sweet spot to be a fictitious or unexplored problem in a real problem space. With this scenario I found that interviewers were aware of the background and had consistent data to hand as it was real data but they never dismissed ideas as we hadn’t yet looked into the specific problem. This meant each candidate got the same input from the interviewers, the interviewers could play themselves (and play to their strengths) but they all kept an open mind regarding potential solutions.  

In terms of who is in the interview, I like to have the triad members for the squad the role is in present plus any additional designers in the team. So, in the interview we have the product manager for the team who we’re hiring for, the tech-lead for the team, myself as the hiring manager and also other designers from the squad. This may seem a bit daunting to the candidate but it is exactly the same people they would be working with if they landed the role so is an excellent way of testing for character and fit. Plus, we do have a laugh in the interview like we do in real life so we actively try and make it less intimidating.  I brief all the interviewers beforhand to answer honestly and if we don’t know an answer then we’ll state an assumption and move on.

In terms of how does a candidate pass the challenge, this is what I look for:

Did the candidate have a plan?
Ideally for both the workshop and the whiteboarding.

Did they do at least some research about the problem space?
We don’t expect them to slavishly research and we try to use design problems that anyone could easily relate to such as scheduling or filtering or e-commerce check out processes. 

How are they with the other participants, are they able to ask questions of them and collaborate?
Better still, do they proactively engage with all the interviewers, engaging each on their specialist subject.

What questions do they ask to get to the root of the problem?
How long does it take to get to a question like “Why is this a problem?” or “How do we know this is the real problem?” 

Do they make best use of the time?
Good designers tend to use a structured design process but when you only have twenty minutes you need to know which bits of that process are essential and which can be left out.

What do they carry through from research into design?
How do they take what they learnt in the first twenty minutes and apply it to their interaction design process?

How do they lead the design session?
What ideas do they bring and how do they then refine, explore or invalidate them with the interviewers?

Do they actually bring any good ideas to the table to solve the problem at hand?
Do they go for the obvious answer or explore other avenues? Do their ideas address the identified root problem?

This is an example of the brief we give to the candidates

This was made in Google Slides using images from unDraw.

Reflections

I’ve run this design challenge dozens of times across many role with a variety of problems and I find it to be exceptionally revealing in terms of showing how a design processes a problem and collaborates with peers.

These are some of my observations from running this challenge:

• It’s hard! A lot of designers interview strongly and then perform badly or very badly in this challenge. This is not a bad thing, after all this is why we test.
• All high performing candidates came prepared, lots of designers try to wing it, some even do an ok job of it but all the notable high performers prepped and planned for the session
• Most designers run out of time. We’re not time bullies, we do give extra time but again the notable high performers tend to run pretty close to on time
• I’ve had feedback that we’re just testing workshop facilitation. I don’t think so, we set a workshop context as this is a great vehicle for ideation and collaboration. These are the skills we really want to see. It doesn’t really matter if the workshop was a little rough or rushed or overran or we didn’t get to the end. We just want to see how you explore a problem and how you collaborate with peers.
• Some people have commented that we’re not testing for interaction design finesse but we use our work samples as a second interview. In order to have gotten this far a candidate would have had to get through a phone screen and a first interview containing a brief portfolio review. I would hope that these steps are sufficient to understand a candidate’s design ability and maturity.
• I added a step to the work sample brief document offering a chance to ask questions before the session and even get feedback on what they were planning. All the successful candidates take advantage of this opportunity to refine their thinking before the day.
• What about remote interviews due to covid? We have run several rounds of interviews remotely and we tell the candidates that we appreciate it adds another level of pressure but we also reassure them that we’re not looking for Mural masters or Zoom Maestros we just want to see how you dissect a problem and collaborate. As long as whatever it is you do demonstrates your process and drives some collaboration that’s fine. 
• Mural and Miro are terrible whiteboarding tools for sketching. I have run this challenge both IRL and online, a physical whiteboard nearly always ends up with more designs produced and a higher level of fidelity. But again, the good designers find a way. Sometimes they drop in screen grabs and sketch over them or they just use boxes and lines to make ultra-simple wireframes.
  Candidates seem to find the process enjoyable and challenging. I’ve had more than one design say that they want to take the process and use it in their next place or work.

And finally...

One final note on this design challenge. I always offer unsuccessful candidates an opportunity for face to face feedback where I tell them where they could have improved and how successful candidates handled the same scenario. I think this is both respectful and essential. Candidates have put in the effort to complete the challenge and so the least we can do is be prepared to offer feedback. I personally really hate it when organisations just give a generic “no thanks” response. How is the candidate expected to get better? These feedback conversations have more than once proved really useful, sometimes resulting in candidates reapplying for subsequent roles and being successful based on what they learnt from the feedback.

I’d love to hear your thoughts about this topic. Feel free to email me on julian.munford@googlemail.com or tweet me on @julesmunford 

Project thumbnail image by Amélie Mourichon on Unsplash
https://unsplash.com/photos/wusOJ-2uY6w

Contact

Jules Munford
Phone: 0431 414322
Email: julian.munford@googlemail.com
Twitter: @julesmunford

© Julian Munford 2020