The Design Sprint

At Google Ventures, they use a five-day process for taking a product or feature from design through prototyping and testing. It's called a product design sprint. This is a compilation of multiple posts on running your a design sprint.

Before the sprint:

Review the following:

Pick a Big Fight

The first thing you need is an important design problem. It might be:

  • Something big, like defining your product for the first time
  • A big redesign
  • A new feature
  • Improving conversion on a single user action

It just has to be really important to the company, and it has to be something you’re struggling to start or to make progress on.

Get the Right People

The ideal sprint team is between four and eight people. Make sure you have at least one:

  • Designer: If your startup doesn’t have a designer yet, try to bring in a ringer.
  • CEO: At a small startup, the CEO is the key decision-maker and needs to participate in order to get an actionable solution out of the sprint. At a bigger company, you’ll still need buy-in and it’s best to include the CEO, but if they can’t be there the whole time, you can bring them in at key decision-making moments.
  • Product manager: The PM (or whoever is filling this role) will likely need to implement the solution that comes out of the sprint.
  • User expert: The person on the team who has the most direct contact with customers often has great input, and can be the lead on user testing.
  • It’s also great to include:
    • Engineer
    • Marketer
    • Anybody else who’s interested
Schedule the User Study Before You Have Anything to Test

Recruit users and schedule the user studies for the last day of the sprint. This hard deadline, even though it’s artificial, will help you move faster and make tough decisions to focus your work throughout the sprint.

Put it on the Calendar

Clear everybody’s schedule for five consecutive days. It’s also very important to have a dedicated room for the duration of the sprint, usually a conference room with lots of whiteboards.

Gather the Ingredients

You don’t need anything fancy to run a sprint:

  • Sticky notes: I like the yellow 3×5 size.
  • Drawing pens: Any standard black or blue pen is probably fine. I like these, or you can get these for extra credit.
  • Whiteboards: If your war room doesn’t have a lot of whiteboard space in it, find another war room or some rolling whiteboards, or heck, get some IdeaPaint and get busy.
  • Whiteboard markers: I like to use these instead of Sharpies because they’re so versatile. Buy some good ones and be sure to have enough black markers for each person in the sprint.
  • Dot stickers: for voting. You want something small with uniform color. Post-It brand dots are great.
  • 8.5 × 11 blank copy paper: Nothing special, just have a pack of this on hand.
  • Time Timer Clock: Optional, but totally awesome, see here. I guarantee you’ll find it useful during the sprint, and probably during regular meetings afterward.
  • Snacks: You’ll need caffeine and food handy. Trail mix, bananas, and dark chocolate covered raisins have proved especially popular at our sprints, although it is possible that it’s just me eating all of it.
  • Sticky stuff: You’ll need to stick your drawings and storyboards on the wall. This removable gummy material is inexpensive and works great, with less fuss than tape.

Understand: Day 1

Full article.

Chances are that everyone involved in the sprint has different perspectives on the problem — and different information that might be helpful. The goal of the first day is to encourage everyone to share what they already know and develop a common understanding with the rest of the group. By starting at the beginning (even if some people are already familiar with the problem), it nudges the group into a beginner’s mindset and leads to fresh solutions.

Use exercises to help build understanding
  • We use the exercises below to get a ton of information on the table and quickly build understanding
  • You can do them in any order you want — and even omit the ones you don’t find valuable
  • Try capping each presentation or discussion at 10 minutes
  • This will keep the day moving and help everyone pay attention.
  • During the exercises, everyone in the sprint should be jotting down questions on sticky notes

"How Might We...": We use the “how might we” format to capture opportunities that might be interesting to explore. For example, “How might we build trust?” or “How might we figure out the user’s style?” Often, these end up being extremely useful in the next steps of the sprint.

Exercises:
  • Business opportunity — The CEO or product leader should walk the sprint team through the business opportunity and market.
  • Lightning demos — Look at competitors’ products. It can also be helpful to look at non-competitive products that solve a similar kind of problem in a different market.
  • Lay it out — Print out all the important screens in your product, lay it out, and walk through it as a user would.
  • Success metrics — How will you measure the success of this design? Now’s a great time to talk about success metrics. We like to use Kerry Rodden’s HEART framework.
  • Existing research — If you have user research for your product, that’s awesome, and you should be sure to go over it. If not, you should talk about whatever data you do know about your customers.
  • Team interviews — Knowledge about the problem is usually distributed across the company. We’ve found it very useful to go around interviewing people at the company who have specific expertise, whether that’s engineering or sales or customer service. (Customer service people often have incredibly valuable information about the problem.)
  • Analytics — Look at any data you have on feature usage, where customers drop off your site, conversion rates, etc.
Sketch the most important user story

As a group, use your common understanding to collaboratively map out the user story that’s important in this sprint. The facilitator (or another volunteer) should stand at the whiteboard and sketch the flow. This doesn’t have to be fancy to be useful. Here’s an example:

Sprint Story Diagram

Focus

You’ve got a rapidly approaching user study and there’s a good chance you can’t prototype everything you uncovered, so you’ll need to choose which ideas to move forward. Use your story diagram on the whiteboard to frame this discussion.

At the end of the sprint, when you show your prototype to users, what do you hope to learn? What do you need to design and prototype in order to learn those things? What you decide here will set your course for the rest of the sprint.

Diverge: Day 2

Full article.

Expect this step to take between two hours and all day.

I call this step “diverge” because when everyone is cranking out quick sketches. Everyone in the sprint will be working quietly and individually.

Dust off those old ideas

In my experience, some of the best ideas that come out of sprints were usually around before the sprint started. It’s not that they were bad ideas; they just hadn’t gotten enough love yet. The sprint gives you a chance to put all solutions on a level playing field. If you don’t bring out your pre-existing ideas, you do yourself a disservice.

Paper first

In a design sprint, we start designing on paper for a number of reasons:

  • It’s faster
  • Everyone can contribute (not just designers)
  • Nobody gets too attached to the ideas that are generated because they’re so quick and rough. We purposefully use thick markers to make sure nothing gets too precious.
  • Did I mention it’s faster?

Run the series of exercises below to guide everyone from note-taking through sketching and sharing. We’re not going to share any of the materials until we make storyboards so you’ll have plenty of time to polish those up.

1. Choose part of the problem

In Day 1 you drew a user story diagram. Look at it together as a team. If the user story is more than two steps long (and it probably is) you’re going to need to divide it up before you start sketching. This is as simple as finding natural chunks in the story and drawing a box around them, like this:

Story

Now decide which part to focus on first. It usually makes sense to have everybody in the sprint focus on the same part of the problem at once. If you take that approach, you’ll do one cycle for each part of the problem, with everybody collaborating on each part as you go. You can also divide and conquer — everybody picks a piece of the story they’re interested in and works on that. This way is usually faster, although it introduces the risk that people don’t think about the user story holistically.

2. Take notes (5 minutes)
  • At this point in the sprint, the whiteboards and walls are probably covered in diagrams, notes, and “how might we” sticky notes
  • This is your chance to reload that stuff into your brain
  • Everyone takes a piece of paper and jots down anything they think is useful
3. Mind map (10–15 minutes)

Now you’re going to add all the other ideas that are in your head, mix them with the notes you just took, and loosely organize them on paper. The mind map is going to be your “cheat sheet” you can use when you’re sketching UI ideas. Here’s an example:

Mind Map

4. Crazy Eights (5 minutes)
  • Everybody folds a blank sheet of paper in half four times, then unfolds it, so they get eight panels
  • Then you have 5 minutes total to draw eight sketches, one in each panel
  • Yes, you did the math correctly, that’s about 40 seconds per sketch, which is crazy... but it’s a great way to crank out variations of ideas quickly
  • And since these aren’t shared with the group, there’s no need to worry about making them pretty
  • If you get stuck, try repeating an earlier sketch with a small variation
  • For best results, do two rounds of Crazy Eights. On the second round, everyone will have the hang of it

Crazy 8s

5. Storyboard (10–20 minutes)

Now we’re going to make that user story diagram more concrete, and we’re going to make something that will be shared anonymously and critiqued by the group. The goal is to take the ideas we’ve generated so far and sketch an actual UI showing how a user would move through this part of the story — where they click, what info they enter, what they think, etc.

  • Start with a blank sheet of paper
  • Put 3 sticky notes on it
  • Each sticky note is one frame in the storyboard

I ask everybody to draw UI in the three frames of their storyboard showing a progression: first this, then that, then that.

Storyboard

There are three important storyboard rules:

  • Make it stand alone — Just like a real product, your drawing has to make sense by itself, without you there to pitch it. In the next steps, people will be looking at these, but you won’t have a chance to talk about your idea until the end.
  • Keep it anonymous — Don’t write your name on your drawing. You’ll want all ideas to start on a level playing field and it can be distracting to know which one was drawn by the CEO.
  • Give it a name — Come up with a catchy title for your idea. That makes it easier to discuss and compare later.

When you finish the storyboards, hang them on the wall with some sticky stuff. Pro tip: Hang them side by side (like an art museum) so people won’t have to crowd in too tight to see them.

6. Silent critique (5–10 minutes)

Give everybody a bunch of dot stickers. Then, without speaking, everybody looks at the different storyboards and puts a sticker on every idea or part of an idea they like. There are no limits to how many stickers you can use, and I don’t even prevent people who want to brazenly vote for their own ideas. By the end, you’ve got a kind of heat map, and some ideas are already standing out.

7. Three-minute critiques (3 minutes per idea)

Next, everybody gathers around the storyboards one at a time. First, people talk about what they liked, then we ask the person who drew it if we missed anything important. Usually the best, most popular ideas are the ones people can understand without an explanation, so the author of the storyboard often doesn’t have anything else to add. This process works far better than letting people explain their ideas first — which almost always uses up a lot of time.

Sometimes I like to do this step on a projector, especially if there are a lot of ideas to get through. I’ll take photos of each storyboard on my phone, upload them to Dropbox, put them in a Keynote file, then make notes about parts we like with outlines and text labels as we go through on the projector. This is easier for everyone to see, and you have a digital artifact of the ideas for later. The downside is the setup: count on 15 extra minutes to capture and upload photos.

8. Super vote (5 minutes)

Once we’ve looked at all the ideas, everybody gets one or two “special” stickers (which can be the same dot stickers from before with a pen mark on them). These are “super votes” for the ideas you think are the very best. Between the original heat map and these super votes, it’s very easy to see which are the strongest concepts.

The super votes offer a unique way to tweak the process to reflect the decision-making structure of your team or company. Does your CEO make all final decisions about the product? If that’s the case, be honest about it, and give her three super votes and everybody else one. Or maybe it’s a UX director or maybe a tandem of product and design who call the shots. The simple rule is to give the deciders extra votes.

Repeat

Now it’s back to the first step to start the whole cycle over again. (Don’t worry, it gets easier with every repetition.) If you split up the user story last time, it may be time to move on to another chunk. Often when I’m running a sprint I’ll realize at this point that our scope was too large, and we should just double down and keep working on the same section. Either way, the end of a cycle is a good time to take a few minutes and carefully decide where to focus next.

Expect a team to be able to do this cycle two or three times in a day before getting burned out.

Day 3: Decide (Storyboarding)

Full article

You can’t build and test everything. Today we’ll look at how to decide which solutions to flesh out, and how you’ll fit them together into something you can rapidly test with users to learn what’s working and what isn’t.

Combat the group effect

Companies and teams have a natural way that they make decisions - but in a sprint, the group effect can cause decision-makers to behave more democratically than they do in real life. To combat this effect, the facilitator often has to draw out the decision-maker to give their honest, true opinion.

IF YOU AREN’T CONSCIENTIOUS ABOUT BRINGING THE DECIDER IN NOW, YOU’LL HAVE A PROBLEM LATER.

One method is giving "super votes" to the deciders during design critiques, which you did in day 2. But most of the time, there are no special techniques. You just have to be blunt.

Search for conflicts

The first thing I like to do in this phase of a sprint is comb through storyboards from the previous day looking for conflicts. A conflict is a place where there are two or more different approaches to solving the same problem. Conflicting approaches are super helpful, because they illuminate the choices for your product.

Every time you find one, write it down. I like to put the topic and solutions on sticky notes, like this:

Conflicts

Best shot or battle royale?

You have two basic options for what kind of user study you’re going to run at the end of your sprint. You can prototype several different approaches and test them against one another (the "battle royale") or you can go with a single prototype (the "best shot").

CONFLICTING APPROACHES ARE SUPER HELPFUL, BECAUSE THEY ILLUMINATE THE CHOICES FOR YOUR PRODUCT.

The "battle royale" works well for newer spaces where there really aren’t many conventions, and you need to figure which one is going to work best for the user. The disadvantage is that it takes more time, and your testers may run out of patience before you get all of the information you’d like to have from them. You may have to bring in more participants and run more studies.

On the upside, the results of a "battle royale" can be very surprising. When working with startups, I’ve often seen a dark-horse design turn out to be the strongest in user studies. When that happens, we thank our lucky stars we didn’t "best shot" it, or we never would have known.

You may also do some kind of hybrid. Occasionally, if you choose the "best shot" approach, you’ll get into testing and find that something’s really not working in your prototype, and you need to go back and have a "battle royale" over that specific feature.

So how do you know which to pick? Start with a gut check: If everyone is excited about one option, you may be ready for a "best shot." But if it feels more like you’re sitting there and scratching your heads about what to do next—or else you want to throttle each other because those fools just won’t agree with you—well, you may need a "battle royale."

Test Your Assumptions

What else should you test in your user study? Listing out your underlying assumptions is a good way to revisit the big picture, especially when you’ve been heads down in a sprint for a few days.

Some of those assumptions might be about the users (example: "Users are willing to upload a profile photo"), some about the business ("the designer-with-glasses-and-beard market is large enough to support our product"), some about technology ("we can automatically cluster profile photos by beard shape"), and maybe even some about messaging ("people will still find beard jokes to be amusing, even for the third time in a single paragraph").

I can tell you that last assumption is false right now, but for most others, you’re going to need some kind of research. Guess what? You can test a bunch of them by showing a prototype to users.

For example, if you have a big assumption that users will be comfortable sharing private data in your product, you may want to pick the most aggressive sharing defaults you can think of to prototype. When you show it to users, you’ll find out pretty quickly whether your assumption was correct.

Try to come up with a way to test all your assumptions, either in the user study or in some other parallel task that can start right away (e.g., ask the engineers to spend a few hours hacking at that beard-clustering algorithm). If you can’t test every assumption now, keep a list for next time.

Whiteboard the User Story

Now we’re going to make a storyboard that shows exactly how the user will step through your prototype, click by click. This storyboard will become the spec for building the prototype. This is an activity that the group does together — it’s actually the last group step before you break for prototyping.

Story Board

  1. Start by drawing a big grid on the whiteboard — each cell should be about as large as two sheets of copy paper, and for most sprints, you’ll cover one or two whole whiteboards with your grid.
  2. The idea is to draw a comic book that tells a story starting when the user opens the prototype and ending when they complete all necessary tasks.
  3. In each comic book frame, you’ll draw a single action — whether it’s a pointer clicking on a button, text being entered, or a stick-figure user doing something in real life. 1. You don’t have to worry about layout or design in great detail, but you do have to think through every action that takes place in the story.
  4. Drawing the storyboard is hard work, and you’ll want to facilitate carefully.
  5. Get one person to draw, but don’t make them figure everything out on their own.
  6. The group should be engaged and discussing what happens next and giving that brave soul holding the whiteboard marker as much help as possible.

When you begin drawing, imagine you’re framing the prototype for your user study participant. How will they get to your product? What will they be trying to do when they get there? That’ll help you figure out whether the first frame of your comic book is an email or a Google search or an advertisement or the App Store or whatever—and hopefully the story will flow easily from there, following the outline you laid out in day 1.

Keep the Gloves Off

As you storyboard, there will be lots of small decisions to make that didn’t come up earlier in the day. That’s expected, since you’re working at a finer level of detail now.

The facilitator has to work hard here to not let people be too nice. You don’t want design by committee.

If there’s a good argument going, don’t try to find middle ground or make people agree. Help the team place a bet on one of the opposing solutions and keep the other in your back pocket if it fails. Call on the CEO to make a tough call when needed. If both solutions are viable, you may want to opt for a "battle royale" — just don’t use it as an excuse to avoid decisions.

Prototype: Day 4

Full article

On day 2 you drew concept sketches. On day 3, you made a plan and a storyboard for your prototype. Now it’s day 4 and the clock is ticking. You’re going to create a real-looking version of yesterday’s storyboard and show it to users tomorrow.

What your prototype should look like

Quite simply, a prototype is anything a person can look at and respond to. A prototype doesn’t usually have to be very complex in order to learn what you need to know.

Make it minimally real You’ll probably be amazed at how much real feedback a user can give you on a slide deck of mockups that aren’t even pixel-perfect.

They can tell you what they understand about your product — and what they don’t. They’ll tell you what they expect things to do (“I’d click here because I’d want to see a list of your customers…”), and when they get confused.

You’ll also learn things that metrics alone can’t tell you, in particular why users do the things they do, rather than just what they do. And you’ll learn much faster than if you had waited to build something.

Write real text While it can be tempting to use “lorem ipsum” when you’re building a quick prototype, don’t do it — always write real text for your prototype.

Why? First, user interfaces are mostly text. Punting on the text might save you time, but it won’t get you closer to solving your problem. Plus, in design sprints we’re often figuring out how to explain things and testing whether users understand. Lorem ipsum skips all of that. When you use dummy text, you avoid tough decisions and limit how much you can learn. (37signals has a great essay on using real words in their book Getting Real.)

Keynote versus code Occasionally you’ll need to write some code for your prototype. This can be really valuable when you’re testing assumptions relating to data quality or personal content like email. But nine times out of ten we find that Keynote is sufficient. More on this later.

Here are some examples We have created dozens of minimally real prototypes for the startups in our portfolio. Here are a few examples, covering the spectrum from grayscale and low-fidelity to nearly pixel-perfect.

Examples of prototypes we've made:

Keynote is the world’s best prototyping tool

If you’re building a product for desktop, mobile, or iPad, the fastest and easiest prototyping tool you can use is Keynote. I’ve spent my career working with fancy tools like Photoshop and Illustrator and Fireworks. (OK, Fireworks might not be fancy, but it’s almost certainly a tool.)

Here’s why Keynote wins for prototyping:

  • It’s fast.
  • It’s easy to make things look pretty good…
  • But it’s impossible to make things look perfect, so you don’t get too precious.
  • Anybody can quickly learn to use it; not just designers.
  • The slideshow format is a natural fit for story-based design.
  • It only costs $20.
  • The animated transitions are a lightning-fast way to make your prototype look way more real than it really is.
  • If it’s a mobile prototype, you can export a PDF and open it on your device with reasonable results.
  • When you’re done, you not only have a prototype, you have a presentation, like, for free!
  • After your user study, when you learn all of the problems in your design, it’ll take minutes to make changes instead of hours or days.

The final Keynote prototype might be linear or it might have a few links that allow the user to click around. It might be a grayscale wireframe or it might be somewhat polished. Either way, these make for good prototypes. They’re going to be realistic enough that your user study participants will forget they aren’t clicking on a real product.

Keynotopia templates make it even faster
Here’s our favorite secret prototyping weapon: Keynotopia. It’s a template with buttons, menus, and all kinds of plug-and-play elements that you can just drop into your prototype without any design skills at all. This makes it easier for CEOs and marketing directors to dive in and make something to communicate what they’re thinking. And it’s available for PowerPoint too, if that’s how you roll.

What to do in day 4

Here’s a rough schedule for your day of prototyping. Expect to spend about an hour in the morning to review and make your plan.

Divide and conquer
What if you have too much to design and not enough time? Work together, baby.

Take a look at your team. How many people can help in Keynote? Chances are good that almost everyone can — in our sprints, we frequently have engineers, PMs, and CEOs cranking out good work in Keynote. Remember that everybody’s work doesn’t have to be genius level. A designer can always clean it up afterwards, but it’s usually faster to clean up a slide after someone else has put the building blocks in place.

Now take a look at yesterday’s storyboard — or, if you’re doing a “battle royale,” storyboards plural. Break it into chunks and assign them out. It’s probably obvious, but it’s helpful to think about where different people have expertise, or where it’s best to spend your designer’s time, or who moves the fastest.

You probably want to divvy up the best of the storyboard sketches on paper, too. Those are often the blueprint for many of the mockups you’ll have to make, and they can save you a bunch of time.

Assign one person to be the stitcher. The stitcher’s job is to take everybody else’s work and put it into one cohesive flow. If you only have one designer, or if one person is really fast at Keynote, she might be a good candidate. If you’re doing a “battle royale,” you may decide to have one stitcher for each competing version.

Breaking up the design work and stitching it back together probably doesn’t sound like the way great design is done. And it’s not. Realistically you’ll get better results if you’ve focused enough that your designers can do all the work here. But if you need it, dividing and conquering is an incredibly fast way to build a good-enough-to-learn-from prototype.

Build an asset library
Another way to increase efficiency is to take a few moments at the beginning of the day to build a template slide deck. Include anything that everyone will need — screenshots, user avatars, logos, formatted text; whatever you think might help. And don’t forget to include a browser bar at the top for realism. You don’t want to go through 99 slides at the end, scooting each one down to make room.

Use a timer to maintain focus
Once again, the Time Timer is your friend — although if you don’t have one, I guess some other kind of clock might work. We’ll often say something like: “Let’s work heads-down for the next hour and then we’ll take a break.” Timeboxing can make the project feel less daunting, and reduce the impulse to check your email and burn up valuable make time.

Appoint an email sheriff
It’s sometimes helpful to appoint someone — usually the facilitator — as the “email sheriff.” This person’s job is to publicly shame anyone who is checking their email (or surfing the web or whatever time-wasting tactic you prefer). It’s usually enough to just make a big deal about saying that someone is going to do this… the threat alone practically makes the job unnecessary.

Lightning critique
If you have a bunch of people working in parallel, it might be useful to do a quick critique midday. It’s useful to ensure consistency and get outside eyes on your design. But critiques can eat up a lot of time if you aren’t careful. Just like on day 2, you’re going to want to limit the time spent talking about each design. I recommend 5 minutes.

You’re also going to want to prevent people from trying to redesign somebody else’s work — they might have a great idea, but there just isn’t time in this part of the sprint. Instead, use the lightning critique as an opportunity to raise questions and criticisms, but make it clear that the person who designed those components is going to be responsible for figuring out the solution, not the group.

Review with an outsider
Schedule 30 minutes with someone who is not doing design work today. It might be your user researcher if you’re lucky enough to have one, or it might be an executive — seems like everybody’s got one or two of those hanging around. The outside eyes will help prevent you from going too far down any groupthink rabbit holes. Just make sure to do this early enough in the day that you have plenty of time to respond to the feedback afterward!

Pointers, text, and other final touches
If you’re creating a linear prototype with no links, which is the very fastest thing to do, it’s a good idea to draw a mouse pointer and add extra slides showing where the user would click, and extra slides when text is entered. It takes only a little extra time but makes the user study much more realistic.

Other crucial details:

  • Check for consistency and typos, especially if you’ve got some made up user data. Don’t let it start out as Sally and end up as Suzy. That stuff is distracting in a user study.
  • Try to make any content current and relevant. If you’re running the study in Seattle, and you need to show a newspaper, show the Seattle Times, not the Milwaukee Journal Sentinel.
  • If you get stuck, remember what you’re trying to learn — don’t waste 30 minutes tweaking a button style if you’re doing a study about whether people understand the value proposition.
When Keynote isn’t enough

Nine times out of ten, you can learn everything you need to know in a user study with a click-through of mockups.

But sometimes you can’t avoid it. When you’re testing big assumptions involving data quality, or when users need to interact with their own stuff (email, docs, contacts, etc) to really understand the product, there’s no substitute for a code prototype. In this case, get someone with engineering chops to help.

Just make sure everybody, especially the engineer, knows that this code will be throwaway. That’s really important: The. Code. Is. THROWAWAY. We gotta move fast, so don’t get attached. Like the Keynote prototype, it just has to look sort of real, it doesn’t have to be anywhere near perfect.

It’s OK if you’re not satisfied, Michelangelo

It’s better to be done with something good enough than to be half-finished with a masterpiece. Remember that the goal is to learn from the user study tomorrow, not to have everything perfectly figured out and finished.

Validate (Day 5)

Full article

Today, we test our prototype and learn which ideas worked, which didn’t, and what to do next.

Today you’re going to be running a user study, showing your prototype to 4–6 real humans. By “real humans,” I mean people who don’t work at your company — more specifically, people who you’d like to have as users.

Wait, what if I don’t know how to run a user study?

Google wrote a detailed series on how to run a 4-day research sprint!

Before the first session: List your key questions

Start at the end. When the day is over, you’ll want a concrete list of ways to improve your prototype. Talking about your game plan before the study starts will help you get there.

The interviewer and the observers should make a list of the key questions for the day. Here are a few tips:

  • Review your conflicts and assumptions. Those assumptions you flagged as testable with a user study? Now’s the time!
  • Are you testing multiple prototypes in a battle royale? If so, be sure the interviewer understands the differences between the two versions, so they can ask the right questions.
  • Consider showing participants some real products for comparison — they’re like free prototypes!
  • What else do you want to see through your users’ eyes? Today’s study is not a usability test — you have a chance to find out how your users understand your product, what competitors or substitutes they use, and more. Think beyond the prototype and you are guaranteed to get some unexpected insights.
Set up the observation room

Everybody who participated in the sprint should be in the room. There’s no substitute for watching real humans use your product, and this is a golden opportunity to do it!

I like to reserve two rooms for the day: one for the user interviews, and another where the sprint team can watch live video of the sessions and take notes together. It’s kind of like a Battlestar Galactica marathon, only instead of comparing guesses on who’s a Cylon, you’ll be comparing guesses on which parts of your design are going to go up in flames. It’s actually pretty fun.

Test the A/V ahead of time

Setting up live video for observation shouldn’t be too hard — at Google Ventures, we’ve had success with WebEx, GoToMeeting, Apple Airplay, and Google Hangouts.

Just be sure to test your video before you start the study — ideally the night before and the morning of — because nothing’s worse than putting in all this work and not getting to watch the outcome. OK, there are a few worse things. But you see what I mean.

A few more A/V tips:

  • A conferencing mic can provide a huge improvement in sound quality.
  • Don’t forget to mute the audio in the observation room!
  • Quickly re-test the A/V between each session. Seriously, stuff gets wacky.
Don’t diss the user

If you see people struggle to understand the prototype, keep in mind that we’re testing your design — not the participant. If they don’t get it, it’s not because they’re dumb. It’s because you haven’t nailed the design yet.

Make it clear in the observation room that it’s not OK to diss the participant. It’s tough to wade through a prototype while people are watching. You owe the participants your respect.

Every observer takes notes

Everybody should take notes on things they see during the interviews: good, bad, and other. Insist on paper note-taking — it’s best to keep laptops closed, lest you lose your fellow observers to email.

Designate a court reporter

One person can use a laptop in the observation room, but they have a tough job: the court reporter. They’re responsible for typing a word-for-word transcript of the interview in real time. Assign a different court reporter for each session — it’s cruel and unusual to make the same person do it all day.

It’s kind of a pain, but these notes become an incredibly valuable reference after the study, and a text document is a heck of a lot faster to scan for quotes and reactions than a video recording. Often times, we don’t record the study and rely on these notes instead.

Make a scoreboard

Clear one big whiteboard to collect the group’s notes. Make a column for each participant and a row for each part of the interview (e.g. background, first prototype, second prototype, etc).

At the end of each session, write down the highlights from everybody’s notes — you can double check any questions against the transcript. I like to color code: green for things that went well, red for problems, and black for everything else. It makes it easier to find patterns at the end of the day.

Score board

Observing humans: the emotional roller coaster

I’ve never been in mission control when one of those rover things lands on Mars (NASA isn’t returning my calls). But I imagine it’s pretty similar to the atmosphere in the observation room of a user study. There’s tension and excitement and nobody wants to be the one who designed the doohickey that blows up. Here’s what you might expect to feel throughout the day.

First session: “We’re geniuses!” or “We’re idiots!”
The first user may love your prototype. Or they may hate it. Either way, take a deep breath. People are different, and I offer you an ironclad guarantee that not everyone will react in exactly the same way.

Regardless of how that first interview goes, resist the urge to make changes to your prototype. Unless it’s something simple like a typo or broken link, you risk “fixing” something that the next four users would have actually liked.

Sessions 2–4: “Oh, this is complicated…”
It’s not uncommon for the second and third participants to have dramatically different feedback, which means you’re going to feel a little confused. Just sit tight and keep taking notes. It’s still too soon to tell.

Actually, that’s not entirely true. Sometimes you know for sure that part of your prototype is rotten after two or three interviews, and watching more people suffer through it is punishment for all involved. If everyone agrees that something is way off, talk to the interviewer in between sessions and ask them to guide people around that part of the prototype.

Studies 5-6: “There’s a pattern!”
After the final interviews it’s easy to see the big patterns, but it’s worth double checking the notes. Go to the scoreboard and look for things that showed up two or more times. Mark good stuff with a big green dot, and bad stuff with red.

Now make two lists on the whiteboard: “things that work” and “problems to solve.” These are your top-line findings. The CEO or decider for the project should bless that list before you leave the room.

The sprint is complete. Take a deep breath.

How to start your next sprint

The vast majority of these studies ends with mixed results. Some of your solutions work, and some don’t. The outcome usually falls in one of three buckets:

A. Most stuff worked
This is pretty uncommon for the first sprint on a project, but if it happens to you, everyone on the team is probably on the same page about the fixes and tweaks you need to make.

What to do next: Tune your existing prototype and keep going. Try starting your next sprint at step 3 (decide).

B. Some big questions
The most common outcome after a user study is a mixed bag: a few hits, a few tweaks, and a couple of real head-scratchers. Fortunately Keynote prototypes are easy to change, and as long as some parts of your design are solid, you can probably build on what you have.

What to do next: You can move fast on the tweaks, but you’ll want to come up with multiple solutions for the bigger problems. Start your next sprint at step 2 (diverge).

C. Everything exploded
I’ve seen a lot of of my designs go up in flames. It’s OK. You learned that something didn’t work, and it only took you a few hours to build it in Keynote. This is great progress and — relative to building and launching for real — very cheap progress at that. Think what would have happened if you’d spent weeks or months implementing this solution!

What to do next: Start your next sprint back at the drawing board with step 1 (understand). (Hint: the results of this study are perfect fodder for reviewing and building understanding as a group.)

The next sprint will be easier

You may feel tired. Actually, if you don’t feel tired, you probably didn’t do the sprint properly. It’s an intense week. But you’ve built up a tremendous head of steam — even if everything exploded, you now have a much better understanding of the problem and possible solutions.

At the end of a sprint, CEOs often tell us they appreciate that they have a clear list of what to do next. Now that you’ve built up all that knowledge and momentum — not to mention a prototype — you’ll find you can do the next sprint more quickly, or with less effort, as long as you do it right away. Don’t let more than another workday go by before you jump back into it. If this problem is important, you’ve got to bear down and finish it off before people get distracted.