Hiring Product Managers: The Interview07.27.11
After my last post on "Hiring Product Managers: 5 Things to Look for" [link], my friend Tom wrote in and said, "great post nelson! can you write a future post focusing on different interview questions and strategies for figuring out who has these qualities you look for?"
So here's what I do in the interview. I start from a place where I ask myself two things:
1. Who is this person? (I'm trying to build a profile of the candidate.)
2. Is this a hire or no hire?
So in terms of "who is this person?" -- I start with 3 questions and pivot from there with the express purpose of trying to build a profile of the type of candidate they are. Do they have good business instincts? Will they ship? What's their product sensibility? Will I like working with them? Do they work hard?
In terms of "Is this a hire or no hire?" -- I try not to wimp out on interviews. When I say wimp out, it's basically writing in and saying, "This person is fine if you want to hire." -- basically abdicating responsibility. I really push myself to have an opinion, if not a strong opinion, of whether or not this person should be hired or not. So I push myself in the interview to really refine and critique my own opinion of the person. Do I want this person at the company or not and do I have specific examples that I can articulate in the interview feedback that is useful to others?
Let me back up for a second. For me, this all starts with the resume. Let me give 3 (made up) examples and discuss what I see in each of the examples. For simplicity's sake -- I'll just take undergrads graduating -- so no work experience.
Candidate A: Princeton, majored in History, 3.2 GPA, no major accomplishments, internship at Morgan Stanley
Candidate B: Penn St., major in computer science, summa cum laude, worked on failed startup (website still active / I can view)
Candidate C: Northwestern, majored in EE, 3.7 GPA, active in activities in college, internship in IT consulting at Accenture
Again, I'm starting from the perspective of trying to build a profile of the candidate. When I interview them, I'll be asking questions to confirm or change my profile. I am not dogmatic about my profile. For example, I had a boss that literally said if there was 1 typo on a candidate's resume, he would throw the resume out. I would not. However, what I would do is look for confirmation that this candidate is sloppy. 2 or 3 of those -- sure. But not after one.
Candidate A: Here's the problem with the resume. Princeton is a good school but History is not a hard major. 3.2 GPA is not a great GPA. Grade inflation is terrible at top schools. There should be a reason why they didn't excel in a major like that (e.g. large work study, heavy into sports, etc.) -- they need to show tangible things they've done. This is the same even for people working. I care about where they worked, but I care probably even more what they did once they were there. It's not enough to accumulate strong brands. I would actually ding this candidate at the resume stage.
Candidate B: I would bring this person in any day of the week. Penn St. isn't a great school, but the level of difficulty to rise to the top of any giant school is very high -- there's just a ton of people. Obviously the assumption of, "They went to Penn St. because they couldn't go to Harvard, MIT, Stanford, etc." -- is not a terribly unreasonable assumption. However, for someone that ends up at the top of their class -- the likely answer to that could be -- they slacked off in high school and then stepped it up in college, their family couldn't afford the other schools and they had a scholarship to Penn St., etc. The fact that they worked on something entrepreneurial is great. But broadly, anyone who does excellent at a large state school (or even a smaller school with some name recognition) -- I would interview those people over people who did average / above average at the top schools.
Candidate C: I see a ton of these types of candidates. They went to a good school -- let's say top 20, but not top 5. They have a solid GPA but didn't necessarily finish summa or magna. They got a great internship or job (Accenture) but maybe not the top (e.g. McKinsey, Bain, or BCG.) Not to broadly overcharacterize -- but these folks generally fall into the good but not great PMs. That's fine -- good PMs are hard to find too. But I would like to see (1) place where they particularly excelled -- created or started something that's maybe quintessentially them. If you're looking for great -- you're betting that they'll do something that's atypical to their background, that's all. Not to say they won't do it, but they don't have a history of it.
I'm doing this type of analysis throughout the entire resume -- every line, both in terms of the actual content and how they word it -- I'm asking myself, "Who is this person?" Is this the type of person who talks about doing things and doesn't get anything done? (A big tell is they talk about all the areas they "managed" and not about what actually shipped.) Is this potentially a great team player but not someone who would be a great, independent leader of a cross-functional team? (A big tell is they've checked all the right boxes -- the right schools, maybe an MBA, good jobs / internships -- but nothing about their resume stands out.)
So onto the interview. In my previous post, I talked about 5 things to look for: project manager, good intuition, technical, ships / bias for action, and product ideas / vision.
I don't ask 5 different questions for each area. What I do instead to build this profile is to use 3 simple questions to lead me throughout the interview, and then pivot off those questions depending on the responses and my sense of where they might be weak or strong. So the first question:
Q. Tell me a little about yourself.
I ask this question to put the candidate at ease. Interviews are stressful and there's a lot at stake. However, I also gain a lot of information about them. The first thing I'm looking for is social intelligence. Do they have a good sense of how to answer this question? The biggest problem responses I usually see are either a) too long or b) rambling / no cohesion. They should be able to know that I'm looking for a relatively concise answer that doesn't regurgitate their resume that's coherent.
Let's say they give me a rambling response. I'm then thinking, "This person might be a mess in terms of managing things." I'm going to look for more signs of that. Let's say they go on for close to 10 minutes. I generally won't stop them. I'm then looking for signs that they will be difficult to work with (e.g. doesn't listen; listening is also non-verbal)
Ok -- so now the warm up is over. My next question is:
Q. Choose a website / web app / Internet company that you think is particularly innovative. Tell me why you think so. Don't choose either the company you're interviewing for or a company you worked for.
There are a few keys to the question. First, I don't let them choose the company they're interviewing for. The reason is because it's really difficult for me to get a good read on the quality of their answer. I have a lot of domain knowledge about the company (because I either work there or I work with them in some capacity) and frankly, the breadth of product ideas that we've covered internally is massive. Also, they prepared (or should have prepared) for some variant of this question. For similar reasons, I don't want them to discuss a company that they previously worked for -- the balance of knowledge is too skewed in the other direction. So I ask them to pick another company and we go from there.
The companies frequently presented are large, well-known tech companies: Facebook, eBay, amazon.com, etc. I usually handicap them a little if they do this since that's not terribly inspiring -- but if they have great ideas coming off of it, then no problem. Sometimes, they pick companies that I don't think are particularly innovative. That's fine. But they have to convince me they're innovative. Sometimes, they pick super innovative (but unknown) companies -- those are usually the most promising interviewees. Which company they pick immediately gives me a lot of information about them.
I then ask them if they were the PM on that product, what would they do? The first thing I'm looking is to see where they gravitate. Most interviewees will gravitate towards (minor) product improvements. Some will gravitate towards business ideas. Let's say they have a ton of product ideas and some of them are quite good. I'm just going with my gut looking for that moment where I hear an idea and think, "Oh, that's clever." -- so if we're discussing Twitter and the candidate has a bunch of ideas immediately that I like, that's a great sign. But at that point, I'll then think, "Ok -- this person knows product. Do they know business?" Then I might ask, "Let's say you were tasked with tripling Twitter's user base. How would you do that?" or "Let's say you needed to grow Twitter's revenue to $1B in 3 years. How would you do that?" Most of the time, product folks will just keep moving back towards product and so I'll try to push them towards more pure business questions.
The opposite is also true. If folks come in and just answer with straight business answers (let's say they have lots of great marketing ideas) -- I'll flat out say, "Give me 3 ideas you would do to improve Twitter's interface." It's that specific. I want to see if they can handle the spectrum of issues that will be thrown at them. This question generally gives me all the information I need with respect to: product ideas / vision, some on intuition, and technical. It tells me about intuition, for example, because if they give me an idea that's an ok idea and they think it's great -- well, that's not so good. Or even revenue driving ideas -- it's ok if they don't necessarily have ideas right off the bat, but they need to sense that's the case. Or they need to sense that a particular idea will drive some revenue but maybe not $1B. The technical filter is a little trickier. The way I work it is I try and sense their technical knowledge. Some of this is off their resume (if they have a CS degree from MIT, I'm generally not that concerned), but a lot of it is how they talk about products. The tip-off here is usually that there's no technical depth in any of their areas. They don't have a technical degree. The companies they worked for are big names and they worked in a non-PM capacity. The companies they talk about -- they're also big companies and their ideas are more like, "amazon.com should compete with Fresh Direct" rather than "here's what I would do with Amazon Web Services." So in this case, if I sense this, I would then take an aspect of the company we've been talking about and ask them how they would architect it. For example, let's stay with Twitter. I might then pull up Twitter and then ask, "Let's look at the 'Who to follow' feature." How would you architect this feature? They should be able to architect it on a high level / understand how it works or could work -- even if they're just using logic, they should be able to do it.
I generally stick with all product on this question -- and if I do so, that's when I try and break out some more general business questions. I should note that in the past, I've sometimes been tempted to skip this section -- I'll meet someone with a good/great resume, they have great product intuition -- seem really buttoned up, but a number of them have crashed on this question. So I do use this question frequently. Here's the question I typically ask:
Q. Estimate for me Wal-Mart's annual revenue in the United States. I want you to come up with a number.
I constrain this question for a few reasons. Wal-Mart is sufficiently known that they should be able to have an intuitive sense of what they do, what they sell, etc. I'm looking for annual revenue -- so it's the broadest measure and one where people should have an idea (whereas if I asked for profit, that's much more complex because if you don't know retail margins, they might be tempted to try and do a blended margin which will chew up enormous amounts of time, etc.) Then I constrain to the U.S. just because global #s introduces too much complexity for something off the top of their head. I say, "I want you to come up with a number." because I want them to come up with a number. When I didn't say that, 50%+ of folks wouldn't actually estimate for me the revenue, they would just talk about things they would consider. That's useless. I'm actually looking for their ability to model. Modeling tells me a ton about them. It tells me how they organize their thoughts. It tells me their intuition for numbers. Is a $20 average selling price reasonable or is a $50 average selling price reasonable? It tells me how careful or careless they are (do they check their numbers? do they try and solve the problems on a micro and a macro level?) I can also see them work. I can see how they lay out their calculations. I can see their reactions to their assumptions and calculations. If they come up with $10B for the #, does that strike them as intuitively wrong?
This problem also puts some candidates under stress. I'm not looking for them to be under stress and it's not why I ask it either. I suspect the reason for it is because financial modeling -- especially for these types of candidates -- it's rare. However, this problem is fair. It can be solved with logic. Smart people can do this problem -- or at least do a competent job. The candidate doesn't have to do a great job -- I'm just looking for data to confirm or deny certain assumptions I have about the candidate. But I've had candidates that have REFUSED to do this problem. I've had candidates who have refused to do this exercise because they could just "look at Wal-Mart's revenues." True -- but let's say you have to estimate Wal-Mart's revenue next year. Or Wal-Mart's revenue for a new territory. Or the revenue for a new product? You can't look those things up. More importantly, that's a terrible attitude to have in an interview.
So by now, 3 areas are generally clear to me: product intuition, technical, and product ideas / vision. What about project management and ships / bias for action? Let me tackle ships / bias for action. I'm looking for the softer side of how they present. I'm looking at how their resume reads. Let's take a great marketer. A great marketer can actually say, "You know this campaign? The one on JetBlue? Or the one on these billboards? Or the ones that played during the Super Bowl? I worked on those, here was my role, and here's why we did what we did." The bad marketers are the ones that say, "I managed $X in marketing spend which was divided among these tactics and I had this many people in my team." See the difference? One does, the other talks about what they managed. This is the same for product or frankly, anything on a resume. When I ask them to tell me a little about themselves -- it's the same thing. How do they present it? Are they presenting tangible, discrete items?
So now we're down to "project management." I don't have a specific question here. It's just a gut feeling. It's the equivalent of asking myself, "If this person had to plan a party, could they do it?" "If this person had to renovate their house, could they do it?" It's not rocket science. But it does require someone to be organized, to follow up on things, and be able to work with other people. The tricky thing about product management is that it's a very left brain / right brain dynamic. The project management aspect is very analytical -- very left brain. Great PMs have the creativity / innovative spark -- that's very right brain. They're not necessarily exclusive from each other -- but it's hard to find people who are great at both. So if you find someone who is really right brain -- great ideas, really creative -- that's when I start to try and hone in and ask myself, "Can this person lead a team?" Sometimes the answer is, "They might not be great at it, but they'll be adequate." Messy launches are ok. Launches where people want to kill each other are bad -- but sometimes acceptable if something great comes out. It depends on the type of company you have and your tolerance for messiness. A friend of mine is a *great* PM. There was a lot of criticism of him. The criticism generally was along the lines of, "He leaves a large wake in his path." Good! He gets a ton down, burns down bridges when he needs to, and shipped some amazing stuff. Ideally -- you have great products and no wake. But that's really rare.
Before I close, here are some of my personal interview / hiring traps:
1. Speaks Well
This is generally the MBA disease. MBAs are quite polished -- they speak well, they've been prepped -- they do the right things right. Here's the problem. I've found too often that they don't ship. So again, scrutinize their resume and how they talk about accomplishments. Do they have specific, concrete, discrete items that they can point to as things they shipped? This should permeate their life.
2. Not Smart Enough
These are the folks who are pleasant, you like having them around, and are probably great product managers. I'm not saying "not smart enough" to be mean -- but this is the person that you do not trust to make decisions independent of you. Here are the 2 questions you need to ask yourself. "Do I trust this person to go to a meeting with a senior executive and represent me?" and "Do I trust this person to make product decisions without any input / review from me?" The answer to both questions must be yes.
3. Can this person get it done?
I've run into quite a few PMs who are smart, have good product intuition, and have some accomplishments on their resume. Here's the problem. In a hyper-aggressive environment where resources need to be battled over and powerful people fight for fun -- they will get eaten alive. For sport. This is just a gut feel, but the operative question here is to think of the meanest, nastiest group within the company -- would you be comfortable sending them into that meeting and them not only emerging, but also having made significant and tangible progress on their product? Will they be able to solve problems without always escalating? Or will they emerge from that meeting having been fully co-opted? PMs are wired to ship. So whether it's escalating, baking cookies, making friends, being buttoned up, taking as little time as possible -- everything is on the plate to get things done. They have to be able to deal with all situations. This situation often falls under the "too junior and won't grow fast enough" category.
One last thing. I always write my interview feedback up. I don't like discussing (initially) in person. Especially on interview loops, people have heavy biases (usually to hire) in terms of the candidate. I want to write my feedback down -- with specific examples of why I feel the way I do -- and then let that stand on its own. I love to have the conversation where the hiring manager comes back and says, "You have XYZ concern. We disagree. Here are our data points why. Let's discuss." I have *never* been approached this way. The way I get approached is, "I like this person. I want to hire this person. Will you change your feedback?" The answer is no. I will not change my feedback without new information. Show me new information. If you want to hire this person over my objections -- go right ahead. But don't delude yourself if you don't actually disagree with my objections. Get new data. That's the standard I would encourage when hiring PMs. Get as much data any way you can -- resume, interview, take them to lunch, etc. -- form a profile of who they are. Have an opinion of who they are and how they will do in each area that's important to you. Then each new piece of information confirms or denies your assumptions.