Back To Blog
Product: Quality Improvements vs. New Features
11.24.11

A startup I work with is in private alpha of their product. As we were hitting the site, coming up with different ideas, etc. -- the board chair asked me a simple question, "How do you prioritize quality improvements versus new features?" 

I want to open this post on Steve Jobs. Walter Isaacson has written a great biography on him [link] and in it, it profiles Jobs in terms of how he works. The particular aspect of his personality / work style that has been lauded has been has relentless pursuit of perfection. Take the MacBook Pro -- an already amazing and beautiful machine but then to take it to the next level, they created a unibody enclosure. If you're not familiar with what I'm talking about, check it out on Apple's website and watch the video where Jonathan Ive walks through it [link].

The story of Jobs and his work style essentially gets reduced to well-meaning individuals saying they can't do something, or miss many important details, or are pushed just to get something out the door -- and Jobs pounds the table, fires people, calls people stupid and insists on revision after revision until the product is perfect. A site to behold. The iPad. The iPod. The iPhone. The Apple Store. This is true to a large extent but I think is also slightly dangerous thinking. The first reason is Jobs had at his disposal, essentially unlimited resources. Revisions, delays, many many prototypes -- are all very costly. Especially within a large company where you have high fixed costs (salaries). The second reason is that the greatness of what we know Jobs primarily for (let's say Jobs from 2000-2010) ignores a vast history of Jobs' work. Jobs has always had a brilliance about him -- but also, to some extent, failures attributable to his quirks. One of the points that Isaacson makes in the book is that a key part of the Jobs narrative is the idea that his firing from Apple was the best thing that ever happened to him because it changed who Steve was -- that that piece of his narrative is not true. Instead, it is causation and correlation. Jobs' firing allowed him to form NeXT and suddenly Jobs was able to indulge in all the quirks that he previously could not at Apple. So he insisted on having the walls in his factories to be painted white and for the place to be spotless (incredibly costly and unnecessary.) He built a super high end computer without a market. He worked and reworked the product so that it might be years delayed. Don't get me wrong -- ultimately Jobs came up with something that became the heart of OS X -- but, the lesson was clear. Jobs had to balance the things that he did great (intuition / judgement, relentless pursuit of perfection, recognition of great ideas) vs. the things that he did less so (shipping a product that was good enough but not perfect.) He brought those lessons with him back to Apple and became the CEO he never was.

It's hard to argue against quality improvements, but that's what I'm going to attempt to do here. I think it's slightly reductionist to boil down the argument to quality improvements vs. new features. It makes sense in the context that if you have X number of engineering hours -- they can only work on so many things -- so that might be bug fixes, minor site enhancements, or a new version of the product. What will it be? I want to frame this argument around the concept of value though. Here's how I think about product. Product -- and why people use / pay for / invest in a product -- is all around value. Why do we use our car? Why do we pay for insurance? Why do we buy a new bag? All of them provide some form of value. Perhaps it's pure utility -- or maybe it's piece of mind, brand value, or a hedge. But all of it is still around value. Even if we don't pay for the product, we still pay for it by investing our time to use it.

In the context of value -- how does quality improvements vs. new features fall? Let's think about a product fundamentally. I'll use two examples of sites that I used today -- amazon.com and Fandango. Outright bugs (e.g. broken links) are rarer -- but I still find significant quality improvements even in a site like amazon.com. Maybe it's a detail page that's screwed up, or a problem with the shopping cart, or missing sort functionality. At the other end, there are tons of useful features that I (and surely lots of other people) have thought of or seen elsewhere that sure would be useful for amazon.com to incorporate. However, amazon.com is a rock solid site. Transacts billions of dollars every year and I can buy a huge swath of products and get it delivered to me in 2 days. It is not a buggy site. So in a case like that -- because the fundamental value is there, they likely will be pursuing new features / new initiatives. Let's take Fandango. This site is horrendous -- easily one of the worst sites I've ever seen by a company. I'm not even trying to use the site but a couple of specialized movie theaters in Los Angeles exclusively use Fandango. 3x the number of necessary steps to purchase a ticket, telling me a screening is sold out when it isn't (and I know because when I go back through a second time, I'm able to buy a ticket), me getting lost in the site when I'm trying to do the primary thing the site is built for (purchasing tickets.) In the case of Fandango -- I would put *all* my resources into quality improvements and none into new features. Why? The core thing it's supposed to be good at, it's bad at. It's negative value. I have such a horrible experience that I actively try and avoid using the site. They need to completely revamp the user experience. They need to simplify the site. They need to get someone with taste so that ads and popups aren't assaulting a user. 

That's the theoretical -- let's drill down a little into more practical aspects. The bugs system I'm used to has a classification of bugs from P0 to P4. Here's a rough breakdown:

P0: site is down
P1: not critical, but something is really screwed up and needs to be fixed soon. Search is buggy. Homepage formatting is off. Link is broken. You're not paging someone to fix it, but it needs to get done.
P2: this is the next set of things you want to get accomplished. Could be a new site feature or it could be a more subtle improvement like giving users more options to sort their search results.
P3: a nice to have set of features but you know in the back of your mind these probably won't be tackled for a while
P4: won't get to these unless they get upgraded. Used more to keep a log of every idea / feature request / etc.

For a site launch, it's basically clear out all P0 and P1 bugs -- and then ship it. You might argue a little over what's a P1 bug -- but the team then at least has a shared understanding of what's critical to get done before you're ready to send off. The trickier part is what to do once the site is launched. Note that for many companies -- once the product is out there, you rarely see improvements. I purchased something from B&H Photo today. I'm pretty sure that website hasn't really changed in a while -- they're probably in maintenance mode. Forget new site features, they're probably not doing much even in terms of site quality. So across the board -- knowing that many (maybe even most) websites don't continuously improve and sharpen the customer experience, I think it's critical to invest a reasonable amount of resources against quality improvements. I would estimate 20-30%. 30% right after you launch because there will be more obvious things to improve with it decreasing over time as you close out various bugs / small improvements. The balance, 70-80% of remaining time, should be devoted towards new site features / hitting a next version of the product. That's the lifeblood of product. Constantly iterating and improving. Apple is one of the great companies of my lifetime and their work is staged. Each new release of each product is eagerly awaited. So getting a great new version is critical. 

There's something a little more subtle that underlies this entire posting and that's the product manager and his/her personality. I think self-reflection / awareness is a critical (and often missing) trait of product managers. Why? Let's take me for example. By nature, I'm a little more cautious by nature. So in response, I generally try and ship a little early. I know that I like to have time to really get everything tightened up, so I balance that by pushing myself out the door a little sooner than I'm comfortable with. Over time, I've been able to balance my own personality and judgement (one obviously builds up a library of examples over time -- both things they've worked on directly and things they've studied) -- to figure out when something should / shouldn't go out the door and the next steps associated with it. 

I frequently see product managers / startups fall into two categories. The first is the ones who don't want to ship. They're more cautious by nature -- worried if their product is good enough and they keep refining it over and over again. It's hard to argue with this position, but I'm going to try to. The problem with this is not that fit and finish is not important. It is. The problem is it can be difficult to know the difference between fit and finish and fundamental improvements to the product. Especially for a new product, there are a lot of site features that are potentially useful but work before they're really useful / deliver value. However, information at this stage is sparse. Users don't really know the product that well (even if it's a new version of an existing product.) It's only when the user actually uses the product will you get that information. That's why it's critical to ship. You get a brand new set of information in terms of how the product works, what delivers value, what improvements you need, etc. I like to say that I feel differently about a product at every stage: when we whiteboard, early mockups, actual mocks, internal alpha/beta, when it's live. Every stage -- I can feel the product just a little differently. This is not even counting actual user feedback -- this is just my own gut intuition. That's why you need to keep pushing the ball forward -- so you can get to the next stage to get that intuition and feedback to know what best to do.

Here's the second group. I suspect it's this second group that causes the first group to act the way they do. The second group is comprised of the folks who don't know their product is half baked. They're either inexperienced or, potentially, just have bad judgement. Maybe not fundamentally bad judgement -- but it's sort of like when you see people who wear clothes that don't fit them. It's not that they couldn't understand when a shirt does or does not fit them (forget about how it looks on them), it's just that they don't know. I think the first group thinks of these people and says, "I don't want to look foolish like those people so I'm going to make sure my product is perfect before I launch." I chatted with a startup the other day and they do something around the visualization of data. The CEO told me how they were early but they were getting amazing feedback and that they were aware of bugs and issues but in the spirit of getting an MVP out there (this seems to be the new catchy term of the day "minimum viable product) -- they put it out there. He quite liberally talked about the failings of the product. Here's the problem -- I literally could not understand their graphs. Could not understand them. I tried to gently tell the CEO this by saying, "Can you walk me through how to read your graphs?" This is fundamentally what this startup does and I didn't understand it. I should point out that a) as an economics major -- I've seen more graphs than I'd like to in my lifetime and b) as someone who used to work with data warehouses pretty extensively at amazon.com -- I do have background in using and displaying quantitative information. So for them to think that this is something that they would show customers, investors, etc. -- I just totally disagree.

In all cases though -- it's a sense of where you might fall and then calibrating your judgement / behavior accordingly.

Engineers would often outlast me on a project. I might work on a version of a product and then move onto another project while the engineering team was much more stable. Months later, I would still be getting emails saying that XYZ bug got closed out. Here's what happens. Engineers will literally go through how you prioritized bugs and close them out, one by one, in rank order. That's one of the reasons the role of the product manager is so critical. In terms of day to day work -- you're literally saying, "Work on this, not that." You're deciding whether it's more important to fix a typo on the homepage than to incrementally get closer to v2. At the heart of this question is judgement and value -- having good judgement to know those tradeoffs and a fundamental understanding of your product, how it delivers value, and where investment needs to occur to maximize that value for the customer.

Comments