Failure

Katie Dill: The Design Team’s Rebellion

Lenny: Today we’ve got another very special compilation episode. Something I’ve been pulling on more and more with the podcast and the newsletter, in case you’ve noticed, is failure. Normally I spend a lot of time researching how the best companies and the best product leaders operate, but you can learn a lot and often a lot more from failure. And so what we’ve done with this episode is we’ve looked at all of the past episodes we’ve done and pulled out all the most interesting and insightful stories of failure and turned it into this very focused episode on failure. I hope that you find this useful and interesting. Let us know what you think in the comments on YouTube or on lennysnewsletter.com, or just let me know on Twitter. If you like this, we’ll keep doing this. If not, we’ll, probably not. Either way, I hope you enjoy. Before we dive in, here is a short word from our wonderful sponsors.

Let me tell you about a product called Sendbird. The all-in-one communications API platform designed for both web and mobile apps. In a world saturated with multi-channel communication, product teams are discovering the effectiveness of in-app communication with Sendbird, businesses can elevate their in-app experience with decluttered and branded communication featuring AI-powered chatbots, one-way messages, chat, video calls and livestream capabilities, all tailored for commerce, marketing and top-tier support. Forward-thinking companies such as Hinge, Patreon, Yahoo, Accolade, and more use Sendbird to build in-app communication experiences that drive engagement, conversion, and retention. In-app communication has the highest conversion, highest engagement, and highest satisfaction of any communication channel. And when it comes to investing in this channel, trust Sendbird to take your in-app communication experience to the next level. Start today with Sendbird’s free plan and as a listener of Lenny’s Podcast, you’ll get an additional two months of unlimited usage and access to all premium features, including creating your very own generative AI chatbot. Visit sendbird.com/Lenny to begin your free journey. That’s sendbird.com/Lenny.

All right. First up, we’ve got Katie Dill, who is head of design at Stripe and former head of design at Airbnb and Lyft sharing this amazing story of how the entire design team at Airbnb basically rebelled against her soon after she joined and what she learned from that experience.

Katie Dill: I’m happy to talk about it because frankly, it was the biggest learning experience of my leadership career, or at least that happened in one moment, and it happened in my early days at Airbnb. So I was hired to take on the experience design organization, that’s basically the product design team, which was 10 people at the time.

And so they had been reporting directly to one of the founders and they were going to start reporting to me. And during my interview process, I learned a lot about what was working and what wasn’t working and some of the trials and tribulations with the design organization and its collaboration with others. So it seemed like there was room for improvement in how engineering and product management and design all work together. And there was also really low engagement scores in the design team.

And so I came in ready to go and excited to try to help make some change based on all the things that I had learned from various leaders and people across the company. And I came in swinging ready to go, and then about a month into my time there, I got a meeting on my calendar. Thursday 8:30 AM, was an hour and a half with half of the design team, so that was five people, and our HR partner.

The Cannes Stage Nightmare

Lenny: Oh, no. That’s never a good sign.

Katie Dill: Usually it’s [inaudible 00:05:01]. Yeah. And I remember this so vividly. I remember walking into the office and all the rooms in Airbnb’s office are very unique spaces that look like Airbnbs, but of course this was the one room with all white walls and just a gray flat rectangle table. And I walked into the room and there were five of them seated around the table and they had a pack of papers in front of them, and they went on taking turns quietly reading from the papers all the things that they saw that I was doing wrong and all the things that they didn’t like about me. And it was a really hard moment there. I went through all the usual stages of grief when one hears feedback, which is just immediate want to respond to be like, “Oh, well there was a good reason for that.” And, “That’s not how it actually was,” and, “This is why I did that.”

But luckily I had, thank goodness, I had the sense to just listen and not respond in that way. Clearly, what they were telling me is that that was one of the things that was missing. And so I heard them out and took it all in. And regardless of each individual thing, what was very clear was that the missing piece, the theme that was across all of that is that I hadn’t earned their trust. So whether how right or how wrong what I was doing was is the key piece is that I wasn’t bringing the team along with me. They had no idea that they could trust in what I was trying to build and what I was trying to shape and that I cared about them and that I had their best interest and shared goals at heart. And that was absolutely my fault.

And in retrospect, as hard as that was, I’m very grateful and very amazed that they could come together and share that with me. It can be hard to bring feedback forward like that.

And so it was an extremely valuable learning experience and I took from that to then immediately shift how I was operating. And really a key part in building trust was to listen, to hear out what the individuals on the team were setting out to do, what they cared about, what motivated them. And so I started to make pretty fast change and still moving in the direction that was necessary for the org to make the really large impact in how we were operating, but bringing folks along with me. You can inflict change on people, but if you want to do it with them, really trust is the key element there.

And then a couple of months later, we had the best engagement scores in the company. So it actually, it did objectively improve the situation and since then taken that on into next steps in other companies that I’ve joined and just think about instead of coming in swinging, come in listening, so that you can really set out to make change that actually has true positive impact on the folks around you and you bring along with you.

Google’s Failed Social Products

Lenny: Next up, Paul Adams, chief product Officer at Intercom, sharing the nightmare experience of freezing on stage in front of thousands of people, having to walk off stage, and then people hearing him curse because the mic was still on, and then how he recovered. Plus a few stories of building some of Google’s most infamous product failures.

Paul Adams: Some things have happened in work, are very memorable at the time and they don’t really scar you. This goes in the book that have scarred for life. Yeah, let’s go long story short, I was at Facebook just over a decade ago, loved it at the time. I think it was a great place to be at the time. And basically San Francisco, I did a lot of talks for Facebook internally and externally. Facebook had a keynote slot, always have a keynote slot at Cannes, the world’s biggest advertising festival. And the year prior, Zuck had been interviewed. He was the speaker, he’d been interviewed, gotten a hard time on privacy. It didn’t go well, as well as they’d hoped.

So the next year they asked me to do it, maybe it was the Irish accent that made the offer come my way. And yeah, I got out into front the stage of the world’s biggest advertising stage, and I’d say I was like three or four minutes into the talk, a talk I’d given, a very similar talk that I’d given lots of times.

And I just froze. I couldn’t remember what I was supposed to say. It was the first ever time in my life I’d rehearsed a talk word for word. Usually, I have talking points and I’d ad lib and things get mixed around and it’s kind of informal. This was media trained, do not say the wrong thing kind of talk. And I just could not remember what to say. I had some version of a panic attack, walked off stage, I was still mic’d up, cursed. I was laughing. I was like, “Geez, are they laughing at me. Oh my God, this is…” But I can manage to turn it around, I walked back out, I’d kind of been disarmed internally in my head, and the rest of it went well. And I was famous that night out in Cannes afterwards on whatever the seafront, it’s just like rosé everywhere. I was famous and infamous for my performance.

The Value of Failure

Lenny: I feel like you lived the worst nightmare that everybody has when they’re thinking about giving a talk. And I think what’s interesting is you survived, and I think that’s a really interesting lesson is you could freeze in front of thousands of people, walk off stage, and then it works out okay.

Paul Adams: And it all happened organically, I guess, or very naturally. But yeah, ever since then, every time I walk out onto a conference talk stage, still today, I ask myself, I have this tiny doubt in the back of my head. It’s never happened since. But yeah, I think you have to go with it with these things. When life kind of throws you these whatever curve balls you have got to kind of adapt, and it’s not that big a deal. None of these things are that big a deal. At the end of the day you kind of move on, live and learn. Yeah, but I still hope it doesn’t happen again.

Lessons from Pets.com and Quibi

Lenny: I also hate public speaking and I always fear this is exactly what’s going to happen to me. And so I think this is nice to hear that even when the worst possible thing basically happens, things can survive.

Paul Adams: You can turn it around. Yeah.

The Pets.com Shutdown Decision

Lenny: A second area I wanted to hear from is your time at Google and there’s a couple products you worked on at Google. Both of them were not what you’d call big successes, and then there’s kind of a transition to Facebook, which was also kind messy. Can you just share a couple stories from that time?

Paul Adams: Yeah. Similar to the walking off stage thing, you live and learn, and I was at Google for four years. I was at Facebook for two and a half years or so. At Google I worked on a lot of failed social projects like you mentioned, Google Buzz, Google, and later Google+. I think a lot of the motivation for those projects came from a place of fear. It didn’t come from a place of let’s make a great product for people,.let’s really understand the things people struggle with when communicating with family and friends. That’s really, really try and create something wonderful. It came from a place of fear.

And so during those times I learned I think how not to lead in places. And by the way, I should say at the time in Google, there was other things happening that were amazing, like Google building Google Maps. Incredible product, one of my favorite products. I think one of the best products ever made. They were building Android. I was in the mobile team, in the mobile apps team at the time that Android came out. So incredibly good product. So I just happened to be in the social side, which wasn’t as good.

And yeah, Google Buzz was kind of a privacy disaster, and Google+ similar. And so kind of halfway through I kind of published research about groups, and I’ve done a ton of research. An interesting kind of side note there is at the time I asked, I was working in the research, in the US team as a researcher, I was being asked to do a lot of tactical research, like usability study type stuff, like can people use these products? And I ended up doing a lot of formative research as well in the same session. So I’d kind of say to the team like, “Hey, I’ll do the research, I’ll answer your questions, but also I’m going to do this other thing, and I’m going to take 20 minutes doing that.”

And so what we used to do is, what I used to do with people was map out their social network, all the people in it, their family, their friends, how they communicate. We’d map on all the channels, we’d talk about what worked well, what didn’t. And we did this with dozens and dozens of people over the course of maybe 18 months. And the same pattern emerged every single time, which was people need way better ways to communicate with small groups of family and friends. And I kind of look back now and go like WhatsApp, or it maybe iMessage if everyone’s on Apple, but really obvious in hindsight, but at the time, not obvious. And so we kind of tried to build a product around that called Google+, but again, it was kind of came from the wrong place. And so halfway through the research that I’ve done, all this research had been made public through a conference talk and Zuck and Facebook noticed, got in touch, one thing led to another and I left and joined Facebook, which was an amazing thing for me, personally.

Facebook was amazing, an amazing place at the time and exciting. And they were trying to do things for the other reasons, the kind of good reasons, like, “Hey, let’s build an amazing product for people.”

Starting Over After Quibi

Lenny: And this was during Google+ being built? You basically shifted.

Paul Adams: Yeah, midway, I’m stressed to even telling you about it. The project hadn’t been launched. It was still under wraps. It was highly confidential. Google had done a lot of things at the time that were the first for them. I don’t know if they’ve done them since, but things like everyone worked in Google+ was sent to a different building. That building had a different key card. If you didn’t work on Google+, you could not get in. All sorts of counter-cultural things at the time. And as a result, there was a lot of antagonism internally for Google+. And so when I left in the middle of the project, leaving with all of the plans in my head to the enemy, some people saw me as a traitor, understandably. Other people thought I was enlightened. It depends on who you talked to, but it was the right thing for me to do. But at the time it was a hard thing to do.

Lessons Learned from Quibi

Lenny: I know there’s also a lot of scrutiny in what you took with you and the process.

Paul Adams: When I left, Google kind of assumed that I was one of the spies. I was quarantined when I told them I was leaving. They forensically analyzed my laptop, all sorts of stuff like that. So it was pretty intense. Looking back, I can understand why that happened, but the root cause for me is that the project has been run from a place of fear, competitive fear, which I don’t think leads to good things.

Lenny: So one of the themes through the stories you just shared is, let’s say failure is… I don’t want to make it that harsh, but just things not working out. And I’m curious as a product leader, how important you think that is for people to go through if you think that’s something that is almost a good thing. And I guess just is there anything there that you find helpful as a coach, as a mentor, as someone, to people that are trying to become basically you?

Paul Adams: It very, very… It still is. It still is. I’ve personally failed so many times. There are two stories and the Google one is long deep tentacles. There are two stories. I’ve failed a ton of times like at Intercom. I remember when I was at Facebook, I was very happy and I knew I wanted to [inaudible 00:16:39] the co-founders of Intercom and they’re trying to persuade me to join Intercom. It was like 10 person company at the time. But Owen said something to me at that time, which has stuck with me ever since. He said, “At Facebook you can design the product, but at Intercom you can design the company.” And that was extremely appealing to me, a great pitch. He’s like, “Just design the company with us that you want to work in.”

And so part of that was a company that embraces failure that says it’s okay to try things. I’m a big believer in big bets, higher risk, higher reward. I don’t get as excited about incremental things. Now I haven’t said that, there’s of course a place for that too, especially as companies get bigger. But I get excited about big bets and if you make big bets, you’re going to get a lot of it wrong. So a lot of the principles that we built here at Intercom, on building software, we have a principle called ship to learn, and we’ve actually changed it since, still on the wall here. Ship fast, ship early, ship often is what it says now. Used to say ship to learn. Ship fast, ship early, ship often. It’s like in that idea is the idea of failure. It’s not going to go right, and it’s going to go wrong more often than not. But if you ship early and fast and learn fast, you can change fast, and you can improve fast. And that’s the kind of culture that we as much as possible try to embrace and teach people. But it’s much easier said than done.

Lenny: Especially when you’re in the moment. Like Go damn, it, everything’s going to fall apart. I really messed this one up.

Paul Adams: Yeah. And there’s a trade-off with quality that people really struggle with. We’ve high standards of ourselves. A lot of Intercom comes from a design founder background. We value the craft a lot. We never want to be embarrassed by what we ship. So there’s a real tension there, a real trade-off where people have these high standards, which we encourage and we encourage them to ship fast and learn and make mistakes. It’s a constant kind of tension that we’re navigating.

Lenny: Next up, we have Tom Conrad, who is chief product officer at Quibi and engineering leader at Pets.com. Two of the most memorable failures in product history, sharing his lessons from those wild experiences. Tom is currently CEO of Zero Longevity Science, which is a killer business and an app in case you haven’t come across it. I’d definitely check it out. Here’s Tom.

You brought up this phrase of notable disasters and I want to talk about that. You’ve worked at two of the most famous notable disasters of product companies, Pets.com and Quibi. I think it’s really rare someone sees the inside of so much hype and then such a fall at a company. And so I just want to spend some time in these two areas, and maybe the way to set it up is just what’s a lesson you took away from each of these two experiences that you’ve taken with you to future work, and maybe advice you share with people?

Quibi’s Big Bet and Fundamental Flaws

Tom Conrad: Probably the biggest lesson, it’s not really about the specifics of the business. The biggest lesson really is these things make you better. In some instances, actually I think in both instances, they became kind of dominoes that opened doors for me in my own ambition and my own sort of professional life that maybe just wouldn’t have opened at all if I hadn’t gone to those companies and learned those things and had those experiences. And frankly, even in the case of Pets.com, like even the high profile nature of it, I could have worked at one of a thousand e-commerce websites in 1999. And when I went on to some subsequent job interview or something and talked about my experience, I had never heard of the thing that you worked on, but everybody certainly heard about Pets.com.

It’s a pretty funny example too of how some struggles are timeless. That was 23, 24 years ago now. And while as a leadership team, we made, I’m sure, all kinds of mistakes. One of the things that happened was that there were three kind of over-funded pet e-commerce sites, and we all raised in excess of $50 million, which is a tremendous amount of money now. It was a tremendous amount of money then, and we all thought it was a zero-sum game and that we as one player started to spend on promotion or to spend irrationally on national broadcast television advertising. We all did, and it became this kind of unwinnable arms race. So there is, I think a fundamental lesson about having an excess of investment can be its own albatross or lead you to make decisions that maybe would be unwise.

And then of course, it’s just like timing is really important. Chewy is a online pet store. It’s worth $9 billion today. They were a private company and bought by PetSmart and then spun back out. But when they were bought by PetSmart, they were acquired for 3 billion, biggest e-commerce acquisition of all time. And while I think it’s probably unfair to compare, Chewy who executed exceptionally well over a decade, grew their business brick by brick, and turned it into something really remarkable. To Pets.com, which was in a very, very different moment in time and tried to go to market in a really different way.

The critique that is often leveled at Pets.com or at least at the time, was like, this is just a stupid business. They’re shipping dog food around. You could never make that work, and that’s just wrong. You absolutely can make it work. Probably can’t make it work when 80% of the country on the internet is still on dial up. It’s really, really early.

The Need to Fail Completely

Lenny: I saw a stat I think you shared somewhere that you took Pets.com from nothing, to a public company, to completely out of business in 19 months.

Tom Conrad: Yeah, yeah, I think that’s about right. The other thing that’s forgotten in the tale is that we actually didn’t go bankrupt. We shut the company down and returned the remaining balance to the investors, which no public company had ever done before. And the leadership team just reached the conclusion that given the way market conditions had evolved, there was just no way we were going to be able to get more capital into the company. And it was a company that required additional investment to get to profitability. And so it was better to wind down early, take the money that we had in the bank and get it back to investors than to just spend every last penny on what was sort of a fruitless attempt to salvage it.

Lenny: Did not know that. Let’s talk about Quibi. What went wrong there? Do you think there was a path to Quibi having worked out? Any big lessons that you took away from that experience that you bring with you?

Tom Conrad: The kind of miraculous thing about Quibi for me was it relit my enthusiasm for the industry for doing this work. I had left in, I think it was December of 2018, and I thought that maybe I was just done making software. I had done it for a really long time, I had done it for twenty-five years or something. And I had changed a lot. The industry had changed a lot, and I thought maybe I just didn’t have the same passion for it that I had a decade before. And it also seemed like maybe it’d be fun to have another chapter of my life that was just completely different. And I had a whole list of things that I thought I might want to do. They were really, they were kind of ridiculous. Maybe I want to be a pastry chef. Maybe I want to be a landscape photographer. Maybe I want to learn to make bad music to put up on SoundCloud or something. Really the only thing they had in common were they were all things that I knew nothing about. People would be like, “Oh, you think you might want to be a pastry chef? Do you like to bake?” And I’d be like, “No, I don’t know anything about baking.” “Oh, you might landscape photography. Do you take photos?” “No, I don’t make photos.”

But I was kind of committed to the bit, actually to the point where when TechCrunch interviewed me about my departure from Snapchat, I was like, “I’m out. I’m going to do something else entirely.” So that story is very much out there. But a few months after my last day at Snap, I got a call from Meg Whitman and Jeffrey Katzenberg who were starting up, it was called New TV at the time.

And the pitch was, “We’re going to try to take the best of mobile and Silicon Valley and Consumer Tech and sort of weld it to the best of Hollywood-style content production to build something completely bespoke and purpose-made for consumption on the phone.” They were looking for both technology leadership and product leadership and wanted to know if I was interested in one or both. And I took the meeting, even though I wasn’t really taking these kinds of calls from anybody. It just seemed like who’s going to pass up the opportunity to have lunch with the two of them? So I listened to the pitch and politely declined and told them that I was going to be a pastry chef or something. And we kept doing that every couple of months for seven months. We’d go to lunch, they would give me an update on the progress they were making, and I would decline the invitation to get involved somehow.

And then late in that year, I went to lunch one more time and Meg explained that they brought on someone to lead technology and they brought another person to lead product, and both of them really truly for reasons that are completely disconnected from Quibi itself, both of them had left after about six weeks. And Meg’s like, “We’ve raised all this money, and we’ve told the world that we’re shipping this product in about a year. We got an awful lot to do, and I really could use some help, and I would consider it a personal favor if you would come and spend just a couple of days a week helping.” She’s like, “I’ll continue to look for someone who actually wants the job, but it’d be great if you could help me get this off the ground.”

And my wife is a freelance writer, marketing strategist and loves her life as a freelance contributor. And she’s like, “You should do this. Why not? It’s two days a week, it’s just a few months. What’s the worst thing that could happen? Maybe you’ll like it.” And I’m like, “No, no. Here’s the thing that will happen. I won’t do it two days a week. It will immediately be three days, then four days, then five days, then six days. I just know myself.” And she’s like, “No.” She’s like, “Just on Wednesday night at six o’clock, close your Quibi laptop and be like, all they’re paying me for is for Tuesday and Wednesday and then open it back up on Tuesday morning. That’s all you’ve got to do.”

Well, she’s right about most things and she’s wrong about this. I fell deeply into it right away, and it was just so fun to get to build a team from scratch and to design and build a product from scratch and to take advantage of all of the sort of modern software architecture stuff that had come into being over the course of the 15 years since we had started Pandora. And I’m embarrassed about some of the what happened with Quibi for sure, but I’m super grateful for the experience, because I just really fell in love with the industry again and was reminded of just how rewarding it can be to build something and to try to put it out there even if you stumble pretty mightily along the way.

Lenny: Is there something that you took away from that experience that taught you what to try to avoid, to try to pull towards?

Tom Conrad: I think I sort of misunderstood or misjudged companies sometimes by thinking about them really focused on the product execution. If you find an interesting problem that people care about and you solve that problem in a really beautiful, elegant, delightful way, that’s 10 times better than anything else that they can get in that same space, they’ll tell their friends and all the rest will take care of itself. And so that was always my ambition. Find a thing that I cared about building, do a great job building it in a delightful way, go really deep on listening to people and their feedback and iterate your way to success and breaking through that membrane that we all strive to get across, the really great word of mouth.

But I think the thing I’ve come to better appreciate is that companies are also, they’re kind of a math problem that describes how you take investment and pour them into the equation, and out the other side comes returns on some time horizon. And yes, there are variables in that equation that are influenced by the product that you build and all of the little details and decisions that you make about making that product great. But if the equation is fundamentally broken or a big swing in and of itself, no amount of iteration and execution can get you out of the failed outputs of the broken equation.

And I think Quibi made a bet that you could build an entirely bespoke content library that was sufficiently scaled to get people to subscribe and retain for a couple billion dollars. It was a huge amount of money, but we made 70 shows in 18 months, which is more content than all of the major broadcast networks combined made in a single year. So it was a pretty major accomplishment. And we made a bet that we would augment those sort of episodic and serialized or Hollywood style shows with a bunch of daily content that we produce at the level of network television, nightly news, and so forth that would be an alternative to some of the sort of daily content that you might otherwise get on YouTube. And that was going to be about a third of the content spend.

One super interesting thing that no one talks about is that all of that content was designed to be made day of or day before it aired. So there was no back catalog of it, and it was all designed to be shot in these professional studios that we built out, and it was really expensive. Like I said, it was a third of the investment we were going to make in content, almost half the investment we were going to be making content. And we launched two weeks into Covid, and we couldn’t make any of that content except literally in the garages of the host’s homes. And so we had this thing that was supposed to seem really set apart from YouTube that literally now was being made exactly YouTube content, which is sort of like self-produced at home with very little sort of the support infrastructure of Hollywood.

Now you can argue, I think the content on YouTube is really, really exceptional in this category, and maybe we were never going to do better than that. But I think what was really fundamentally broken with Quibi was that the actual foundational equation of can you make enough premium content that’s totally bespoke and made for the service and takes advantage of the nature of the phone, is that enough content to get people to sign up and retain, and can you do that for a couple billion dollars? And I think the answer is no. The library has to be much, much bigger and you have to have, like any company, you have to have sufficient time and energy to iterate on the content format itself. Our roadmap really wanted to innovate on the content format. And so I think part of what happened is pretty quickly it became clear that the math was just wrong. It wasn’t going to take 2 billion, it was going to take six or eight or 10 billion. And the risk reward profile of betting 10 billion on the format was just more than anyone can stomach.

Lenny: Next up, we’ve got Sri Batchu, former head of growth at Ramp who shares something that I’ve thought about ever since we had this conversation, which is this idea that when you fail, make sure you fail conclusively to make this failure an actual learning that you can build off of versus just a waste of time. Here’s Sri.

Trade-offs in Running Experiments

Sri Batchu: Growth experiments in my history are typically like 30%-ish success rate. So the vast majority of things that you try don’t work. And so you want to create a culture where people aren’t afraid to take risks and aren’t afraid to fail. And for me, failure is not that you didn’t drive revenue, failure is not learning. So it’s really important that you learn when you fail. And so we celebrate failure as long as you’re learning, and you can only learn if you’ve designed the right test and you failed conclusively, because otherwise I think many of us have been in situations where there’s intuition that something might work and it doesn’t work, and then you end up doing it over and over for years because every time a new executive or somebody else has the same idea, you try it again. And it’s because you haven’t been able to design the test to fail conclusively.

It’s hard to do. But at the end of the day, there’s only two ways to make an experiment successful. Either you have a very large M or you have a very significant treatment, which is what you’re doing in the experiment itself. And in B2B, you don’t usually have the luxury of large M, which you [inaudible 00:34:54] consumer.

Facebook can get [inaudible 00:34:56] in two hours. A B2B company could take two years to get to the same number of touch points. And so to counteract that, I recommend people just trying to maximize the treatment effect, which is like if you have a hypothesis that you’re testing, just throw all of the possible tactics and resources that you think would move that needle because you can always cost rationalize later if it works.

And so just maximize the treatment effect. And if with all of that it didn’t work, then you can say, “Hey, we’re not going to try this again because we literally did try everything that we could to test this hypothesis. And if it doesn’t work in the best version, and it’s expensive as it is, this is not worth spending more time on.” But if it does work, great. Then you do another version of the test with half the tactics or whichever tactics you think work better or worse and you optimize over time.

Top Mistakes for New Product Managers

Lenny: Is there an example you could share when you did that?

Sri Batchu: Account-based marketing is something that is very common in enterprise software where you’ve selected certain customers that you think are high priority and you’re saying, “I want to touch them in as many nuanced ways possible to see if that drives conversion.” And this is something I’ve seen tried many times where people do it, but they kind of do it halfway where they’re like, okay, tried these three things. Conversion of the control group wasn’t higher, and so we think it is not going to work. And then a new go-to-market executive comes and they have to do it again. They have to do it again. They have to do it again. It’s like a very common one wherever this happens. And so when we did it at Ramp, we did exactly what I just described, which is like, let’s really be thoughtful about the experiment design, both in terms of maximizing the number of people as well as maximizing the number of ways and types of ways that we’re effectively touching these target customers to show the value one way or the other.

Airbnb’s Biggest Product Mistake

Lenny: So what it sounds like is the hypothesis isn’t like this email will have a big impact on conversion. It’s like this strategy of coming after customers is what we’re testing.

Sri Batchu: That’s the example there. And I think for example, if you had the… This kind of framework is more important for cross-functional, larger scale, bigger tests rather than an email modification. But we can even use it on a micro example like an email modification where you are like, “Okay, I think this particular email is underperforming because it’s not talking to this part of the customer’s pain point or journey or what have you.” And you could just, the simplest test would be, okay, let me make some tweaks to the text and edit that, and that could be the end of that test. And if that doesn’t work, you’re like, “Oh, maybe those weren’t the right text edits. Let me do a different text edits or whatever.”

And that’s fine, that’s low cost. It’s not the end of the world and it’s for you to be wrong there. But an alternative that you could do is like, “Oh, what are all of the things that I could change about this email in the same test?” Is it the trigger of the email? Is it the text content of the email? Is it additional personalization? Is it the design of the email? Trying to think of what are all of the various levers that you think could be wrong and put them all together to test your hypothesis of this touch point is wrong, and how do I improve that?

The Two Sides of Everyone

Lenny: Well, obviously the downside of that is you, if it doesn’t work, you don’t know if it’s like, oh, maybe it was this thing could have worked in the subject.

Sri Batchu: Yeah, so there’s always trade-offs on this, but what you’re hoping is you’ve done a complete refresh where you did all the things that you thought were intuitive that should work. And if it doesn’t work, then you’re like, okay, maybe my hypothesis wrong. But you’re right. There’s always going to be a challenge if maybe the execution is wrong. And I did too many things potentially in that case.

Lenny: Our next story is from JZ, who is a colleague of mine at Airbnb, head of product at Webflow when we recorded this episode, and this is her sharing the story of one of the biggest product misses at Airbnb.

You’ve seen a lot of new PMs, and you’ve seen these PMs succeed, you’ve seen some fail. What are the most common mistakes that you find new PMs make in this experience of helping new PMs get into the field?

Jiaona Zhang (JZ) (00:39:27): I think something that is really hard to untrain, but I think every human does it, is you jump to solutions. And so one of the biggest things I see, not just in my course, but also just as a PM and some of the mistakes that you make as a PM is the idea of you get really attached to a solution, a way of implementing something, something that you can see in your head that you want to build. And so that’s the first thing I really want to like unteach in our course.

And so a lot of people will literally come in, they’ll be like, “I want to build X startup,” or, “I want to do this thing,” or, “I am in blank school, and I’ve been doing a lot of research on this particular area.” And so untraining that and being like, “Hey, we’re going to go out there. We are not going to think at all about the thing that you want to build, but instead we’re going to be focused on users and people in the real world and their problems. And the first step is to understand their problems and then understand if there’s an opportunity here as opposed to, hey, you want to build X thing for Y person.” So that’s the biggest mistake that you really have to unteach and retrain thinking around.

So let’s go to the other side of this question. We talked about what mistakes new PMs make. I’m curious, what’s the biggest product mistake that you’ve made?

Jiaona Zhang (JZ) (00:40:34): Wow, that’s a good one. It’s so interesting. I feel like as product people we’re always making mistakes and we’re always learning. Maybe I’ll give an example from Airbnb since you and I were both there.

And this one does stand out to me. So we’re working on this concept called Airbnb Plus. If you took a step back, what we’re really trying to do is to be like, “Hey, not everyone trusts Airbnb in terms of it’s a platform. It’s not like it’s managed inventory, it’s not a hotel. How do you go in and really make sure that we’re all the Airbnbs are meeting the quality bar?” But I do think we were very solution first, and I think we’re also competitor afraid at the time. So it was during a time where there were managed marketplaces, there were the Saunders out there, and I think that as a company we’re very much like, “Oh, look at this. What are we going to do in the world of managed marketplaces?”

And so we went really hard down the solution space. We essentially were like, “Let’s go inspect our inventory. Let’s actually try to manage our inventory more.” And really what we should have done is taken a step back and be like, “What’s the real problem?” The real problem is people want to know what they’re getting themselves into. We need to represent the homes a lot better. And I think the other piece here that’s really important is what, as a company, is there strategic strength? And what’s in your wheelhouse? So for example, Airbnb, we weren’t that strong in operations. We again, we’re this platform with this marketplace. And so if you don’t have that muscle and then you’re asking the company, the teams to essentially build it from the ground up, that’s really, really difficult. Not to mention the unit economics. Are the unit economics actually going to work, even as you scale?

Yeah, I feel like Airbnb Plus is an untold story that somebody should tell, and that could be its own podcast, I guess.

Jiaona Zhang (JZ) (00:42:09): You and I can tell it.

We could tell it. This could be Airbnb Plus the hidden, the story. As you said, the problem it was trying to solve was people don’t really trust, they don’t want to even consider Airbnb. Like, “No, I don’t want to stay in someone’s home. I don’t know what it’ll be. It’s unpredictable.” And so as an outsider, it felt like a really clever approach. We’re going to get them, we’re going to make sure they’re awesome. There’s a minimum bar. And I guess this is the question is do you think it was just like this is never possible because we’ll never make money as a business doing this, because we don’t make that much booking and investing time, resources, sending people pillows, all that stuff is ever going to be economical. Or do you think there was a path, and it was just not executed well?

Jiaona Zhang (JZ) (00:42:51): I think there wasn’t really a clear path. I think there was [inaudible 00:42:55]. Exactly. And it was more just like if you understood, again, this is my point around unit economics, there are things where I think you have magical thinking around unit economics. You’re like, “Well, when we get to the scale of X, it’s all going to work out. We can make these things happen.” I think you actually need to really make sure the unit economics work right at the beginning. So that is definitely one lesson.

And I think the other thing is, and going back to the spirit of what are you trying to achieve. If you’re trying to achieve this idea of really knowing the quality of the place, and for a platform like Airbnb, the right way to go about doing is through our reviews, through our guest reviews, which are essentially free as opposed to literally sending out inspectors.

And I think that the other things are if you can get signal on what are the things around quality that people care about? Is it cleaning? Is it the, “Hey, I’m locked out.” And I think that there are other solutions besides inspection that then get at that. So for example, it is actually cheaper to go send everyone a lockbox than to deploy an inspector and go look at your property, right? It is actually cheaper to maybe do a partnership with a bunch of cleaners in different local areas, and then get that as part of the feat as opposed to doing inspection.

So again, it’s really about what are you really trying to achieve? What is the user problem in each of these areas, and can you target that problem with the particular listing that you’re looking at? And so yeah, I personally don’t believe the unit economics ever would’ve really worked out. I think we should have known that, or we should have dug into that more at the very beginning, and then to get very tailored instead of one blunt instrument to solve it all. Hey, we’re going to go inspect. It’s like, what is the problem for this listing, and what’s the best solution to fix that problem?

Our second to last story is from Gina Gotthilf, who was an early growth leader at Duolingo. She’s currently COO of Latitud, and this is her sharing a wide-ranging and important point about how everyone has both an A side and a B side to their career, and people often only share their A side. So this is Gina sharing her B side.

From Tumblr to Duolingo

Gina Gotthilf: We are very encouraged in our lives, especially professionally, to talk about our A side all the time because that’s what impresses people. That’s what opens doors, that’s what allows us to keep growing, and it’s so important. So it means that a lot of what you hear in podcasts and on stage ends up being the Instagramable version of someone or a company or a country’s trajectory. It’s just the highlights. And when I talk about my A side, it’s very impressive. I did things like, we’ll talk about, I met President Obama, I worked on the Mike Bloomberg presidential campaign. I helped Duolingo scale from three to 200 million users. I worked with Tumblr, helping them scale Latin America, Andreesen Horowitz invested in my company, etc.

But between all of those highlights, there were so many B moments that get shoved under the rug because it’s just easier for me and it’s more impressive for others. But I really like to highlight those because I think that most of us have a lot of B moments every day, every week, every month, and every period of our lives. And it’s easy to think that things aren’t just not going to work out for us because we’re in one of those B moments if we don’t recognize them as moments.

Career Ups and Downs

Lenny: I love this concept. We’re going to talk as you expected about a lot of your A side stuff. Is there any example of a B side story of your life that would be interesting to share?

Gina Gotthilf: Look, I think those are the most interesting because they’re funny or ridiculous. I had a lot of B sides, and I still do. For example, I had no idea what I wanted to do. I actually wanted to be. I thought I wanted to be an actress. I either wanted to be that person in Sea World, who goes like this with a dolphin. This is before Sea World was canceled. Or I wanted to be an actress. I applied to schools. I didn’t get into any Ivy League. I didn’t get into any of the top schools I wanted to go to. When I got to college, I actually ended up dropping out because I got so depressed, incredibly depressed, couldn’t get out of bed depressed. Ironically, I dropped out of Reed College, which is the same college that Steve Jobs dropped out of. So I was just destined for greatness. I knew it at that moment.

Careers Are Long Enough

Lenny: It all makes sense looking backwards, as he said.

Biggest Growth Team Lesson: Badge Experiment

Gina Gotthilf: Totally. I was dropping out, being like, “Yes, this is exactly the path.” No, I was miserable. I thought there was no path forward. And I finally went back and graduated. The college counselor looked at my curriculum and said, “What have you even done with your life? There’s nothing to show for.” And it was shocking because I was always the overachiever who wants to do the maximum curriculum and ace all of my classes and do whatever. I did three diplomas in high school, the international, the American, the Brazilian.

And so that for me, I think my one learning there that has stuck with me, and I think it can work for other people too, is that it’s not just about doing things that actually matter and learning. It’s about being able to tell the story, and it’s about understanding what other people perceive as valuable. I applied to a hundred companies. I didn’t hear back from most of them. I finally got an internship at kind a tier B/C digital marketing agency in New York City because I wanted to live in New York so badly. And they forgot to apply for my visa on time. So I lost my visa and had to go back to Brazil, and then I ended up leaving that organization to go work for another one. And I won’t even go into the details of the shadiness of that company that I worked for, but then they ended up laying me off. So I lost my visa again, had to go back home, found another opportunity, got fired that time. So there was just a lot of rockiness in my start that I don’t think you would imagine when you see someone up on stage leading a conference for 5,000 people. That I think is important.

And even when I started working for Tumblr, I was like, “This is it. I made it. This is a really interesting company. This is going to work out.” That was super rocky because it was an early stage startup. So for example, they couldn’t figure out how to wire money to Brazil. So I was not paid for six months. And at one point, me and my colleagues were trying to get money out of the teller to pay contractors because we had no money to pay them, and we borrowed money from people. And finally they also laid me off because they decided to sell to Yahoo.

And then I had to figure out what am I going to do? No one’s going to hire me. I’ve been fired and laid off so many times. So this is all before I started an agency to help US-based tech companies and startups grow in Latin America because I figured I was in this really great place to make that happen. And it eventually worked for, well-known companies such as Duolingo. At the time they weren’t well-known. They were a tiny little startup. They didn’t have an Android app. And that’s how I started working with Duolingo because their head of marketing connected with someone they had worked with at Flickr and said, “I noticed Tumblr grew a lot in Brazil last year. Can you recommend a company or an agency to help?” And they said, “This girl.” And I was twenty-six. And so that’s how they connected me with Duolingo. And I started helping them grow in Brazil as a consultant. They were like, “This is great. Can you help us grow in Chile, Argentina?” And I was like, “Yes.” They were like, “How about Mexico?” And I was like, “Yes.” Did I know anything about these places, Lenny? Did I know people there? No, but you can figure it out.

And then they ended up asking me to come on full time, do that across the world, Japan, China, Korea, Turkey, Spain, France, et cetera. And then to own growth, which ended up meaning communications, social media, government partnerships, anything to grow. And then eventually became an A/B testing growth engine with engineers and PMs and designers of which I knew nothing about.

And even after that, I left Duolingo five years later, didn’t know what to do with my life. You’d think, “Oh wow, you have it figured out now. You left Duolingo, you have the world in front of you.” And I’m like, “Maybe I can finally go work for nonprofits,” which is what I actually wanted to do in the first place. Tried a hand at that. Had a couple of experiences before going to work for the Mike Bloomberg campaign.

Working for the Mike Bloomberg campaign is impressive. But you know what? Mike Bloomberg didn’t win. He’s not the president. So that was not a successful campaign if you really look at it. And yeah, Latitud seems like it’s a really promising path, but there’s A days and B days. So it’s just a lot of that. And just staying resilient and believing in yourself and getting back on the horse when you fall on your face.

Lenny: Amazing. That’s such an important message. I think one of the threads from what you’re describing, something that I think about a lot is people kind of underestimate how long their career is. There’s just so much time to do stuff and for things to start to work. This going to sound really fancy, but I think Marcus Aurelius has this quote about how our life is actually very long. We just use it really badly and we just waste a lot of our time.

More Failed Attempts

Gina Gotthilf: I think you’re so right, Lenny, and I love that because people are going around being like, “Life is short, life is short.” But that’s so true. We waste so much time. But also I think we don’t recognize how much opportunity we have in front of us. And as a 26-year-old, I definitely thought my career was over. I was like, “I blew it.” And looking back, it’s funny.

Great Interview Question: Worst Product Launched

Lenny: Yeah, I know exactly what you mean. I spent nine years at my first job at a random company in San Diego in a startup. I was like, “What am I doing here so long?” And it turned out that was really useful for the thing I did next. And then eventually, wow, things started to really take off. So I think that’s a really good lesson for people. It’s just a long time. This is my fourth career. I’ve switched careers many times. I was an engineer then. I was a founder. Then I was a product manager, and whatever this is. Whatever you call this thing.

Gina Gotthilf: I guess me, too. I was an operator. I was a consultant. Well, I was an employee. I was a consultant. Then I was an operator, which is a fancy way to say employee at a startup. And then now I’m a founder and a VC and an angel and whatever this is..

Closing Thoughts and Conclusion

Lenny: Awesome. Awesome. So I think that’s a really important takeaways. Just there’s a lot of time to do stuff and don’t stress if things aren’t moving as fast as you want. I’m curious, what’s a mess-up or a big mistake maybe that you made or your teammate that was like, “Oh wow, that was a big waste of time.”

Gina Gotthilf: Yeah, look, a lot of things didn’t work out. More than 50% of our A/B tests didn’t work out. We made bets that didn’t make sense. I will say though, that in the spirit of A and B sides, I, and I think in general, we are really good at forgetting the B stuff. I talk so much about all the stuff that worked that it’s hard to remember all of those moments that didn’t actually work. And the thing that I tend to talk about, which is this mistake that we made as the growth team is almost like one of those, when you get asked in an interview, what’s your biggest weakness, and you’re like, “I’m a perfectionist.” It’s like one of those things that actually makes you sound good because it’s the story about how my team really wanted to implement badges. We spent a lot of time playing all the games that were popular at the time, trying to understand how those gamification growth hacks that we could find in those apps would potentially overlay onto Duolingo and how we would do that.

And badges was just pervasive in all of the top games. And so it seemed like a no-brainer, but since we ranked all of our experiments in terms of ROI and return being like how many users we think we’re going to get from this DAUs, and time investments, it never made sense to focus on this because we thought that the time sink would be too high. So I actually ended up not letting the team run this experiment for six months so that we focused on lower hanging fruit. So that’s a mistake on my end.

Then we decided to run this experiment in the most lean way possible. We’re like, “You know what? There’s MVEs. There’s minimum viable experiments. We don’t have to run a whole badges thing. We can just do something more simple and actually see if that leads to growth in an interesting way, and then we’ll know.”

And we ran this very simple experiment that was like, you signed up and then you get a badge. And it was like this girl with a balloon. I don’t know, she was happy or whatever. And of course in retrospect, it led to no results, because no one is proud of signing up. It’s not an exciting moment, and you don’t even have badges to collect. You can’t show it to other people. None of the things that make badges compelling were there, but we were like, “Okay, well, we tested it didn’t work.” And then we moved on.

So we moved on for another, I don’t know, eight months and we didn’t look back. And then when we did look back, first of all at that point we discovered that we hadn’t been dogfooding, which also was embarrassing. Looking back, we hadn’t been dogfooding in the growth team. We just come up with hypotheses. We were super careful about prioritizing them and making sure that we were doing the best possible write ups and all these things. But the dogfooding piece, I didn’t come from a product background. I was a marketer and I didn’t really understand the term dogfooding, but when we thought, we had a conversation, we were like, “You know what? If we had just tested that, we would’ve all known that this was a super lame badge.” And I was like, “Why are we not testing our experiments?” And so that became part of our practice.

It’s still relevant. I just had a conversation yesterday with engineers at Latitud. I haven’t explained what we built yet, where we’re building it. Maybe we’ll get there, but I was talking yesterday, [inaudible 00:55:35] at Latitud, and they’re awesome. In terms of product team, we have the number eight employee at NewBank. You might’ve heard of NewBank, but it’s this massive banking fintech in Latin America. And we have people from other fintechs. And then we have this guy who was a lead PM at Twilio, and I was explaining to them why we should be dogfooding. And they were all like, “Oh yeah, we should dogfood.” It’s just easy to forget stuff like that.

So that was a mistake that we could have probably gotten to the growth that we got to with badges much earlier on. And not only did we get to growth with badges, but it became this amazing treasure trove of opportunity because once you have badges and people want them, you can now ask people to do anything. Go find friends, go buy things, whatever it is. And so we impacted almost all metrics across the company positively, including some we hadn’t expected, but it’s easy to talk about a mistake that ended up being a win. So that’s why I compared it to the interview thing in the beginning. But we tried making Duolingo a social app really early on and failed. It was called Dual Duels. Dual Duels. You could duel.

Lenny: Very clever.

Gina Gotthilf: Yeah, I know we were clever, but people didn’t use it and we didn’t figure out why. We tried making a Duolingo for schools platform. We couldn’t get it to pick up. I went and watched dueling go in China and it got downloaded by a million people in the first day, and then the app got blocked because of the government, and then we couldn’t figure out what to do. And then everyone rated the app like one star because it didn’t work. And so then we had a lot of trouble actually recovering from that. We launched Duolingo in India and didn’t realize because we couldn’t have unless we went there, which we finally did, that most people set their phone UI in India to English, because typing in Hindi is hard. And of course there’s a lot of languages throughout India, and we were making it so that when you downloaded Duolingo, whatever UI you open your app, your phone was set to, we offered not that language for you to learn. That was your base language. So we were telling people learn French, Spanish, German from English, and they were all trying to learn English, so they didn’t find what they were looking for and they left. There were so many mistakes and luckily I think we were able to bounce back from most of them in terms of how Duolingo was doing today.

Lenny: And our final story is from Maggie Crowley VP of product at Toast, and one of the most beloved episodes of the podcast. This is Maggie sharing something a little bit different, her favorite interview question about failure and what it tells you about the person you’re interviewing. Plus a story of her own product failure. Here’s Maggie.

Maggie Crowley: A question I ask in every product interview is, what’s the worst product you’ve ever shipped? And that’s because I don’t think you’re a good PM if you haven’t shipped something that’s really shitty. You just haven’t had enough reps, you haven’t done it enough times. And it’s not only that you’ve done it, but that you can admit it and which one it is. That’s so important.

I remember… It was so dumb. I’m still so mad about this that we did this. I won’t name which team, which company, I’m not going to call that out, but we decided we needed to do a rewrite red flag number one of existing product, and engineer who I’d worked with many times, we had a really good relationship and this person was like, “Yeah, yeah, it’s going to take six months. No problem.” Core part of the product, been around for forever. One of those things that the code is still the code written by the founders kind of thing.

It didn’t take six months. It took two and a half years. It still wasn’t done. It almost never… It went on for so much longer than it should have. It took us forever to get to feature parody. It was the worst project. So many people rotated in and out of it. Everyone thought it was dumb. Sunk cost fallacy, just the worst. And it’s because, A, we got arrogant and we thought we could do it. B, we skipped discovery. We didn’t really write a one pager. We just went for it. We didn’t do enough technical and design research into what the requirements would actually have to be. And there you have it.

Lenny: And did not work out, or was it a huge success in the end and it changed the trajectory of the business?

Maggie Crowley: Absolutely not. But you know what? I didn’t get fired, so it’s fine.

Lenny: I feel like I’ve gone through those experiences and then three, four years later, it’s like another… Maybe this rewrite and redesign may work. We haven’t updated this thing in a long time.

Maggie Crowley: Just don’t do it. Don’t rewrite. If anyone ever tells you to do a rewrite, don’t do it. A side-by-side rewrite, nope.

Lenny: Yeah, I’ve never had… What I run into is once you get too far down a redesign slash rewrite, everyone’s building in that new world, and then you launch, and experiment’s negative, and then it’s just like, “Oh, we just got to launch it. We’re going to call it back. We’re going to figure out how to get back to neutral someday.”

Maggie Crowley: Yeah yeah. Don’t do that.

Lenny: Good times.

And that is a wrap. I hope you enjoy these stories of failure. I want to give a huge special thank you to all of our amazing guests for being vulnerable and sharing these stories of failure in their career. I hope you leave this episode with a new perspective on how setbacks and challenges and failure can often be exactly what you need to get to the next step of your career or your life. If you’ve got a great story to tell about failure, I’d love to hear it. Leave a comment either on YouTube or on Lennysnewsletter.com or just DM me on Twitter. Or you could reach out at LennyRachitzky.com and click the big contact button. Thank you for listening. Bye, everyone.

Glossary

Reformatted by reformat_english.py