这将让你成为更好的决策者 | Annie Duke(《Thinking In Bets》作者,前职业扑克选手)
This will make you a better decision maker | Annie Duke (Thinking In Bets, former pro poker player)
Annie Duke: It’s so incredibly necessary in improving decision quality to take what’s implicit and make it explicit. It’s not that intuition is crap, your intuition is sometimes right. If you don’t make it explicit, then you don’t get to find out when it’s wrong.
Introducing the Guest
Lenny Rachitsky: When you look at companies that have read your book, what do you find are the brainwashing tactics that really stick?
Annie Duke: People generally think the purpose of a meeting is for three things, discover, discuss, decide. The only thing that’s ever supposed to happen in a meeting is the discussion part.
Memories of Daniel Kahneman
Lenny Rachitsky: Something that comes up in product a lot is this idea of pre-mortems.
Adversarial Collaboration with Matthew Killingsworth
Annie Duke: So a pre-mortem, it’s great only if you set up kill criteria. Commit to actions that you’re going to take if you see those signals.
Decisions and Quitting
Lenny Rachitsky: You have a very interesting framework for how to think about decision quality when the outcome is very long-term.
Advice on Parenting
Annie Duke: There is no such thing as a long feedback loop. And the way you choose to shorten the feedback loop is to say, what are the things that are correlated with the outcome that I eventually desire?
Lenny Rachitsky: Today, my guest is Annie Duke. Annie is the author of the bestselling book Thinking in Bets, and also her more recent book Quit: The Power of Knowing When to Walk Away. She’s also a special partner at First Round Capital, which we spent some time on and is incredibly fascinating. Prior to this part of her career, she was a professional poker player. She’s won over $4 million in tournaments, including winning a World Series of Poker Bracelet, and she’s the only woman who’s won the World Series of Poker Tournament of Champions and the National Poker Heads-Up Championship. Currently, she spends her time helping companies make better decisions. In our conversation, we cover the many lessons that she’s learned from her friend Daniel Kahneman, who recently passed away. What simple change she’s found has the most impact in a company’s ability to make better decisions, how to make better quick decisions when the feedback loop is very long, and also why she doesn’t actually believe in long feedback loops, how she changed the way that the partners at First Round Capital make decisions, which is incredibly interesting.
Now you can assess risk, secure the trust of your customers, and automate compliance for SOC 2, ISO 27001, HIPAA and more with a single platform, Vanta. Vanta’s market-leading trust management platform helps you continuously monitor compliance alongside reporting and tracking risk. Plus, you can save hours by completing security questionnaires with Vanta AI. Join thousands of global companies that use Vanta to automate evidence collection, unify risk management, and streamline security reviews. Get $1,000 off Vanta when you go to vanta.com/lenny. That’s vanta.com/lenny.
Resulting and Parenting
Annie Duke: Thank you so much for having me.
Lenny Rachitsky: It’s my pleasure. I was telling a bunch of people that you were coming on this podcast and every single one of them was so excited that we were doing this, so I’m quite excited that this is happening. I want to start with a question about Daniel Kahneman. I know you two were very close. You did a podcast together. I know you grew to be good friends over the past few years. Sadly, he passed away about a week ago at this point.
How Much Can Decision-Making Improve?
Annie Duke: Yeah.
Which Decision Frameworks Work Best?
Lenny Rachitsky: I’m curious, what are some lasting lessons that you took away from your time with him? Maybe one or two things that stick with you and you think will stick with you?
Restoring Independent Judgment in Decisions
Annie Duke: It’s hard for me to describe properly how humble this man was, and it seems so impossible because his influence was so huge and his intellect was so huge and his insight was so change-making in the areas that he worked. But he was so humble, he really wanted to hear what you thought. He said, “I don’t know a lot”, he would change his mind. It’s funny, I got asked in a class that I was teaching about thinking fast and slow and someone was talking about how there’s a lot of studies in there that didn’t end up replicating.
And what I’ve seen from a lot of other people in the field who really where their work is defined by some line of work that didn’t replicate, where they’re really arguing against the fact that it doesn’t replicate. Just trying to justify or say, “But no”, or this and that. This real lack of open-mindedness to the idea that you know what? Maybe it turns out not to be true and that should be okay. But the thing about Danny was when you talked about it, he was the first one to tell you, “I wish priming wasn’t in the book. It doesn’t replicate.” He was just so open-minded to the idea that he knows what he knows now, but he knows that he doesn’t know everything.
Another thing that people don’t know about him outside of academics I think are that he was one of the pioneers in something called an adversarial collaboration. So one of his favorite things, and he in some ways invented it, was find someone who really disagrees with you and then write a paper with them and try to figure out with them what’s the study that would resolve the issue. So an example of that was Danny did work on happiness as it relates to wealth. Kahneman and a collaborator, Angus Deaton, they published a result that got a lot of press about the relationship between money and happiness, and the idea was money doesn’t buy happiness beyond 75,000 now because it would adjust for inflation. But it was like the idea being once your basic needs are met and you’re not that worried about paying rent and so on and so forth, it’s not going to have that big of an effect.
But then a decade later, Matthew Killingsworth said, “That’s not true that actually money and happiness are correlated through all the income levels. The more money you have, the more happy you tend to be.” Okay, so this is a classic example where people will tear each other down sometimes and that guy’s wrong and whatever. And instead, it was Kahneman’s like, “All right, let’s do it together.” And they joined forces along with Barb Mellers to basically try to resolve that issue. He was just always seeking out, it was just like everything was like, why am I wrong? And he was eager and he was excited and he wanted to…
I remember having a lunch with him where I realized after the lunch that the whole lunch was him asking me questions about my work. Why is he asking me questions about my work? Right? He’s Danny Kahneman. But he was just so curious about other people and he just made time for everybody. He just loved the people around him so much. And I think that in the end, it’s hard to believe, but I just think that that dwarfs his body of work, which is spectacular. But he as a human being was just so spectacular. And I do think you hear all the time about people who are so incredibly brilliant. It’s the brilliant but asshole thing, and it was just so the opposite with him. He was so kind.
Lenny Rachitsky: That’s amazing. And it feels like there’s probably a strong correlation between people that are curious, always looking for how they’re wrong, and finding time to spend it with people to debate and discuss and ask. Feels like there’s going to be a strong correlation with that and finding really interesting insights uncovering new things.
Why “Alignment” Is a Bad Word
Annie Duke: Yeah, yeah, yeah. [inaudible 00:09:12].
Lenny Rachitsky: Well, thank you. Thank you for sharing all that. I wish I had known him. Let’s talk about decision-making. So there’s two areas I want to spend time on. As one would not be surprised, decision-making and quitting. And I know that you’ve been on a lot of podcasts. I imagine many listeners have read your book or have heard a lot about the stuff that you teach. I’m going to try to come at this from a bunch of different directions and do it a little differently. First of all, I know you’re a parent. As last I heard, you have four kids. I just had a kid, he’s almost 10 months now.
Shifting from Coercion to Curiosity
Annie Duke: Oh, congratulations.
Lenny Rachitsky: Thank you. I’m curious, what your frameworks you find most helpful in parenting and in raising your kids that have stuck with you there?
Feedback Loops in Venture Capital
Annie Duke: Well, let me just first off say I get a lot of people who are pregnant who come and ask me for my advice about parenting. And I’ll tell you the two pieces of advice I give people. The first is this, there’s all sorts of parenting books, there’s all sorts of styles. A book needs to sell, so they need to sell you something new or different or whatever, okay. But the only thing that matters really is that your kids know you love them, really know deep in their bones, know you love them.
And we can argue breastfeeding versus not breastfeeding or co-sleeping versus not co-sleeping or attachment parenting or sleep training, we can talk about all that stuff, right? Do you homeschool? Do you send them to private school? Do you send them [inaudible 00:10:49]? We can have lots of [inaudible 00:10:51], but it all dwarfs in comparison to your children know that you would lay down in the street in the front of a bus if it meant that they would be alive and happy. That’s the number one thing I say. Number two is at some point, you will drop your baby on its head. So I bet this has happened to you already.
No Long-Term Feedback Loops
Lenny Rachitsky: Well, yeah. Sort of. Okay.
Self-Narratives and Feedback Loops in Decisions
Annie Duke: Sort of, right.
Lenny Rachitsky: Keep coming.
Revamping First Round’s Decision Framework
Annie Duke: So here’s the thing about babies, is babies can’t move. And so you get very used to as a mom or a dad or you’re sitting on the couch and you’re folding laundry and your baby is right here next to you and your baby can’t move. And then one day without warning, your baby can roll over. And you think it’s fine because you put a pillow there or something, but you turn your back for a second because you’re like whatever and then your baby falls on its head. It happened with every single one of my children. And they don’t fall that far and whatever, but they’re totally fine and you’re not a bad parent. And it’s just this, there are things you don’t know and stuff’s going to happen and it’s not going to be perfect. And every little thing isn’t going to have some lasting impression on your child because at some point, you will drop your baby on its head and you will not mean to, you didn’t intend to.
But in my case, it was always like, oh my gosh, my baby can roll over. I did not think that was going to happen today. And gosh, by the time I got to the fourth one, it’s like I’m chasing after one baby to try to get the diaper on and this and that, so then the other baby hits its head. So I think there’s just a lesson in that which is like it’s a small thing, don’t sweat it and don’t get mad at yourself because you made a mistake. It’s like the mistake would be, oh, I realize my baby could roll over now. So now you should make sure that you’re taking extra precautions about your baby rolling over off the couch. But it’s really just a funny way for me to say your baby’s going to hit its head or walk into a coffee table or get cut or whatever. It’s like they’re very resilient and you didn’t do anything wrong as a parent, so don’t beat yourself up about that. And then I do actually have one other thing I say.
Unexpected Findings from Analysis
Lenny Rachitsky: Yeah, go for it.
Surprising Discovery on Decision Scoring
Annie Duke: Be having four, you have very little, you have no influence whatsoever basically on what your child’s personality is. What you can do as a parent is make sure they say please and thank you. So what I find with parents of one is particularly if that one is very well-behaved, they think they’re a great parent and they may be. That would be resulting. They may be, but once you have four, what you realize is like, oh, my parenting really didn’t have anything to do with it. But my children do say please and thank you.
Premortems and Kill Criteria
Lenny Rachitsky: This is really valuable, good advice. I need to hear this. My wife is going to need to hear this, too. She’s always worried about the soft spot on our kid’s head and getting hit into some things.
Annie Duke: No, it’ll be fine.
Making the Implicit Explicit
Lenny Rachitsky: Okay, great.
Annie Duke: It’s actually it’s part of why it’s there. Really, your baby will be fine. I mean, that’s the thing. We used to cart babies around in the wilderness, we were living in case. We’re fine, we’re built for it. We’re built for a much rougher world than the world that babies actually grow up in now. I will say my oldest child when she was in high school, just for whatever reason, she had no interest whatsoever in alcohol or drugs or anything like that. And I remember just feeling so smug about how well I had raised this child and how great she was, and oh, these other parents who clearly weren’t doing a good job. I mean, I didn’t really but because it’s hard not to. And then my next child went through high school and I was like, [inaudible 00:15:02]. That’s just a temperament thing.
Thoughts on Quitting
Lenny Rachitsky: It’s a good example of resulting in action.
The True Cost of Quitting
Annie Duke: It is a good example of resulting, but all my kids have turned out fine and they’re all very different, but yeah. But as far as decision-making is concerned, which is a totally different thing, I’ll actually quote Danny Kahneman on this one. Nothing is as important as it seems when you’re thinking about it, that’s a really important one. One of the things that I used to try to do with my kids all the time was mental time travel, which is actually a very good decision tool. So they would be really upset about something and it could be something that happened at school or it could be like they were grounded for something or whatever.
And the thing is when you’re in the moment, it just feels like so big. And the thing that I used to say and they say all the time now is this is going to be so great for you when you’re 40 at Thanksgiving. You’re going to be able to tell these stories to your children and it’s going to be the best, like you should be thanking me so that you can talk about your crazy mother at the Thanksgiving table. And it allowed them to get some time and space from it to realize this is going to be funny at some point.
At some point, it’s going to be a hilarious story that you tell people. No matter how horrible it seems now, it’s going to be a hilarious story at some point when you’re older. And it sounds like a fun thing I was doing with them, but it’s actually a really good decision-making tool for us in general, right? Look, there are absolutely things that 20 years from now are going to matter, for sure. But most things, no. If you think about most things that ever happen to you, like you’re 21 and your girlfriend breaks up with you or whatever and you just think it’s the end of the world and you’re never going to recover and all these things, but if you could get yourself like, well, how do I really think I’m going to think about this in 10 years? Am I still going to be heartbroken in 10 years?
Probably not. And I think that we need to get that perspective of time so that we can get out of the moment because in the moment it just feels so important and the feelings are so big. And it’s like the focusing effect on this second and the feelings you’re feeling right now are so huge that we forget the scope of time. And I think that that’s absolutely one of the best things you can do with your kids, and you can do it in small ways too, right? You can choose to play this video game or you can choose to study for your task. A week from now when you get that test back, how do you think you’re going to feel about those two choices? Right?
So that was a tool that I used a ton and it’s just generally an amazing decision-making tool, just generally for people. It’s one that I happen to use with my kids all the time. And then I would say that the other one was using the word nevertheless, which is this is a great leadership skill, the word nevertheless. So let’s say my child got caught doing something wrong. I don’t know, I found a bag of red solo cups in my backyard that they forgot to throw out after a weekend that I was away. That’s hypothetical. So I’m grounding them and it’s a lot of argument back, right? So they think this is a debate and they’re giving me all of their input and their opinions on why it’s unfair and all the other kids do it and the other parents don’t get mad and whatever, whatever the argument is. You have to have the balance between them feeling heard, which I think is incredibly important that your children feel heard, and following through on what you know or believe is right.
So it’s I hear you. Nevertheless, you’re grounded for two weeks. I hear what you’re saying and I understand. Nevertheless, this is what’s going to happen. And obviously, the words that you can use for that might be different. You have so much more authority obviously over your children than you do in other places. But in the workplace, this is very good because employees gripe all the time at decisions that you make in leadership because they think they’re right and they want their way. And to have the ability to say, I heard you and your input, trust me, was incorporated into the decision, nevertheless, this is the path we’re going to take. Right or wrong, that’s what we’re going to do. So I think there’s a lot of things that basically you can say. These are good things to do with kids, but they’re actually just generally really good decision strategies.
Conclusion and Resources
Lenny Rachitsky: I’m taking notes on all these things. These are going to be extremely useful.
Annie Duke: Nevertheless is a really good one with children.
Lenny Rachitsky: It connects to something Dr. Becky teaches, if you know her, of telling someone, “I believe you.” When your kid says something and they’re upset about something, start with I believe you.
Annie Duke: Nevertheless.
Lenny Rachitsky: And here’s nevertheless, exactly. Fair enough. I think we’re going to start having to pivot this podcast to a parenting podcast because there’s so much stuff here, but I’m going to try to resist. So we’ve been talking about decision-making and frameworks and things like that. Something I wanted to ask you is how much better can someone get at making better decisions? So for example, say someone listens to all of your podcasts, reads your book, studies it intensely. What’s the delta you find in somebody being able to make a better decision? And where [inaudible 00:20:35] comes from is Daniel Kahneman of all people. People ask him, “Do you just live such a optimal life now that you know all of these biases we have and all these mental errors we make?” And he’s like, “No, I can’t…” I think he said famously, “I don’t, actually. I can’t use these in practice. This is just stuff that I have learned, but it doesn’t actually impact my day to day.” So I’m curious just how much better can someone get? What kind of delta have you found in terms of making decisions?
Annie Duke: Okay, so it depends on whether you actually do the things. I mean, I think that that’s what the issue is. I think Kahneman did some work way back that started him on this journey. That was basically work on hiring and it was taking it from completely unstructured Lenny, saying, “I just know a great product manager when I see one”, whatever that means, to me going to Lenny and saying, “Okay, I understand that you think that you see a great product manager you know one when you see one”, but can you explain to me what that means? In the abstract, what are the things that you’re looking for in someone that you want to fill that role? And we can then excavate that, right? And make what is implicit because you’re applying some implicit model to how you’re thinking about the person that you’re interviewing and how they map onto the role that they’re going to be, but let’s make that explicit.
So we can make that explicit, we can turn it into a decision rubric, we can create a structured interview process out of that. And then what he found was after you’ve gone through that process, if you then use your intuition after having done that, not before, that then you actually can really drastically improve your hit rate on hiring. And in that case, it was from about a 50% hit rate to 65. So that’s pretty huge, so that’s a really big difference. The problem is nobody does it. So I can tell you that most of the conversations that I had with Danny that were about my work in particular were just him saying, “How do you get anybody to do it?” Now that’s not to say that I would be better at that than him, it’s that he didn’t do my job, right? So he was an academic doing research and so on and so forth, I’m living in companies embedding for years.
I’ve been with one client for five years, one client for four years, one client for four years also, almost four years, three and a half years. And then I just took on a new client who I’ve been with for six months, and that’s my whole client roster because I stay with them so long. So if you do it, the answer is quite a bit. Like at least in terms of Danny’s work in relation to something pretty noisy like hiring, you went from 50% hit rate to 65% hit rate, which is huge. It’s enormous. The problem is that the way that you can make decisions in order to be better at them is not supernatural to the way that humans make decisions.
I think we think a lot more highly of our intuition than we really ought to. We think very highly of our ability to notice things in the moment and act rationally toward them in a way that really we ought not to. We tend to think that we have insight that other people don’t have when actually the other people probably have more insight into our situation than we do. So I think the answer is a lot with a big if, with an if you can get people to do it. And I think that’s where the issue comes in. Now, the good news is that you can then reverse that and say if I’m willing to do it, think about what an edge I’m going to have over people who aren’t willing to do it, right?
So we can think there’s people who don’t know about it, there’s people who do know about it but don’t do it, and then there’s people who do know about it, but do do it. And it’s that last group that’s so tiny. So the answer is I think a lot, but nobody does it.
Lenny Rachitsky: So maybe following that thread, when you look at companies that you’ve worked with or companies that have read your book and really dove in and started to implement some of your piece of advice to make better decisions, if you could look back at the ones that had an impact, what do you find are the mental models or frameworks or tactics that really stick that most often have an impact and the biggest impact?
Annie Duke: I think the one that’s easiest to implement is this. And it’s so easy, and I just wish more people would do it. The best way to get somebody’s opinion is independently of other people’s opinions, independently asynchronously. So the way that I put is I want people to stop talking to each other so much. When we think about what did people generally think the purpose of a meeting is, they think it’s for three things. Discovery, which is I want to discover what your opinions are, what your judgments are of something, right? So I want to find out what you think about something. So for example, if we’re in product development and we’re trying to figure out a timeline, we’re trying to develop a product roadmap and we’re trying to figure out how long it’s going to take to release certain features or something like that, we all come into a room and we start yelling it out together.
It’s so bad for decision-making, I can’t even tell you. There’s cross influence, the loudest person in the room tends to then have an outsized influence on the decision, the most confident person in the room tends to have an outsized influence on the decision. And that’s great if the most confident person is also right, but the problem is that’s not always so. And so that’s the discovery piece, right? And we tend to do that in a group. And then there’s discussion, which is I’ve now discovered the way that Lenny is modeling this problem or what his judgments are about certain things or his forecasts are, his estimates are. And now, we’re going to discuss those ideas and we’re going to discuss your ideas in comparison to my ideas, and that’s going to happen in the group setting. And then there’s also now, we’re going to decide in the group setting. So we’re going to make some set of decisions about say what the roadmap is going to look like.
Here’s the thing that I think is the easiest to change, is to realize the only thing that’s ever supposed to happen in a meeting is the discussion part. So we should absolutely be coming together as human beings to discuss everybody’s judgments and opinions and the way they’re modeling the problem and their forecasts. In particular, it’s really good to come in and discuss the places where people are different. So you’ve been in many meetings, I’m sure about 80% of the time is double clicking. Oh, I agree. And I want to now we’ve literally just used different words to say the exact same thing because I’m in agreement.
If we can focus the discussion on places that people disagree, we’re much better off. But so now we can take this, discover, discuss, decide. So I’m saying discuss is only supposed to happen in the meeting, that’s the only thing that’s supposed to happen in the meeting. So what’s happening with the other stuff, the discover and the decide? Well, and this is the thing that I think I actually have been able to get people to do is discover what people think before you get in a room independently of each other. So how would you do that? Well, let’s imagine that you’re going to have a meeting about the product roadmap.
So you would say to yourself in this meeting, what are the opinions that I need to get from the people in the room? It could be a brainstorm. What are all the different features that we could develop? So it could be a brainstorm. Fine. Write to them independently and say, “Hey, free for off. Just come up with all the different features that you think would be reasonable for us to consider developing and then give me a forced rank from best to worst of your own ideas with some three to five sentence rationale as to why you have these things in this order.” And so I could do that, but maybe we already have a list. I can send that out for a forced rank to everybody. Okay, so here’s the final list of things we’re really considering. Prioritize those for me, just force rank them. And then again, give me a rationale.
Give me a little bit of free writing as to why you think this should be, that you have these things in this order. Maybe we’ve decided now on what our top five priorities are. Great. I could, before the meeting, send the top five priorities out. “Hey, we’ve decided on this stuff.” For each thing, I’d like to understand what you think a reasonable timeline is, how many sprints it’s going to take, so on, so forth, right? So we can ask for that type of information so that we can start making estimates because that’s going to affect budget and what we’re going back to the board with and so on so forth. But regardless, just figure out what is the thing that we’re going to be discussing in this meeting? And I’m going to send it out to everybody independently and I’m going to say, “Don’t reply all, just send it back to me independently.”
If it’s a repeated decision, you can actually create a rubric that lives on Airtable or Coda or Google Sheets or whatever, I don’t really care. And people can input their decisions there where they can’t see anybody else’s decision. So with Google, you can use Google Forms in order to do it, and then it dumps into a spreadsheet that only you can see. That’s a great way to do it. So anyway, so you do that and now you can now see everybody’s opinions that you then now send out to the group and say, “Everybody, look this over before we come in and discuss.” So notice you’re still working as a group, but you’re working as a group where you’re not in the same room together and talking at the same time. And there’s a word for that, which is nominal group. So it’s a group that at that moment is working independently and asynchronously of each other.
So if you can get people to do that, and I do have companies where I don’t consult with them, but I’ve just come in and talked to them briefly or whatever that do actually implement that piece of it. That is a huge piece of it, that’s ginormous. And then you do the same thing for deciding. So the decision should not be made in a room, it’s made either. I love the one decision maker model, but not everybody’s down with that. But you can have a vote forum where people go vote in private about the way they’re leaning.
You can do a variety of things with that, but just don’t do it. Don’t do it in the meeting. And then just the last thing that I’ll add, which is a muscle that you really have to exercise, is I think it’s really important to understand that the word alignment in terms of we’re all aligned as a group, right? The word alignment is stupid and it shouldn’t be used. And I know I’m saying that very harshly, but it’s true. It’s dumb because it doesn’t exist. You have 10 people in a room and they’re all really different people with different opinions, and they’re never going to come out of the room agreeing with each other. And it’s really bad if the expectation is that they’re supposed to, and it’s really bad for a few reasons. One is it isn’t reality, and that I don’t like coming out of things without reality actually kicking.
So it’s not reality. People don’t actually agree, they’re not actually aligned on the decision. That’s just the thing that makes you feel better, right? So I think that that’s problem number one. But problem number two is that if the goal is alignment, if the goal is agreement, then the meeting becomes coercive and you never want that. So the way that I’m supposed to talk about my ideas is to convey why I believe what I do, not to convince anybody that I’m right because if I’m working to convince people that I’m right, it becomes coercive and that’s horrible. So you have to get comfortable with walking out of the room. This is the nevertheless. Walking out of the room understanding that once you have that discussion, it’s not that Lenny, your opinion might change. It could, right? I could say my thing and it could be just so damn brilliant that you change your mind somewhat, right?
So you come maybe you’re like, “Oh, I’m thinking about this differently and actually”, but maybe you don’t and you still believe a thing that’s very different than me. And leadership has to say, “That’s fine. I’ve heard both of you and I know that this isn’t going Annie’s way. Nevertheless, trying to think about what all of our goals are, this is what the decision is going to be and it’s totally fine that you ended up not agreeing with each other because it’s reality. And what it allows me to do is get a better sense of what the space of decisions is.” So those things I have been able to get people to do and they’re actually quite impactful.
Lenny Rachitsky: Imagine a place where you can find all your potential customers and get your message in front of them in a cost-efficient way. If you’re a B2B business, that place exists, and it’s called LinkedIn. LinkedIn Ads allows you to build the right relationships, drive results, and reach your customers in a respectful environment. Two of my portfolio companies, Webflow and Census, are LinkedIn success stories. Census had a 10X increase in pipeline with the LinkedIn startup team. For Webflow, after ramping up on LinkedIn in Q4, they had the highest marketing source revenue quarter to date. With LinkedIn Ads, you’ll have direct access to and can build relationships with decision-makers, including 950 million members, 180 million senior execs and over 10 million C-level executives. You’ll be able to drive results with targeting and measurement tools built specifically for B2B. In tech, LinkedIn generated two to five X higher return on ad spend than any other social media platforms.
Audiences on LinkedIn have two times the buying power of the average web audience, and you’ll work with a partner who respects the B2B world you operate in. Make B2B marketing everything it can be and get $100 credit on your next campaign. Just go to linkedin.com/podlenny to claim your credit. That’s linkedin.com/podlenny. Terms and conditions apply. There’s an implication here you touched on a bit that there’s a DRI essentially, there’s one decision maker. Sometimes people start to feel like, “Oh, my voice isn’t heard. I don’t have a lot of say, I can’t be part of this decision.” And you talked a bunch about just how to make people feel included, you get feedback along the way. But any advice there if you try to move to this model of making people feel like, “Okay, I actually have impact on where this goes”?
Annie Duke: Yeah. So here’s the really wonderful thing about moving from a coercive model to really, I guess you could say a model of curiosity, right? You want to be curious, not coercive, which means that the way that people in the meeting are talking about what their opinions are is in the mode of conveying information, not trying to convince anybody. So once we move away from that coercive model, and when I say coercive, I’m not saying anybody is purposely trying to set up a culture of coercion. Sometimes that’s true, but for the most part, everybody’s trying to do a good job and nobody’s trying to set up a culture of coercion. But as soon as you say, “Are we all in agreement? Are we all in alignment?” And as soon as you’re allowing people, and I’m sure you’ve been in these meetings, right? If you allow people to interrupt, if you allow people to say this, “I think you’re wrong”, “I disagree”, “Here’s why”, those are all very coercive things to have happen.
Interrupting someone is silencing them. Saying, “You’re wrong”, well now, you become tribal and people aren’t going to be open-minded and they’re going to stake their ground and it’s all really bad. So let’s move away from that. Number one, that’s already going to help. But when we think about the way that meetings normally happen, again, there’s all this crosstalk and some people aren’t speaking and some people are and so on, so forth. And then of course, not everybody feels equally heard. But when you work as a nominal group, let’s talk about something as simple as we’re going to make a forecast of how long it’s going to take to launch this product feature. Okay. So I’m going to send out to everybody, what’s your point estimate? What’s your lower bound? What’s your upper bound in terms of timeline?
And you could do it in some way, like how many sprints do you think is going to take whatever language you want to use? Now, everybody independently now gives their forecast with a rationale for why they believe that. And then you come into the room and you run a discussion where everybody’s getting to say what their estimate was and why they believe that, and people are getting to ask questions. “I have a question” always is a clarification, it’s just I don’t understand. And as the leadership, the way that you would do it is, let me give you an example from a real one because that’ll be easy. I did one of these discussions for a question about remote work, like what did the company want in terms of remote versus hybrid and that kind of thing. And so there was a lot of disagreement about whether whatever the policy was, it should be consistent across functions. Lots of disagreement.
So let’s take somebody who was on the, no, it shouldn’t be consistent across functions. So they now say why they believe it shouldn’t be consistent across functions and they say things like, “Well, different functions have different requirements, right?” So there’s some functions that have to be in the office, like if you’re IT as an example, but there’s other functions which are more collaborative and creative and whatnot where it makes sense for people to be in the office, whereas engineers doesn’t really matter, right? So there’s different needs of different functions in terms of how much they need to be in the same space. So I’m not agreeing or disagreeing with this, I’m just saying what somebody said. So now as a facilitator, I never say, “Oh, I agree.” What I say is I just want to make sure I understood what you said and I reflect it back.
So I say, “So what I heard you say is that not all functions are created equal in terms of how well they work remotely or how well they work in person or what the flexibility might be. So what you’re saying is there’re functions that have to be in person. Period. And then other functions where that collaborative element being in the same office would be more important versus some functions where being remote is totally fine. Is that what you meant?” And then they have the ability to say, “Yes, that is what I meant” or, “Actually, I meant something slightly different. Let me say what that is”, and then I reflect that back. Okay, so literally you’re just going around, you’re calling on people to do that, and then you’re reflecting back what they say without offering your own opinion. I don’t know, Lenny. Tell me how someone doesn’t feel heard in that situation.
Lenny Rachitsky: Yeah. Absolutely. I would feel so good if somebody just clarified and made it clear they know exactly what I’m saying, even if the decision doesn’t end up the way I want.
Annie Duke: Right, so that’s the thing. And then what actually ends up happening is that the people in the room feel more, the psychological term would be endowed, more endowed to the decision. In other words, they feel like they have ownership over the decision. And whatever the decision is, they generally will see their selves reflected in it because they were heard. And they also will generally understand that nobody, no one really ends up with exactly every single thing that they wanted to see in the decision because they also get to see the true spread of opinions on the team. And what you see in that situation is that there’s lots and lots and lots and lots of disagreement, which you don’t see if you talk in a group. It narrows the space, right?
So for example, particularly if you’re senior, if you say, “I think this is going to take three months to launch” and I was thinking four weeks, you’re never going to hear four weeks from me, ever. But if we get those opinions independently, you will actually hear that I think it’s going to take four weeks. You may tend to be more right there being more senior, but I may have something interesting to say. You should hear what I have to say. And this also allows me to learn from you, too. But because I haven’t heard your opinion first, then I’m not going to conform my opinion to yours. So it actually spreads the surface area of disagreement that you see on the team, which then makes people feel much better about the decision not being exactly what they want because they recognize like, “Oh, this is actually a hard problem. People really disagree on this stuff.”
Lenny Rachitsky: Amazing. I think this is going to be really helpful to a lot of people. Just to close the thread on this little summary maybe is core advice here is brainstorm separately. So I guess first, there’s discover, discuss, decide. To discover ideas completely independently, basically brainstorms. You’re a big advocate of brainstorming independently, sitting on your own, thinking through ideas. And then bringing people together to discuss all the things they’ve come up with and especially where they disagree. And then ideally having one person that makes a decision once she or he has taken all the input.
Annie Duke: Yeah, you can have one person. I mean, I work with people where it’s like a partnership and there’s six people who are going to vote, but they still do it independently. They go to a forum. So I don’t know what your vote is. And all six people don’t have to agree. I mean, I just think it’s really important that once you get above an end of one, you shouldn’t necessarily expect the people to agree, which I think is just really important. And this is good for more than just brainstorming. It’s for forecasting, any kind of project planning, budgeting. Yeah, I mean, we do this, for example, at First Round, we have a structured forum for evaluating a company in terms of whether you should invest in it because there are facts and then there are the way that you model those facts in. So it’s a lot, what are the investor’s opinion of the founder and the product and that kind of thing.
And we just have structure around how we’re eliciting those opinions, which is really helpful. The other thing that I’ll just add to what you said is that you can actually be in the same room together and still discover information independently. So this will happen on the fly all the time where people are talking and they start talking. Something comes up, like someone suggests some new feature or something like that, and then people start saying, “Yeah, but that’s going to take…” And you go, “Stop. Okay, everybody take out a piece of paper.” So you can still get that same independence on the fly as well as doing it in advance. But yes, that was a very nice summary.
Lenny Rachitsky: If this is just the one thing people take away from this conversation already, I think that could have a big impact. Speaking of First Round, one, we actually just had Todd Jackson on the podcast talking about-
Annie Duke: Love Todd.
Lenny Rachitsky: … love Todd, talking about product market fit. That episode will have come out before this episode. Also, so I asked Brett Berson, your colleague at First Round, what to ask you.
Annie Duke: Oh, okay.
Lenny Rachitsky: Yeah, yeah. By the way, your title of First Round is amazing, special partner. I’ve never seen that before.
Annie Duke: I am. It’s a title just for me.
Lenny Rachitsky: Oh, my God.
Annie Duke: I adore Brett Berson.
Lenny Rachitsky: Me, too.
Annie Duke: So I’m interested, I’m interested in what he said you should ask me.
Lenny Rachitsky: Nothing too spicy, it’s along the lines of what you talked about. So you said that you have a very interesting framework for how to think about decision quality in the short term when the outcome is very long term. For example, investing. Also, many decisions we make in business, things you need to decide now that you only find out years from now. Can you talk about your advice there and your insights here?
Annie Duke: Oh my gosh, I’m so happy that that’s the question that he asked.
Lenny Rachitsky: Yep, I see.
Annie Duke: Okay, so can I give a tiny bit of background to this?
Lenny Rachitsky: Absolutely.
Annie Duke: So prior to talking to First Round, I have another client too who, they’re amazing, Renegade Partners at Roseanne Wincek and Renata Quintini. They’re incredible. Before finally hooking up with them, and they work at different stages, First Round is C, obviously. Renegade is more like A B, tiny bit of dabble in C. But before running into them, I talked to quite a few venture firms who are interested in talking to me post-Thinking in Bets having come out, so this would be 2018. And there was a theme, there was a theme across them all. The first one was, well, the kind of decision-making you’re talking about we don’t need to do because we just know a good founder when we see one. So that is a sentence that came out of many people’s mouths. And as I just said to you, okay, I have no doubt, but don’t you want [inaudible 00:46:52] make that explicit?
There’s all sorts of great things that come from making it explicit, not just in terms of the increase in decision quality in the moment, but it actually allows you to close feedback loops much better. So that was one thing that I found quite surprising. But the one that I really found very interesting was being told, “Well, what you’re talking about doesn’t apply because our feedback loops are a decade.” And in poker, you got an answer right away. You won or lost the hand right away. So the way that you’re thinking about decision-making doesn’t really apply, until I met First Round and then Renegade where they actually heard what I had to say because I gave the same answer to everybody. So we’ll just put aside that wouldn’t you want to make that explicit. The first thing that I would say is, oh, poker is much noisier than you think because when I win a hand, I have no idea why.
So I do actually have to wait a long time because I have to play many, many, many hands before I actually know, do I actually have an edge? Because I actually don’t know very much. On one hand, for one thing, I almost never see my opponent’s card, so I’m left in a dust of uncertainty. But separately from that, the main thing that I said was, how could you possibly think that the feedback loop is 10 years? And this is what I think really caught First Round’s eye because when I was talking to Josh Kopelman about it, he said, “Well, what do you mean? We don’t get an exit for 10 years.” And I said, “Oh, I’m sorry, do you invest? And then you go to sleep like Rip Van Winkle? And then you wake up 10 years later and you go, ‘Hey, how’d that go?’ Or are there all sorts of things that happen in between?” The simplest thing, the simplest thing is does it fund at Series A?
And the little pushback that I would get there is, but we’re not investing for Series A. And I say, “Well, I know that”, but have you ever had a company that exited for more than a billion dollars that did not fund at Series A? And the answer is no. And I’m like, okay. So it sounds like that’s necessary. Might not be sufficient, but it’s necessary. And it’s certainly a signal that is actually more highly correlated with exiting out well than the investment at C. And then, oh, right, you have series B. And that’s separate from all the other things that you can look at, like what you talked to Todd Jackson about. Is it achieving product market fit? We know that eventually for it to be successful, it’s going to have to achieve product market fit, right? So you can look at what’s happening with that, just general things about traction, what’s happening with net new ARR, ability to retain top talent churn.
I mean, there’s so many different things that you can look at, all of which are things that you know must happen in order for the big thing to happen. Okay, so what that means is that this is the big, I’m going to make a bold statement here. There is no such thing as a long feedback loop. You can make a decision about how long the feedback loop is. That is your choice to live in a long feedback loop, and you can choose to shorten the feedback loop. And the way you choose to shorten the feedback loop is to say, what are the things that are necessary but not sufficient? That’s one thing, for getting a good exit, or what are the things that are correlated with the outcome that I eventually desire? And what that means is that when you’re at the decision point, right? Like in First Round’s case, I’m going to invest in a company.
What you have to understand is that you are making a prediction about how the world is going to unfold, how the future is going to unfold. And those things that you’re predicting. You can track, and you can track them back to the decision and you can do it pretty darn fast, mind you. I mean, think about being in 2021. There were companies that were raising in A six months after seed. Today, it’s a little more 16 months-ish. But even so, let’s just say that that was the only thing that you decided to do. I’m going to forecast the probability that this company’s going to fund at Series A. And then obviously, those companies start to fund or not fund at Series A and you’re finding that out in 16 months. Here’s my question for you, Lenny. Is 16 months shorter than 10 years?
So it’s probably why he said to ask me that question because I just really do. I mean, I have a very strong opinion about this. The feedback loop is as long as you choose it to be. And if I take that back to some of the things that I heard early on when I was talking to people, what I would say is that I think that there is a certain amount of psychological safety in allowing the feedback loop to stay long because really of two main factors. One is that, look, if I was early into Uber and now I’m a celebrity investor or something, I don’t really want to know if I’m good or not. Do I? Right? I don’t really want the world to know that. I mean, if I’m good, that’s great. But it feels like they already believe that I’m good because I happen to be early into Uber.
So since people already think I’m good, I’m just losing to that decision, psychologically speaking. Not investment quality speaking, but psychologically speaking. Okay, so here’s the problem though. Why was I early into Uber? Did I have an insight into a real pain point in a developing market, blah, blah, blah? Or did my buddy start Uber and I was like, “Sure, I’ll give you some money?” Right? I mean, obviously, I’m talking about the extremes here, but we don’t actually know what the decision quality was, right? All we know is that you had a good result and given that you had a good result and people think very highly of you, what are you going to gain? Right? So it’s so nice to just let that feedback loop sit there and allow people to have the opinion of you. That is really nice, feels good, right? And not actually find out the answer because why would I want to?
Unless you’re really super focused on decision quality, then you would want to do that. So that’s part of the psychological safety. And what that goes to is, the real core of it is that it’s very, very difficult for human beings to deal with feeling wrong in the moment, even if it helps them in the long run. It’s just hard. And the tighter the feedback loop, the more that you risk finding out you are wrong in the short run. Now that helps you to learn and improve your decision-making if you’re focused on it and you’re good at it, right? That’s going to help you. And then in the long run, you’re actually going to do better. But human beings are notoriously good at trading off the long run just to feel good in the short run. That’s why we’re all eating chocolate and cupcakes and stuff that we know is bad for us because it feels good.
And so much of our decision-making is trying to advance this positive self-narrative and the idea that yeah, we’re going to have a more positive narrative ourselves in five years if we do some stuff. Most of us are like, “No, I don’t really want to make that trade. I’d rather just feel good now”, and I can use the fact that we are living in power law, under the influence of power law. I can use that to just confirm a lot of things that I wish to believe that are true of me. And if you take that away from me and you take the uncertainty away from me, it’s going to be really hard. And I will tell you that’s what I love about both Renegade and First Round is that they’re just like, I want to know. It would be such a horror for me to think that I was making good decisions when I actually wasn’t. And that’s what really matters to me and I think that it’s just so special.
Lenny Rachitsky: Hearing all this makes me want to be an LP in First Round. Not that they would let me in there, but just knowing that you’re in there futzing with everything and the way they’re thinking is really inspiring. I’m curious if you could share an example of anything that they tweaked as a result of this analysis that you did and how they evaluate.
Annie Duke: Well, first of all, let me just say they didn’t really record a lot about their decisions when I first came in. They voted and they had a record of the vote so they knew who said yes and who said no, but they didn’t have a lot of other information. So the first thing that happened was just that. Okay, what do we really think is the way that you would model whether you should invest or not invest in? Very broadly, you would say you’re reading the market, the team, the founder, the product broadly. We’re going to make sure that those opinions aren’t just like it’s good or it’s bad, but it’s on a scale of one to seven so that we can actually get some precision and some spread among the partners in terms of, say, strength of market. We’re going to make sure that we have shared definitions of those things, which you’d be surprised people don’t.
So when I am thinking about market and market quality, I might be thinking about something very different than you. So we want to make sure that we have a shared definition of that, and that’s reflected in something that we would call mediating judgments, which are judgments that you make related to market prior to actually judging what you think of the market in general. So you could think of something like competitive landscape, so you would judge that. So that turns into what these mediating judgments are, which is basically an implied definition.
So you create that, and then you also think about what are the forecasts that are important. One thing that you already know is you’re going to forecast the probability the company funds at Series A. So that was a huge change, just a very different way of making decisions. What we’ve been able to do with that now because I’ve been there for five years is we now can actually look, say the partnership as a whole, and look at these ratings that they’re making of the component pieces of parts of how they’re modeling, like what makes a good investment, these forecasts that they’re making.
And we actually know how these companies have now unfolded. So in the simplest sense, we know whether hundreds of companies have funded at Series A or not. And now, we can actually look and say, how good are the partners at actually forecasting this thing? Right? Are they random? Are they better than random? And we actually know that and we can feed that back to them so that they can understand their own accuracy around these pieces of the decision because the fact is that whatever a seed investor says to you, whether that company is going to fund at Series A is part of their decision. It’s included in the decision. So they’re making that forecast whether they make the forecast explicitly or not. So what we’re saying over at First Round is let’s make it explicit because you’re doing it implicitly anyway, and then we can actually start to look at your accuracy.
We can now feed that back to you and let you know how accurate you are, which will then help you to become more accurate, right? We can also look, because we know in any given vintage what are the best companies or what are the worst companies. Remember, we’re having people do these ratings on a scale of one to seven of say the quality of the market, and we can look across the partnership and say, “Look, Lenny, when you think the market is great, how does that map onto how well the company is doing in the future? When you say the market is terrible, how well does that map onto how that company is doing, how that company ends up doing in the future?” And I can come to you and I can say, “Oh, Lenny, by the way, your judgments about market are amazing.”
You know a good market when you see one and it’s really mapping on in a great way onto how that company unfolds, but you’re not so great with founder, or maybe you’re great across the board, or maybe you’re not. So we can now start to give people, we can give the partners insight into their own decision-making, not only to allow them to improve the decision-making, but also to allow them to understand not to over index on certain things that maybe aren’t as predictive as an example for them. We also can change the rubric based on the evidence now.
So the first version of the rubric is always taking the intuitions of the partners and making those explicit, but then we can start to loop those back together and understand, well, maybe this thing that the partners thought was important actually isn’t predictive across any partner. For example, we can start to develop the rubric based on the data. These were all things that weren’t possible because prior to that, if I had come in and said, “Well, let’s look at decision quality”, how would I do that? I mean, I have no idea why people were… I just know whether they said yes or no. And so it’s very difficult to them to start to do some really serious refining of the decisions if I don’t have that information.
Lenny Rachitsky: I desperately want to know which partner it makes the best decisions. I know you’re not going to share them.
Annie Duke: No, I’m not going to. The partnership as a whole is excellent as we know. And this is what I will tell you is that all partners have strengths and all partners have weaknesses, and they’re not perfectly overlapping, which is wonderful, right? I mean, that’s one of the things. It’s like what’s really wonderful, and I think it shows the power of why would you have more than one person having input into a decision, is some people are very strong on rating a particular aspect of market, or some people are very strong on rating a particular aspect of the founder. There’s overlap and then there’s things where Todd is uniquely great at something, or Josh is uniquely great at something, or Bill is uniquely great at something. So that’s a wonderful thing about it is that everybody has strengths and everybody has weaknesses and they’re not perfectly overlapping. So this is where you can see getting that spread of opinions and really understanding, breaking that decision down into its component parts really shows you the value of diverse opinions as input into a decision.
Lenny Rachitsky: I could talk about this thread forever. Maybe let me just ask one more question just because I’m super curious. Is there anything surprising that stands out, that came out of this analysis so far? Just like, oh wow, maybe market isn’t as important as we thought or this person is amazing at-
Annie Duke: Yes. So I think just generally speaking, when you’re creating the initial decision rubric, there are things that people are really pounding the table about that they think is especially important in making a decision. And one of the things that we found is that sometimes their intuition was absolutely right. The thing that they were pounding the table about is incredibly predictive, not just for them but other partners. But sometimes it’s not at all predictive, and these are equal table-pounding situations. So let’s say you’re pounding the table about something, sometimes the thing you’re pounding at the table about is predictive for you and for all the other partners that it’s actually quite predictive about how the company does. But sometimes when you’re really pounding the table about something, it’s not just that it’s not predictive for the other partners, it’s not predictive for you.
And I think that what’s really important to understand about this, and this is why it’s so incredibly necessary in improving decision quality to take what’s implicit and make it explicit, is that our intuition is sometimes right and sometimes wrong. It’s not that intuition is crap and your intuition is just completely wrong, I mean obviously, that can’t be true. We would die, right? So your intuition is sometimes right, but it’s also sometimes wrong. And if you don’t make it explicit, then you don’t get to find out when it’s wrong.
You don’t get to find out when it’s off base, and that’s a disaster. So that’s the thing that I think is really interesting is that you have equal vehement and confidence that this particular factor is really important. And sometimes it is and sometimes it isn’t. And it’s so surprising because we’re talking about people who are true experts who are great, and I think that we all just have intuition about intuition, right? You just intuit that if they’re so amazing, clearly their intuition about what’s important would have to be good, but not necessarily. That’s the thing, not necessarily. Yeah, I think that was probably the most exciting thing.
Lenny Rachitsky: Keeping it mysterious, but I still appreciate you sharing.
Annie Duke: Well, I have to keep it mysterious.
Lenny Rachitsky: Yes, I understand.
Annie Duke: I can’t give away the trade secrets.
Lenny Rachitsky: I wanted to touch on a different framework that I’ve heard a lot of companies actually using, and something that comes up in product a lot is this idea of pre-mortems, which is essentially think ahead of time what might go wrong. Can you talk about this? Because I think that it’s something that’s easy to implement, really powerful, and a lot of people are actually doing this.
Annie Duke: Yes, okay. So a pre-mortem is great, but only if you attach a pre-commitment with it. So I just want to be super clear about that. What you find with pre-mortems, this actually work. I actually did this work with Maurice Schweitzer and Linnea Gandhi, who are both at Penn. Then when you have people do pre-mortems, it generally doesn’t actually change their plan very much or it changed their behavior. So I think that we have the feeling that if you do a pre-mortem and you think what are the ways that things might go wrong, that that’s going to change your plan, but probably not unless you’re specifically using it for that purpose and you say, “Okay, we’re going to do this, but let’s think about how we might change our plan in light of this information.” But I think what’s actually more important than that is what a pre-mortem allows you to do is to set up kill criteria.
So kill criteria are just a set of signals that you might see that would tell you that it’s time to pivot or stop because once we actually launch something, we’re very, very slow to decide to quit. I’m sure that everybody has felt that way before. Things go on way too long, even like they’re over budget and you’ve blown the timeline. And when you finally shut it down, you realize you should have done it a lot earlier. And this is true across the board because of a variety of biases. The most well-known, and probably the biggest influence is something called sunk cost, which is that feeling, but then I’ll have wasted all the time and effort that I’ve put into this already. So it’s taking into account what you’ve put in in the past and deciding whether to continue on in the future. So what we want to do is actually just get better at that thing. So understanding that when you’ve gotten to the pre-mortem process, you probably are going to launch it, like it’s probably going to be the case.
Use the pre-mortem to set up kill criteria. So I’ll give you an example from a sales team that I worked with. Basically, I sent them out a prompt, all the IC is a prompt that was, imagine that you got a lead through an RFP or RFI and you worked on it for six months. And now, it’s six months later and the deal is dead. Looking back, you realized they were early signals that that was going to be the case, what were they. So this is a pre-mortem. What are the things that you saw that would tell you that this was going to go south? And they came up with all sorts of ideas. Notice this doesn’t mean they’re not going to start off pursuing the lead, right? But they saw, they came up with all sorts of signals. So I’ll just give you three of them.
The RFP RFI was clearly written with a competitor in mind, so they felt that was a very bad signal that was probably going to go badly. Another one was the customer didn’t want to demo, they only wanted to talk about price. Obviously, that’s quite bad. And another one was after the first few meetings, they couldn’t get a decision maker in the room, right? So it was a much longer list than this, but those are three. So for each of those, that now becomes a kill criteria if I see this thing and now you attach an action with it. So in the case of price, they actually just said, “We should kill it.” If they literally don’t want to demo and they’re only asking about price, they’re just trying to beat up somebody else on price like we’re a box-checking exercise. So there, they just said, “We’re going to kill it. We’re not going to pursue the deal anymore.”
So this is great because salespeople will pursue deals forever and leadership is like, “Well, why did you stop pursuing that deal?” And they get in trouble for it. So this is going to help with that problem, right? In the case of the RFP RFI was written with a competitor in mind, they have an action associated with that as well, which is ask them directly if they’re working with a competitor and how far down the road they are, depending on the answer you would kill or pursue. In the case of we couldn’t get a decision maker in the room, offer up executive alignment at the next meeting. And if they say sure, great. And if they say no, kill.
So that’s actually what I feel is the best use of a pre-mortem is to say, I’m going to try to figure out what those signals are along the way that things are going badly. And now instead of just hoping that when I see those signals, I actually act rationally, which is a hope that it will not come true. That’s why there’s many people who climb Everest in the middle of a blizzard, even though they shouldn’t be doing that. Use the pre-mortem to now create structure around those signals that you’ve spotted and commit to actions that you’re going to take if you see those signals. And I think that’s the best use of a pre-mortem.
Lenny Rachitsky: That’s really helpful. And it’s interesting how many of your examples come back to just of a framework that you often talk about, which is make it explicit what is implicit. There’s another example that the First Round example is a great example of that where it’s just, here’s all our assumptions, they just make them actually explicit and just shows how much [inaudible 01:10:32].
Annie Duke: Right, then you can examine them. You can examine them, people can discuss them, you can figure out if they’re wrong or right or whatever. It’s like, I want to be very clear, I’m not anti your gut or your intuition. I think it’s probably sometimes pretty good. I just want you to make it explicit, that’s all.
Lenny Rachitsky: Okay. So we didn’t have time to get into quitting, which is your more recent book. Maybe we’ll do a follow-up episode specifically thinking about quitting, but let me just ask-
Annie Duke: It’s my fault because my answers are long.
Lenny Rachitsky: It’s my fault.
Annie Duke: I apologize to everybody.
Lenny Rachitsky: No apology is necessary. We’ll have plenty of time in the future, hopefully. Well, let me just ask one question. I found this one quote from you where you said you should assume that if you’re thinking about quitting, it’s already probably past the time that you should have quit. Do you still believe that? Is that generally a good rule of thumb? And just any takeaway, tip, lesson on quitting as our one question on quitting?
Annie Duke: So the data is pretty strong that by the time you quit, it’s probably long after the [inaudible 01:11:37]. And it’s really just because, look, when we start things, we’re starting things under difficult circumstances, which come from the uncertainty of the decision to start something. So when we start something, luck is going to have an influence on the outcome, which obviously, we have no control over because luck isn’t in our control. And then there’s also hidden information. So what happens is that after the fact, we know that we’re going to learn new information and it can make it very hard to start things because we want to be more sure than we actually need to be. It’s why Bezos has the 70% role to try to roll people back and be willing to accept that uncertainty in the starting decision. Now the good news is that when you learn that new stuff and the new stuff that you learn is, “Ooh, if I had known this, I wouldn’t have started it”, you have the option to quit generally.
So that’s the good news. The bad news is that the same difficulties that apply to the decision to start apply to the decision to stop. In other words, we’re making that decision under uncertainty as well. So we’re not going to know for sure whether it would’ve turned out well or poorly unless we continue to do the thing that we already started. And we don’t like to walk away from things unless we know for sure. So as Richard Thaler put it, most people won’t quit until it actually isn’t a decision. In other words, the whole thing is blown up, the startup has no money, or you’re up on Everest and the blizzard is literally upon you and you’re stuck in it. Or I think as he said, until you’ve fallen in the crevasse already, then you’ll make the decision to quit because then you know how it was really bad. So people generally, for example, don’t quit their jobs until they feel they have no other choice or relationships or projects or products that they’re developing, it all applies because we want to know for sure.
And then on top of that is the fact that there is this issue of sunk cost, which is when we walk away, we feel like we’ll have wasted everything that we’ve already put into what we’re doing. But of course, waste is a prospective problem, not a retrospective one, even if we treat it like a retrospective one because it’s the prospective one. Well, if you wouldn’t start this today, then that means that everything that you’re putting into this going forward is the actual waste, right? But we do that all the time. We go forward with things that we ought not to be going forward with because we’re trying to protect the resources that we’ve already sunk into it in the past. Then there’s other issues that have to do with, for example, endowment, the ownership over the things that we’ve built. This is particularly bad in product because we’re building things.
And once we build things, we own them. And once we own them, we don’t want to give them up and we actually value them more highly than identical things that we don’t own. And then there’s issues of internal and external validity, which is really just a fancy way to say your identity. How do other people view you? How do you view yourself? Do you feel like you failed? And what that means is that by the time, there’s so many biases against stopping that by the time you’re actually even thinking about quitting, it’s probably already past the time that you ought to have quit. But we’ll still continue on until we know for sure we didn’t have any choice because here’s the thing. When you walk away from something and someone’s like, “Hey, why’d you stop that?” And you’re like, “Oh, I had no choice” and you tell them everything that went wrong and, nobody’s going to question you.
They’re going to be like, “Oh, well, it sounds like you put in your best effort.” But if you walk away early, people are like, “What?” So just quickly, I’ll tell you I think just one of the best stories of this that I’ve got in my pocket here. Let me pull it out before the end. So Stewart Butterfield creating a product called Glitch, and Glitch is a massive multiplayer online world-building cooperative game. Releases it, this is in the aughts, and it’s like a huge hit with the critics. It’s Monty Python meets Dr. Seuss, it’s an incredible whatever. They’re getting tons and tons of great word of mouth and PR not doing any paid marketing. They have incredible investors in Andreessen Horowitz and Accel, they have $6 million in the bank and they 5,000 have diehard users, meaning users who use the game who play over 20 hours a week.
The issue is that customer acquisition was a beast that for every one person who was playing over 20 hours a week, there were between 95 and 99 people who came for five minutes and left. So obviously, this is a customer acquisition problem, which everybody knows. So they make an agreement in 2012 that they’re going to do paid marketing, which they do for six weeks. And during that six weeks, their acquisition of new users, it’s growing like six to 7% week over week, which is amazing. And at the end of that six weeks, this is November of 2012, at the end of that six weeks, that Monday morning, Stewart Butterfield writes a note to his investors and co-founders and says, “I woke up this morning with the dead certainty that Glitch was over.”
Now notice nobody would do this in this case, right? But this is what happened is that the issue is that you really have to see is this worthwhile or not? Would I start this today? That’s a forecast of the future, right? And what he did was some back-of-the-envelope math, and he said, “Look, if we continue to acquire customers at the weight that we’ve been acquiring them at the cost that we’ve been acquiring them, it’s going to be 31 weeks until we break even.” But that’s an absurd assumption because customer acquisition costs, it’s going to go up. Cap has to rise because we’re going to saturate the core gaming market, so it’s got to rise. So what he realized at that point was that this was not a venture scale business, and he was in this for a venture scale business. So even though nobody else saw that he was supposed to shut it down, he saw he was supposed to shut it down.
And not only that, he saw that he was supposed to shut it down for his employees who were working for equity, and he had now realized that the equity wasn’t worth their time and that it wasn’t fair to them to keep going with it. So he shuts it down. Obviously, that feels… Who does that? Right? And he will actually tell you that he knows he should have shut it down before the marketing push, but he needed the marketing push to prove it to himself that he was seeing the future clearly. Now, the code to this story just quickly is two days later, he’s like, “Well, I’m a startup guy. I want to start something”, and he’s got this internal communication tool that his team is using in order to develop this product that everybody loves. And he says, “Actually, they really like that. Maybe that should be the next product.”
So he goes and talks to the investors, they roll their money over into that. And that thing, which had no name at the time, now gets a name which is searchable log of all company knowledge, which is Slack. And so I think this is the important thing to realize is that we get so focused on, “But what about everything that I’ve put into it?” When what we forget is that when you’re doing something, there’s not just the cost of doing something that’s not worthwhile that’s direct, but there’s also the cost of not being able to devote your attention to other opportunities that might be available to you. And as smart as Stewart Butterfield is, he couldn’t see Slack until he quit Glitch. And that is a true cost that he would’ve born of continuing with Glitch. If he had continued with Glitch, Slack would not be something that we’re all using today.
Lenny Rachitsky: That is an incredibly beautiful way to end our conversation. I feel like I have at least a billion more questions to ask you and on the other hand, I feel like we’ve also helped a lot of people make much better decisions through this chat so I’m really thankful that you made time for this. Two last questions. Where can folks find you online if they want to learn about the stuff you’re up to in case they want to work with you? And how can listeners be useful to you?
Annie Duke: You can find me at annieduke.com if you’re interested in working with me. I have a Substack, Thinking in Bets. Please go check that out. I teach a class on maven.com twice a year on effective decision-making. So if you’re interested in that, you can go to Maven and check it out. My next cohort at the moment is in September, although I might do one in May. I’m not sure. But in terms of people can help me, I co-founded an organization called The Alliance for Decision Education. We’re trying to bring the kinds of knowledge that we have about improving human decision-making in adults to K through 12 education to make the world a better place. So I would love it if people could go look at that. If you’re interested in it, get the word out.
Lenny Rachitsky: Amazing. So we’ll link to all those things in the show notes. Annie, again, thank you so much for being here.
Annie Duke: Thank you. Thank you so much, this was so fun.
Lenny Rachitsky: Same. Bye, everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.
Glossary
| English | 中文 |
|---|---|
| Accel | Accel(投资机构名称,保留原文) |
| adversarial collaboration | 对抗性合作 |
| Andreessen Horowitz | Andreessen Horowitz(投资机构名称,保留原文) |
| Angus Deaton | 安格斯·迪顿(诺贝尔经济学奖得主) |
| Barb Mellers | Barb Mellers(人名,保留原文) |
| Bezos | 贝索斯(Amazon 创始人) |
| Brett Berson | Brett Berson(人名,保留原文) |
| Daniel Kahneman | 丹尼尔·卡尼曼(诺贝尔经济学奖得主,行为经济学先驱) |
| decision rubric | 决策评分标准 |
| Dr. Becky | Dr. Becky(人名,保留原文) |
| Dr. Seuss | 苏斯博士(美国儿童文学作家) |
| endowed | 被赋予(拥有感) |
| executive alignment | 高层对接 |
| First Round | First Round(投资机构名称,保留原文) |
| focusing effect | 聚焦效应 |
| forced rank | 强制排序 |
| Glitch | Glitch(产品名称,保留原文) |
| hit rate | 命中率 |
| IC | IC(Individual Contributor,销售代表,保留原文缩写) |
| Josh Kopelman | Josh Kopelman(人名,First Round 联合创始人,保留原文) |
| kill criteria | 终止标准 |
| Linnea Gandhi | Linnea Gandhi(人名,保留原文) |
| Matthew Killingsworth | Matthew Killingsworth(人名,保留原文) |
| Maurice Schweitzer | Maurice Schweitzer(人名,保留原文) |
| Maven | Maven(在线教育平台名称,保留原文) |
| mediating judgments | 中介判断 |
| mental time travel | 心理时间旅行 |
| Monty Python | 蒙提·派森(英国喜剧团体) |
| net new ARR | 净新增 ARR |
| nominal group | 名义小组 |
| pre-mortem | 事前验尸 |
| priming | 启动效应 |
| product market fit | 产品市场契合度 |
| Quit: The Power of Knowing When to Walk Away | 《放弃:知道何时离开的力量》(Annie Duke 的著作) |
| Renata Quintini | Renata Quintini(人名,保留原文) |
| Renegade Partners | Renegade Partners(投资机构名称,保留原文) |
| resulting | resulting(Annie Duke 提出的决策术语,指以结果论好坏的思维倾向) |
| RFI | RFI(Request for Information,信息请求,保留原文缩写) |
| RFP | RFP(Request for Proposal,请求提案,保留原文缩写) |
| Richard Thaler | Richard Thaler(行为经济学家,诺贝尔经济学奖得主,保留原文) |
| Rip Van Winkle | Rip Van Winkle(美国童话中沉睡多年的角色,保留原文) |
| Roseanne Wincek | Roseanne Wincek(人名,保留原文) |
| seed | 种子轮 |
| special partner | 特别合伙人 |
| Stewart Butterfield | Stewart Butterfield(Slack 联合创始人,人名保留原文) |
| sunk cost | 沉没成本 |
| The Alliance for Decision Education | The Alliance for Decision Education(组织名称,保留原文) |
| Thinking In Bets | 《对赌》(Annie Duke 的著作,中文译名) |
| Thinking, Fast and Slow | 《思考,快与慢》(Daniel Kahneman 的著作) |
| Todd Jackson | Todd Jackson(人名,保留原文) |
| vintage | 年份基金 |
| World Series of Poker | 世界扑克大赛 |
| 禀赋效应 / endowment | 禀赋效应(拥有感) |
Reformatted by reformat_english.py
这将让你成为更好的决策者 | Annie Duke(《Thinking In Bets》作者,前职业扑克选手)
逐字稿
Annie Duke: 在提升决策质量的过程中,把隐性的东西显性化是极其必要的。并不是说直觉一无是处,你的直觉有时是对的。但如果不把它显性化,你就无从发现它什么时候是错的。
Lenny Rachitsky: 你观察那些读过你书的公司时,发现哪些”洗脑战术”是真正行之有效的?
Annie Duke: 人们通常认为开会的目的有三件事:发现、讨论、决策。但开会唯一应该做的事情其实是讨论。
Lenny Rachitsky: 在产品领域经常出现的一个概念是”事前验尸”(pre-mortem)。
Annie Duke: 事前验尸只有在设定了”终止标准”(kill criteria)的情况下才有意义。要承诺当你看到那些信号时会采取的行动。
Lenny Rachitsky: 你有一个非常有趣的框架,用于在结果周期很长的情况下评估决策质量。
Annie Duke: 根本不存在所谓的”长反馈循环”这回事。缩短反馈循环的方式是问:哪些指标与我最终期望的结果之间存在相关性?
嘉宾介绍
Lenny Rachitsky: 今天的嘉宾是 Annie Duke。Annie 是畅销书《对赌》(Thinking in Bets)的作者,她最近还出版了一本新书《放弃:知道何时离开的力量》(Quit: The Power of Knowing When to Walk Away)。她同时也是 First Round Capital 的特殊合伙人,我们在这方面花了一些时间讨论,内容非常精彩。在此之前,她是一名职业扑克选手。她在锦标赛中赢得了超过 400 万美元的奖金,其中包括一枚世界扑克大赛(World Series of Poker)金手链,而且是唯一一位同时赢得世界扑克大赛冠军锦标赛和全国扑克单挑锦标赛冠军的女性。目前,她把时间花在帮助公司做出更好的决策上。在这次对话中,我们聊到了她从已故好友 Daniel Kahneman 那里学到的许多教训——Daniel 最近刚刚去世。我们还谈到她发现哪一个小改变对公司决策能力的提升影响最大;如何在反馈循环很长的情况下做出更好的快速决策,以及为什么她其实并不认同”长反馈循环”这个概念;还有她如何改变了 First Round Capital 合伙人们的决策方式,这部分非常有趣。
与 Daniel Kahneman 的回忆
Lenny Rachitsky: 另外,我们还会聊到为什么当你在考虑放弃的时候,很可能意味着你已经等了太久,本应该更早就放弃的。我从这次对话中学到了很多东西,这一定会改变我对很多事情的思考方式。好了,接下来为你带来 Annie Duke。Annie,非常感谢你来参加节目,欢迎来到播客。
Annie Duke: 非常感谢你的邀请。
Lenny Rachitsky: 这是我的荣幸。我跟很多人提到你要来上这个播客,每一个人都特别兴奋,所以我也非常期待这次对话。我想先问一个关于 Daniel Kahneman 的问题。我知道你们关系很近,你们曾一起做过播客,过去几年也成为了好朋友。遗憾的是,他在大约一周前去世了。
Annie Duke: 是的。
Lenny Rachitsky: 我很好奇,你和他相处的过程中,有哪些持久的教训是你带走的?有没有一两件让你印象深刻、并且会一直留在你心中的事情?
Annie Duke: 我很难准确形容这个人有多谦逊。这看起来似乎不太可能,因为他的影响力如此巨大,他的智慧如此深邃,他的洞见在他所从事的领域中产生了如此变革性的影响。但他非常谦逊,真心想听你的想法。他会说”我知道的不多”,他愿意改变自己的观点。有意思的是,有一次我在课上教《思考,快与慢》(Thinking, Fast and Slow),有人提到书里有不少研究最终没能复现。而我看到这个领域中很多其他学者,当他们的核心研究成果未能复现时,他们确实在极力反驳,试图为自己辩护,说”不,不是这样的”等等,对”也许结果确实不成立,而且这没什么”这种可能性表现出一种真正的不开放。但 Danny 的不同之处在于,当你跟他谈论这些时,他是第一个告诉你的人:“我希望启动效应(priming)没有写进书里。它没能复现。“他对一个观念如此开放:他知道他此刻所知道的,但他也知道他并非无所不知。
另一件我觉得学术界之外的人不太了解的事是,他是所谓的”对抗性合作”(adversarial collaboration)的先驱之一。这是他最喜欢做的事情之一,而且在某种意义上是他发明的——找到一个与你真正持不同意见的人,然后和他们一起写一篇论文,共同找出一个能够解决争议的研究方案。举一个例子,Danny 做过关于财富与幸福感关系的研究。Kahneman 和合作者 Angus Deaton 发表了一项引起广泛关注的研究结果,探讨金钱与幸福感之间的关系,核心观点是:超过 75,000 美元之后,金钱就买不到更多幸福感了。这项研究发表于 2010 年。如果你现在重复完全相同的研究,这个数字肯定不止 75,000 美元,因为需要根据通胀进行调整。但其核心观点是,一旦你的基本需求得到满足,不再那么为付房租之类的事情担忧,金钱对幸福感的提升就不会有那么大的影响了。
与 Matthew Killingsworth 的对抗性合作
Annie Duke: 但十年后,Matthew Killingsworth 说:“这不对,实际上金钱和幸福感在所有收入水平上都是相关的。钱越多,你倾向于越幸福。“好,这就是一个典型的例子——人们有时会互相拆台,说那个人是错的之类的。但 Kahneman 的反应是:“好吧,我们一起来做。“他和 Killingsworth 联手,加上 Barb Mellers,一起尝试解决这个问题。他总是在寻找,就像所有事情对他来说都是在问——我哪里错了?他急切地、兴奋地想要……
我记得和他吃了一顿午饭,饭后我才意识到整顿饭都是他在问我关于我工作的问题。他为什么要问我关于我工作的问题呢?他是 Danny Kahneman 啊。但他就是那样对他人充满好奇,而且他为每个人都腾出时间。他真心热爱身边的人。我认为,说起来可能难以置信,但我觉得这一点甚至超越了他那部堪称辉煌的学术成就。而作为一个人,他本人更是令人叹服。你确实经常听到那些极其聪明的人——那种”聪明但混蛋”的套路——而他却恰恰相反。他非常善良。
Lenny Rachitsky: 太棒了。感觉那些充满好奇心、总是寻找自己哪里错了、并花时间和他人辩论、讨论、提问的人,与发现真正有趣的洞见、揭示新事物之间,很可能存在很强的相关性。
Annie Duke: 对,对,对。
决策与放弃
Lenny Rachitsky: 好,谢谢你分享这些。我真希望曾认识他。我们来说说决策吧。我想花时间谈两个领域,大家应该也不意外——决策和放弃。我知道你上过很多播客,我想很多听众读过你的书,或者听过不少你讲授的内容。我会尝试从一些不同的角度切入,做得不太一样。首先,我知道你是位家长。据我所知你有四个孩子。我刚有了一个孩子,快十个月大了。
Annie Duke: 哦,恭喜。
Lenny Rachitsky: 谢谢。我很好奇,在养育孩子的过程中,你觉得哪些框架对你最有帮助、最为受用?
育儿建议
Annie Duke: 首先,我想说一点——很多怀孕的人会来问我育儿的建议。我告诉你我给人们的两条建议。第一条是:市面上有各种各样的育儿书,各种各样的育儿理念。一本书需要卖出去,所以它们需要卖给你一些新的、不一样的东西。但真正重要的只有一件事——让你的孩子知道你爱他们,是真正深入骨髓地知道你爱他们。
我们可以争论母乳喂养还是不母乳喂养、同睡还是不同睡、亲密育儿还是睡眠训练,这些都可以讨论,对吧?在家上学?送私立学校?还是送公立学校?我们可以讨论很多。但所有这些跟”你的孩子知道你愿意为了他们躺在公交车前面、只要他们能活着并且幸福”比起来,都微不足道。这是我说的第一件事。第二件事是——总有一天,你会把宝宝摔到头。我猜这已经发生在你身上了。
Lenny Rachitsky: 嗯,算是吧。差不多。
Annie Duke: 差不多,对吧。
Lenny Rachitsky: 请继续。
Annie Duke: 关于宝宝,事情是这样的——宝宝不会动。所以作为妈妈或爸爸,你会非常习惯坐在沙发上叠衣服,宝宝就在你旁边,宝宝不会动。然后有一天,毫无征兆地,你的宝宝会翻身了。你以为没事,因为你放了个枕头什么的在那儿,然后你转过头一秒钟,心想无所谓,然后你的宝宝就摔到头了。我四个孩子每一个都发生过。他们摔得也不远,而且完全没事,你不是个糟糕的家长。就是这样——有些事情你不知道,事情总会发生,不可能十全十美。不是每一件小事都会对孩子留下什么持久的印象,因为总有一天,你会不小心把宝宝摔到头,你不是故意的,你没有那个意图。
但在我家的情况,每次都是——天哪,我的宝宝会翻身了。我没想到今天就会发生。等到第四个孩子的时候,就是我在追着一个宝宝给他换尿布、忙这忙那,然后另一个宝宝就磕到头了。所以我觉得这里有一个小小的教训——这是件小事,别纠结,别生自己的气,因为你犯了一个错误。真正的教训是,哦,我意识到我的宝宝现在会翻身了。那现在你就要格外注意宝宝从沙发上翻下来的风险。但这其实只是我用一种有趣的方式在说——你的宝宝会磕到头,或者撞到茶几,或者划个口子之类的。他们非常 resilient,而且你没有做错什么,所以不要为此自责。然后我确实还有一件事想说。
Lenny Rachitsky: 好,请说。
Annie Duke: 有四个孩子你就会发现,你对孩子的性格几乎没有任何影响力。你能做的,就是确保他们说”请”和”谢谢”。所以我发现只养了一个孩子的父母——尤其是那个孩子很乖的话——他们会觉得自己是很棒的家长,也许确实是。那就是 resulting。也许确实是。但等你有了四个,你就会意识到——哦,我的育儿方式其实跟这事没什么关系。但我的孩子们确实会说”请”和”谢谢”。
Lenny Rachitsky: 这些真的很有价值,很好的建议。我需要听到这些。我太太也需要听到。她一直担心宝宝头上的囟门,怕磕到什么东西。
Annie Duke: 没事的,会没事的。
Lenny Rachitsky: 好,太好了。
Annie Duke: 其实那个软 spot 存在就是有原因的。真的,你的宝宝会没事的。事实就是这样——我们以前在荒野中带着婴儿四处迁徙,我们住在洞穴里。我们没事,我们的身体就是为了这个而设计的。我们适应的是比现在婴儿实际成长的环境要粗糙得多的世界。我想说,我的大女儿上高中时,出于某种原因,她对酒精或毒品完全没有兴趣。我记得我当时的得意劲儿——觉得自己把这个孩子养得多好,她多优秀,那些其他家长显然做得不够好。好吧,我并没有真的那么想,但因为你很难不这么想。然后我的下一个孩子上了高中,我就觉得——那就是性格问题。
Lenny Rachitsky: 这是一个很好的 resulting 的例子。
resulting 与育儿
Annie Duke: 这确实是一个很好的 resulting 的例子,不过我的孩子们最后都挺好,虽然他们各不相同。但就决策而言,这完全是另一回事。这一点我想引用 Danny Kahneman 的一句话:当你正在想一件事的时候,没有什么事情看起来有它显得的那么重要——这句话非常重要。我以前经常跟孩子们做的一件事就是心理时间旅行,这其实是一个非常好的决策工具。比如他们因为某件事特别沮丧——可能是学校发生的事,也可能是他们因为某件事被禁足了之类的。
在当下那一刻,事情感觉特别大。我以前常说的、他们现在也常说的话是:等你40岁过感恩节的时候,这件事简直太棒了。 你可以在感恩节餐桌上把这些故事讲给你的孩子听,那绝对精彩——你应该感谢我,让你能在感恩节餐桌上谈论你这位疯狂的妈妈。这让他们能从中抽离出一些时间和空间,意识到这件事总有一天会变得好笑。
总有一天,它会成为你讲给别人听的一个极其好笑的故事。不管现在看起来多糟糕,等你长大以后,它总有一天会变成一个极其好笑的故事。这听起来像是我在跟他们开玩笑,但实际上对我们来说这是一个非常好的通用决策工具,对吧?当然,确实有一些事情20年后依然重要,这毫无疑问。但大多数事情不是。想想你经历过的大部分事情——比如你21岁的时候女朋友跟你分手了之类的,你觉得这就是世界末日,你再也恢复不过来了,等等。但如果你能让自己想一想——好吧,我真的觉得自己10年后还会在意这件事吗?我10年后还会心碎吗?
大概不会。我认为我们需要获得那种时间上的视角,让自己跳出当下,因为在当下一切都显得那么重要,情绪那么强烈。就好像注意力完全聚焦在这一刻、聚焦在你此刻的感受上,感受如此巨大,以至于我们遗忘了时间的广度。我觉得这绝对是你能在孩子身上做的最好的事情之一,而且你也可以在小事上用它,对吧?你可以选择打这个电子游戏,也可以选择复习备考。一周后你拿到考试成绩的时候,你觉得你会对这两个选择怎么看?
这是我大量使用的一个工具,也是一个普遍来说非常好的决策工具,只不过我碰巧经常跟孩子们一起用。然后我想说的另一个技巧是使用”nevertheless”这个词——这是一个很棒的领导力技巧。比如我的孩子被发现做错了事。比如说,我在后院发现了一袋他们忘了扔的红色塑料杯——那是周末我不在家的时候留下的。这只是假设啊。然后我禁他们的足,他们就开始各种争辩。他们觉得这是一场辩论,向我抛出各种理由和意见——为什么这不公平,别的小孩都这样做,别的家长不会生气,等等等等。你需要在让他们感到被倾听——我认为这一点极其重要,孩子需要感到被倾听——和坚持做你认为正确的事之间取得平衡。
所以我常说:我听到了。Nevertheless,你被禁足两周。我听到了你的想法,我理解。Nevertheless,这就是接下来会发生的事。当然,你实际使用的措辞可以不同。显然你对孩子的权威远大于其他场合。但在职场中,这个方法非常好用,因为员工总是对领导做出的决策抱怨不休,因为他们觉得自己是对的,想要按自己的方式来。而你有能力说:我听到了你的意见,你的意见——请相信我——已经被纳入了决策考量,nevertheless,这就是我们要走的路。 不管对错,我们就是要这样做。所以我觉得很多这类事情,表面上看起来是育儿的好方法,但实际上它们就是非常好的通用决策策略。
Lenny Rachitsky: 我正在记笔记。这些会非常有用。
Annie Duke: nevertheless 这个词对孩子特别好使。
Lenny Rachitsky: 这让我想到 Dr. Becky 教的一个方法,如果你知道她的话——就是告诉对方”我相信你”。当你的孩子说了什么、因为某件事不高兴的时候,先说”我相信你”。
Annie Duke: 然后接上 nevertheless。
Lenny Rachitsky: 对,就是 nevertheless。有道理。我觉得我们快要把这个播客变成育儿播客了,因为这类内容太多了,但我得克制一下。我们一直在谈决策和框架之类的话题。我想问你的是:一个人在做出更好决策方面,到底能提升多少?比如说,有人听了你所有的播客,读了你的书,认真学习研究。你发现一个人做出更好决策的能力能有多大的提升?这个问题的来源是 Daniel Kahneman——有人问他:“既然你知道我们有这些偏见、会犯这些思维错误,你现在的生活是不是非常优化了?“他说:“不,实际上……”我记得他那句名言是:“实际上并没有。我无法在实践中运用这些东西。这些只是我学到的知识,但它们并没有真正影响我的日常生活。“所以我很好奇,一个人到底能提升多少?你发现人们在决策方面能有多大的变化?
决策能力到底能提升多少?
Annie Duke: 好的,这取决于你是否真的去做了那些事情。我觉得问题就在这里。Kahneman 很早以前做过一些研究,开启了他的这段探索之旅——那是关于招聘的研究。从完全非结构化的方式,Lenny 说”我看到一个好的产品经理我就认识”,不管那意味着什么——到我走到 Lenny 面前说:“好吧,我理解你觉得你看到一个产品经理就知道好不好,但你能跟我解释一下那具体是什么意思吗?抽象地说,你在寻找一个填补那个角色的人时,你看重的是什么?“然后我们就可以把这些挖掘出来,把你隐含应用的标准变得显性化——因为你确实在对被面试者和他们将要担任的角色之间的关系应用某种隐含模型,但让我们把它显性化。
这样我们就可以把它显性化,把它变成一个决策评分标准,从中创建一套结构化的面试流程。然后他发现,在你完成了那个过程之后,如果你在那之后——而不是之前——再运用你的直觉,你的招聘成功率确实可以得到极大的提升。在那种情况下,命中率从大约50%提高到了65%。这是非常显著的,差别非常大。问题在于,没有人真正去做。我可以告诉你,我跟 Danny 之间关于我的工作的那些对话,大部分都是他在说:“你怎么才能让人真正去做呢?“这并不是说我在这些方面比他做得更好——而是他没有做我做的事。他是一位从事研究的学者,而我是在公司里扎扎实实嵌入好几年的人。
我跟一个客户合作了五年,一个四年,还有一个也差不多四年,三年半。然后我刚接了一个新客户,合作了六个月——这就是我全部的客户名单,因为我在每个客户那里待的时间都很长。所以如果你真的去做了,答案是提升相当多。至少就 Danny 的研究而言,在招聘这种噪声很大的领域里,命中率从50%提高到了65%,这是非常巨大的。问题在于,让你做出更好决策的方式,并不是什么超越人类正常决策模式的超自然能力。
我认为我们对自己的直觉评价过高了。我们对自己在当下察觉事物的能力、以及在当下做出理性应对的能力,评价都过高了——实在不该如此。我们倾向于认为自己拥有别人没有的洞察力,而实际上,别人对我们处境的洞察很可能比我们自己更多。所以我的回答是,提升的空间很大,但有一个大大的前提——前提是你能让人们真正去做。我觉得问题就出在这里。不过好消息是,你可以反过来想:如果我愿意去做,想想看我会比那些不愿意做的人拥有多大的优势,对吧?
我们可以这样想:有些人不知道这些方法,有些人知道但不去执行,还有一些人知道而且真的去做了。最后这个群体小得可怜。所以答案是——我认为提升很大,但几乎没有人去做。
哪些决策框架最有效?
Lenny Rachitsky: 顺着这条线继续聊——当你观察你合作过的公司,或者那些读了你的书、真正深入尝试实施你关于更好决策建议的公司,如果你回顾那些真正产生了影响的案例,你觉得哪些心智模型、框架或策略是真正能留下来的?哪些最常产生效果、影响也最大?
Annie Duke: 我认为最容易实施的就是这个。它非常简单,我真希望更多人能去做。获取他人意见的最好方式,是独立于其他人的意见,独立地、异步地收集。我的说法是——我希望人们少说点话,别总是在一起讨论。当我们思考人们通常认为开会的目的是什么时,他们认为是三件事:发现,也就是我想了解你的意见、你对某件事的判断,对吧?我想知道你怎么想。比如,我们在做产品开发,要确定一个时间线,要制定产品路线图,要搞清楚发布某些功能需要多长时间——我们所有人走进一个房间,开始一起大声喊出来。
这对决策来说太糟糕了,我都没法告诉你有多糟糕。交叉影响非常严重,房间里声音最大的人往往会对决策产生不成比例的影响,最自信的人往往也会对决策产生不成比例的影响。如果最自信的人恰好是对的,那当然很好——但问题在于,情况并不总是如此。这就是”发现”这个环节的问题,我们倾向于在群体中做这件事。然后是讨论,也就是我已经发现了 Lenny 是如何对这个问题建模的,或者他对某些事情的判断是什么、他的预测是什么、他的估算是什么。然后我们要在群体环境中讨论这些想法,把你的想法和我的想法放在一起讨论。最后还有决策——我们要在群体环境中做出决定,比如确定路线图最终长什么样。
我认为最容易改变的一点,就是意识到开会唯一应该做的事情其实只有讨论。我们当然应该聚在一起讨论每个人的判断和意见、他们建模问题的方式、他们的预测。尤其值得讨论的是大家意见不一致的地方。你肯定参加过很多会议,我估计大约80%的时间都在重复确认——“哦,我同意。“然后我们用了不同的词表达了完全相同的意思,仅仅因为我们意见一致。
如果我们能把讨论聚焦在大家意见不同的地方,效果会好得多。所以我们把开会的过程分为发现、讨论、决策。我的意思是,讨论才是会议中唯一应该发生的事,也是唯一应该在会议中做的事。那么发现和决策呢?这就是我觉得我真的能推动人们去做的部分——在大家走进会议室之前,独立于彼此地发现每个人的想法。具体怎么做呢?假设你要开一个关于产品路线图的会议。
你先问自己:在这次会议中,我需要从参会者那里获得哪些意见?它可以是一个头脑风暴——所有我们可以开发的不同功能有哪些?可以是一个头脑风暴,没问题。那就独立地写信给他们说:“自由发挥。把你认为我们应该考虑开发的所有功能都列出来,然后对你自己的想法做一个强制排序,从最好到最差,附上三到五句话的理由,解释为什么你这样排列。“这样就可以了。或者也许我们已经有一份清单了,那我可以把这份清单发给大家做强制排序。好的,这是我们真正在考虑的最终清单。帮我排个优先级,做一个强制排序。然后再附上理由。
给我一些自由书写的内容,解释你为什么认为应该这样排列。也许我们已经确定了前五项优先事项。很好。我可以在会议之前把前五项发出去:“嘿,我们确定了这些东西。“对于每一项,我想了解你认为合理的时间线是什么,需要多少个冲刺周期,等等。我们可以收集这类信息,以便开始做估算,因为这会影响预算、我们向董事会汇报的内容等等。但不管怎样,先想清楚这次会议要讨论什么,然后我把它独立地发给每个人,并说:“不要回复全部,独立地发回给我就行。”
如果是反复出现的决策,你实际上可以创建一个评分标准,放在 Airtable、Coda、Google Sheets 或者任何什么上面——我不在乎用什么工具。人们可以在那里输入他们的决策,但看不到其他人的决策。比如用 Google Forms 来做,结果会汇入一个只有你能看到的电子表格。这是一个很好的方式。总之,你这样做了之后,就可以看到所有人的意见,然后再把它发给整个团队说:“各位,在我们进来讨论之前先看看这些。“注意,你们仍然在作为一个团队工作,只不过你们是在不同时、不同地工作,而不是在同一个房间里同时说话。这有一个专门的术语,叫做名义小组(nominal group)。也就是说,在那个时刻,这个小组是独立地、异步地工作的。
让决策回归独立判断
Annie Duke: 如果你能让团队做到这一点——我确实有一些公司,我不是给他们做长期咨询,只是进去做了简短的交流或类似的事情,但他们确实真正落实了这一部分——这就是非常关键的一步,影响巨大。然后在做决策时也用同样的方式。决策不应该在会议室里做出。我个人很喜欢单一决策者模型,但不是所有人都能接受这一点。不过你也可以设置一个投票平台,让人们在私下里投票表达自己的倾向。你可以用各种方式来实现,但就是不要在会议里做。不要在会议里做决策。
“对齐”是一个不应该使用的词
还有最后一点我想补充的,这是一块你真的需要锻炼的肌肉——我认为很重要的一点是,要理解”对齐”(alignment)这个词,就是我们”作为一个团队我们所有人都是对齐的”那种用法,这个词是很愚蠢的,不应该被使用。我知道我说得很苛刻,但这是事实。它之所以愚蠢,是因为这种情况根本不存在。你把十个人放在一个房间里,他们都是不同的人,有不同的观点,他们永远不可能走出房间时彼此意见一致。如果期望他们必须达成一致,那是非常糟糕的,原因有几个。第一,这不符合现实,而我不喜欢从一件事中出来的时候,现实并没有真正起作用。
也就是说,这不是现实。人们其实并没有达成共识,他们在决策上并没有真正”对齐”。那只不过是让你自己感觉好一点的东西罢了。所以我认为这是第一个问题。但第二个问题是,如果目标是”对齐”,如果目标是达成一致,那么会议就会变成一种胁迫式的场景,而你绝对不希望这样。所以,我表达自己想法的方式应该是传递我为什么相信我所相信的东西,而不是去说服别人我是对的。因为如果我的目的是说服别人我是对的,那就会变成胁迫式的,这就很糟糕。
所以你必须接受走出会议室时的状态。这就是”尽管如此”(nevertheless)。走出会议室时要理解,经过讨论之后,不是说 Lenny 你的观点就一定会改变。当然,它可能会变。我可能说出我的看法,而且说得实在太精彩了,以至于你某种程度上改变了想法,对吧?你可能会想:“哦,我对这件事的看法不一样了,实际上……”但也可能你没有,你仍然持有和我截然不同的看法。而领导者必须说:“这没关系。我听取了你们双方的意见,我知道这个方向不是 Annie 想要的。尽管如此,考虑到我们所有人的目标,这个决策将会是这样的。你们最终没有达成一致完全没问题,因为这就是现实。而这让我能够更好地把握决策空间的全貌。“这些东西我确实已经成功让一些团队付诸实践了,而且它们的效果确实非常显著。
Lenny Rachitsky: 这其中隐含了一个你刚才有所触及的问题——本质上就是有一个 DRI(直接负责人),一个单一的决策者。有时候人们会开始觉得:“哦,我的声音没有被听到。我没有太多发言权,我无法参与这个决策。“你刚才谈了很多关于如何让人们感到被纳入其中、如何在过程中收集反馈。但如果你试图转向这种模型,在让人们觉得”好吧,我实际上对这个走向是有影响力的”这方面,你有什么建议吗?
从胁迫式模型转向好奇心模型
Annie Duke: 有的。从胁迫式模型转向——我想你可以称之为一种好奇心的模型——有一个非常好的地方。你希望保持好奇,而不是胁迫,这意味着会议上人们讨论自己观点的方式,应该是在传递信息,而不是试图说服任何人。所以一旦我们脱离了那种胁迫式模型——当我说”胁迫”的时候,我并不是说有人故意要建立一种胁迫文化。有时候确实是这样,但在大多数情况下,每个人都在努力做好工作,没有人想要建立胁迫文化。但是,一旦你说”我们都同意吗?我们都对齐了吗?“,一旦你允许人们——我相信你肯定参加过这种会议——一旦你允许别人打断别人,一旦你允许别人说”我觉得你错了”、“我不同意”、“原因如下”,这些都是在制造胁迫氛围的事情。打断别人就是在让他们噤声。说”你错了”——好,现在部落对立形成了,人们不会再保持开放心态,他们会固守自己的立场,这一切都非常糟糕。所以我们先摆脱这些,光是这一点就已经会有很大的帮助。
但当我们回想会议通常是怎么开的,同样,总是有各种交叉发言,有些人不说话,有些人在说话,如此等等。当然,不是每个人都觉得自己同样被倾听了。但是当你以名义小组的方式工作的时候——我们来举一个简单的例子:我们要预测推出这个产品功能需要多长时间。好,那我发给所有人:你的点估计是多少?你的下限是多少?你的上限是多少?在时间线上来说,你可以用什么方式来表达都行,比如你认为需要多少个冲刺周期之类的。现在,每个人都独立地给出自己的预测,并附上理由解释为什么他们相信那个数字。然后你走进会议室,主持一场讨论,每个人都说出自己的估计是多少,以及为什么这样认为,大家也可以提问。“我有个问题”永远是用于澄清的,就是”我不太理解”。作为领导者,你的做法是——让我举一个真实的例子来说明,这样会更直观。我曾为一家公司做过一次这样的讨论,问题是关于远程办公的——公司在远程、混合等方面想要什么样的政策。然后在某项政策是否应该跨职能部门保持一致这个问题上,存在很多分歧。
Annie Duke: 比如我们来找一个持”不,不应该跨职能保持一致”这个观点的人。现在他们要说明为什么认为不应该跨职能保持一致,他们会说类似这样的话:“嗯,不同职能有不同的需求,对吧?“有些职能必须在办公室,比如 IT 部门就是例子,而其他一些职能更具协作性和创造性等等,人员在办公室会更有意义,而工程师则无所谓,对吧?所以不同职能在需要在同一空间工作这件事上的需求是不同的。我不是在同意或反对这个观点,我只是在转述某人说了什么。那么作为引导者,我永远不会说”哦,我同意”。我所说的是,我只是想确认我理解了你说的内容,然后把它复述回去。
所以我会说,“我听到你说的是,并非所有职能在远程工作的效果、现场工作的效果或灵活性方面都是一样的。你的意思是有些职能必须现场办公,没有商量余地。然后其他职能中,协作要素使他们在同一办公室更重要,而有些职能远程办公完全没问题。是这个意思吗?“然后他们有机会说”是的,就是这个意思”,或者”其实我的意思略有不同,让我重新说明”,然后我再复述回去。好了,基本上你就是这样做一轮,你请每个人发言,然后复述他们所说的,而不发表你自己的意见。我不明白,Lenny,告诉我,在这种情况下怎么可能有人觉得自己没被倾听。
Lenny Rachitsky: 确实如此。如果有人只是澄清并表明他们完全理解我在说什么,即使最终决策不是我希望的那样,我也会感觉很棒。
Annie Duke: 对,这就是关键所在。然后实际发生的情况是,房间里的人会感到更加——用心理学术语来说是”被赋予”(endowed)——对决策有更多的拥有感。换句话说,他们觉得自己对决策有所有权。无论最终决策是什么,他们通常能在其中看到自己的影子,因为他们的声音被听到了。而且他们通常也会理解,没有人——没有一个人——能在最终决策中完完全全得到自己想要的一切,因为他们也看到了团队中真实的意见分歧范围。你会发现在那种情况下,存在着大量的、大量的分歧,而如果你让大家一起讨论,这些分歧是看不到的。讨论会压缩意见空间,对吧?
举例来说,尤其是如果你是资历更深的人,如果你说”我认为这需要三个月才能上线”,而我认为是四周,你永远不会从我这儿听到四周这个数字,永远不会。但如果我们各自独立给出意见,你实际上会听到我认为只需要四周。你资历更深可能确实更正确,但我可能也有一些有价值的观点。你应该听听我要说什么。这也让我有机会从你那里学习。但因为我没有先听到你的意见,我就不会把我的意见向你靠拢。所以它实际上扩大了你能看到的团队分歧的范围,然后人们就会对决策不完全是自己想要的这件事感觉好得多,因为他们会意识到,“哦,这确实是一个难题。大家在这件事上真的有很多不同看法。”
Lenny Rachitsky: 太棒了。我觉得这对很多人会很有帮助。作为这个小话题的收尾总结,核心建议就是分开头脑风暴。所以我想首先是发现、讨论、决策。独立地发现想法,本质上就是头脑风暴。你非常倡导独立头脑风暴,自己坐下来,独立思考各种想法。然后把大家召集到一起讨论他们各自想出来的内容,特别是他们意见不一致的地方。然后理想情况下由一个人在做完所有信息输入之后做出决策。
Annie Duke: 对,可以是一个人。我的意思是,我也和一些人合作,他们是一个合伙人团队,有六个人投票,但他们仍然是独立投票的。他们会进入一个讨论平台,我不知道你的投票是什么。六个人也不必都同意。我只是认为很重要的是,一旦你的人数超过一个人,你就不应该期望所有人达成一致,我觉得这点非常重要。而且这不仅适用于头脑风暴,还适用于预测、任何形式的项目规划、预算编制。是的,比如我们在 First Round 就是这么做的,我们有一个结构化的讨论平台来评估一家公司是否值得投资,因为事实是一回事,你对这些事实建立模型的方式又是另一回事。所以这很大程度上涉及投资者对创始人、产品等方面的意见评估。
我们只是围绕如何收集这些意见建立了一套结构,这真的很有帮助。另外我想补充的一点是,你们实际上可以在同一个房间里,仍然独立地发现信息。这种情况在现场讨论中随时都可能发生——人们在讨论的时候,某个话题冒出来,比如有人建议了一个新功能之类的,然后大家开始说”是啊,但那需要……”,你就要说”停。好,每个人拿出一张纸。“所以你仍然可以在现场获得同样的独立性,既可以是事前准备好的,也可以是即兴进行的。不过是的,你总结得非常好。
Lenny Rachitsky: 如果大家只从这次对话中带走这一件事,我觉得就能产生很大的影响。说到 First Round,第一,我们其实刚刚请了 Todd Jackson 来播客聊——
Annie Duke: 我很喜欢 Todd。
Lenny Rachitsky: ——我也很喜欢 Todd,聊的是产品市场契合度。那期节目会在这期之前播出。另外,我还问了你在 First Round 的同事 Brett Berson,应该问你什么问题。
Annie Duke: 哦,好的。
Lenny Rachitsky: 对,对。顺便说一句,你在 First Round 的头衔太棒了,“特别合伙人”(special partner),我以前从来没见过这个头衔。
Annie Duke: 我确实是。这是一个只属于我的头衔。
Lenny Rachitsky: 天哪。
Annie Duke: 我非常喜欢 Brett Berson。
Lenny Rachitsky: 我也是。
Annie Duke: 所以我很好奇,我很好奇他说了什么让你问我的问题。
Lenny Rachitsky: 没有什么太劲爆的,和你谈到的话题方向差不多。你提到过一个非常有趣的框架,关于如何在结果很长期的情况下评估短期内的决策质量。比如投资。我们在商业中做的很多决策也是如此,那些你现在需要做的决定,要等到几年后才能知道结果。你能谈谈你在这方面的建议和见解吗?
Annie Duke: 天哪,我太高兴他问的是这个问题了。
Lenny Rachitsky: 嗯,我懂的。
Annie Duke: 好,我能先给一点背景铺垫吗?
Lenny Rachitsky: 当然可以。
风险投资中的反馈环路
Annie Duke: 所以,在和 First Round 谈之前,我还有另一个客户,他们也很棒,Renegade Partners 的 Roseanne Wincek 和 Renata Quintini,她们非常出色。在最终和她们建立合作之前——她们做不同阶段,First Round 显然是 C 轮,Renegade 更偏 A 轮 B 轮,偶尔涉猎一点 C 轮——在这之前,我和不少风投公司聊过。他们在《对赌》出版后对和我交流很感兴趣,那大概是 2018 年。我发现了一个共同的主题,一个贯穿所有人的主题。第一个表现是:“你说的那种决策方式我们不需要,因为我们看到好创始人的时候就是知道。“这句话从很多人的嘴里说出来了。而我的回应是,好吧,我毫不怀疑,但难道你不想把这个显性化吗?把它显性化有很多好处,不仅仅是在当下提升决策质量,更重要的是它能让你更好地闭合反馈回路。这是我觉得相当意外的一点。但真正让我觉得很有意思的是有人告诉我:“你说这些东西不适用,因为我们的反馈回路是十年。而在扑克中,你马上就能得到答案,你立刻就知道这一手牌是赢了还是输了。所以你那种关于决策的思路并不真正适用。“——直到我遇到 First Round,然后又遇到 Renegade,他们才真正听进去了我想说的东西,因为我给所有人的回答都是一样的。
我们先放下”难道你不想把它显性化吗”这一点。我首先要说的是,扑克比你想象的要嘈杂得多,因为当我赢了一手牌的时候,我并不知道为什么。实际上我确实需要等很长时间,因为我需要打很多很多很多手牌,才能真正知道我到底有没有优势。因为我实际上知道的很少。一方面,我几乎从来看不到对手的底牌,所以我处在一团迷雾般的不确定性之中。但除此之外,我说的最主要的一点是:你怎么可能认为反馈回路是十年?而我认为这正是真正引起 First Round 注意的地方,因为当我和 Josh Kopelman 谈到这个问题时,他说:“什么意思?我们要十年才能退出。“我说:“哦,不好意思,你是不是投完资之后就像 Rip Van Winkle 一样去睡觉了?然后十年后醒来问一句,‘嘿,结果怎么样?‘还是说中间其实发生了各种各样的事情?“最简单的一件事,最简单的就是:它在 A 轮融到资了吗?
我通常会收到一点小小的反驳:但我们不是为 A 轮投资的。我说:“我知道。“但是你有没有过一家退出估值超过十亿美元的公司没有在 A 轮融到资的?答案是没有。我说,好吧,那听起来这就是必要条件。可能不是充分条件,但它是必要条件。而且它实际上是一个与良好退出高度相关的信号,比 C 轮投资时的信号相关性更强。然后呢,对,还有 B 轮。这还没算上你能观察的其他各种事情,比如你和 Todd Jackson 聊过的那些。它是否实现了产品市场契合度?我们知道它最终要成功,就必须要实现产品市场契合度,对吧?所以你可以观察这方面的进展,还有一些关于吸引力的通用指标,净新增 ARR 的变化,留住顶尖人才的能力,流失率等等。有太多太多你可以观察的东西了,所有这些都是为了让那个大结果发生而必须先发生的事情。
不存在长期反馈回路
好,所以这意味着——我要在这里做一个大胆的声明——不存在所谓的长期反馈回路这回事。反馈回路有多长,是你自己决定的。选择活在长期反馈回路中是你的选择,你也可以选择缩短反馈回路。缩短反馈回路的方法是问:哪些是获得良好退出的必要但不充分条件?这是一方面。或者说,哪些事情与我最终期望的结果是相关的?这意味着,当你在决策点的时候——比如在 First Round 的场景下,我要投资一家公司——你必须理解的是,你实际上是在做一个关于世界将如何展开、未来将如何展开的预测。而你预测的那些事情,是可以追踪的,你可以把它们追踪回那个决策,而且速度可以相当快。想想 2021 年,有些公司在种子轮之后六个月就在融 A 轮了。现在大概是十六个月左右。但即便如此,假设你唯一决定做的事情就是:我要预测这家公司在 A 轮融到资的概率。然后那些公司开始陆陆续续在 A 轮融到或没融到资,你在十六个月内就能知道结果。我的问题是,Lenny,十六个月是不是比十年短?
Lenny Rachitsky: 这大概就是他让我问你这个问题的原因吧,因为我确实非常想问。我的意思是,我对此有非常强烈的观点。反馈回路有多长完全取决于你自己。如果我回到早期和人交谈时听到的一些说法,我会说,我认为让反馈回路保持长期存在是有一定心理安全感的,主要有两个原因。第一是,你看,如果我早期投了 Uber,现在我成了明星投资人什么的,我真的想知道自己到底行不行吗?我不想,对吧?我真的不想让世界知道这一点。我是说,如果我确实厉害,那当然好。但感觉上他们已经认为我厉害了,因为我碰巧早期投了 Uber。
所以既然人们已经觉得我厉害了,从心理角度来说——不是从投资质量角度,而是从心理角度来说——我只是在失去那个好处。好,但问题来了。我为什么早期投了 Uber?是因为我对某个新兴市场中一个真实痛点有了深刻洞察,等等等等?还是因为我哥们创办了 Uber,然后我说”行,我给你点钱”?显然我说的都是极端情况,但我们其实并不知道当时的决策质量如何,对吧?我们只知道你得到了一个好结果,而既然你得到了好结果并且人们对你评价很高,你还能获得什么呢?所以就让那个反馈回路待在那里,让人们继续保持对你的看法,这样真好,感觉很舒服,对吧?不去真正弄清楚答案,因为我为什么要弄清楚呢?除非你真的非常关注决策质量,那你才会想做这件事。这就是心理安全感的一部分。而它背后最核心的是,人类非常非常难以接受在当下感到自己是错的,即便这在长远来看对他们有帮助。就是很难。反馈回路越紧密,你就越有可能在短期内发现自己错了。当然,如果你专注于决策质量并且擅长这方面,这会帮助你学习和改进决策,长期来看你实际上会做得更好。但人类出了名地擅长为了在短期感觉良好而牺牲长期利益。这就是为什么我们都在吃巧克力和纸杯蛋糕之类明明知道对我们不好的东西,因为它感觉好。
决策中的自我叙事与反馈回路
Annie Duke: 我们很大一部分的决策行为,都是在试图推进一个积极的自我叙事——觉得”是的,如果我做些什么,五年后我对自己会有一个更积极的叙事”。我们大多数人会说,“不,我真的不想做那个交换。我宁愿现在就感觉良好。“而且我可以利用我们生活在幂律分布中这个事实,利用幂律的影响来确认许多我希望相信关于自己的事情。如果你把这个拿走,把不确定性从我这里拿走,那会非常困难。我想说的是,这正是我喜欢 Renegade 和 First Round 的地方——他们就是那种”我想知道”的人。对我来说,想到自己在其实做不好决策的时候却以为做得很好,那才是真正可怕的事情。这才是我真正在乎的,我觉得这一点非常特别。
Lenny Rachitsky: 听到这些让我很想成为 First Round 的 LP。不是说他们会让我进去,但知道你在那里推动着各种改进、以及他们思考问题的方式,真的很令人振奋。我很好奇你能否分享一下,在你做的分析之后,他们有没有因此调整了什么具体做法?他们是如何评估的?
First Round 的决策框架改造
Annie Duke: 首先,我想说的是,我刚进去的时候他们其实并没有对决策做太多记录。他们会投票,投票结果有记录,所以他们知道谁说了是、谁说了否,但除此之外没有太多其他信息。所以第一件事就是解决这个问题。好,我们真正认为的判断一家公司该不该投的模型应该是什么样的?宽泛地说,你在考察市场、团队、创始人、产品这几个大方面。我们要确保这些意见不是简单的”好”或”不好”,而是采用一到七分的评分制,这样我们就能在合伙人之间获得一些精确度和区分度——比如,对市场强度的评价。我们要确保大家对这些问题有统一的定义,你可能会很惊讶,实际上很多人并没有。
当我思考市场和市场质量的时候,我脑子里想的可能和你想的非常不同。所以我们要确保对此有一个共享定义,而这会体现在我们所说的”中介判断”(mediating judgments)中——也就是你在对市场做出总体判断之前,先做出的与市场相关的判断。你可以想想”竞争格局”这类东西,你会先对它做出判断。这些就变成了中介判断,本质上是一种隐含的定义。
把这些建立起来之后,你还要思考哪些预测是重要的。其中一件你已经知道的事情是——你要预测这家公司在 A 轮融到资的概率。这是一个巨大的改变,一种非常不同的决策方式。因为我在那里已经五年了,现在我们可以真正去回溯——比如以整个合伙人为整体,去看他们对各个组成维度的评分,也就是他们建模时所用的”什么构成一笔好投资”的各个要素,以及他们做出的那些预测。
我们实际上知道这些公司后来的发展。最简单的层面来说,我们知道数百家公司有没有在 A 轮融到资。现在我们可以真正去看,合伙人们对这件事的预测到底准不准?他们的预测是随机的,还是比随机好?我们确实知道这个答案,并且可以把结果反馈给他们,让他们了解自己对这些决策要素的准确度。因为事实是,不管一个种子轮投资人有没有明确说出来,那家公司能不能在 A 轮融到资,本身就是他们决策的一部分。它已经包含在决策中了。不管他们有没有明确做出这个预测,他们都在做这个预测。所以我们在 First Round 做的事情就是——让它变得明确,因为你反正是隐性地在做,那我们不如把它显性化,然后就可以开始审视你的准确度。
我们可以把结果反馈给你,让你知道自己有多准确,这反过来会帮助你变得更准确。我们还可以做更多——因为对于任何一个年份的基金,我们知道哪些是最好的公司、哪些是最差的。记住,我们让人们对各维度做了一到七分的评分,比如市场质量。我们可以横跨整个合伙人团队来看,“看,Lenny,当你觉得市场很好的时候,这和那家公司未来的表现有多大程度的对应?当你说市场很差的时候,这又和那家公司最终的表现有多大程度的对应?“我可以过来跟你说,“哦,Lenny,顺便说一下,你对市场的判断非常出色。”
你看到好市场的时候真的能识别出来,而且它和公司后续的发展对应得非常好。但你在创始人判断方面就没那么强了。或者也许你各方面都很强,也许不是。所以我们现在可以开始给合伙人提供对自己决策的洞察——不仅帮助他们改进决策,还帮助他们理解不要在某些对自己来说没那么有预测力的维度上过度加权。我们现在还可以根据证据来调整决策评分标准。
决策评分标准的第一版总是把合伙人的直觉显性化,但之后我们可以把这些数据循环回来,理解——也许合伙人认为重要的这个东西,实际上对任何合伙人来说都没有预测力。举个例子,我们可以开始基于数据来优化评分标准。这些事情以前都是不可能做到的,因为在那之前,如果我一进去就说”好,让我们看看决策质量”,我怎么做呢?我完全不知道人们为什么……我只知道他们说了是还是否。所以如果没有那些信息,让他们真正认真地优化决策是非常困难的。
分析中的意外发现
Lenny Rachitsky: 我非常想知道哪个合伙人做的决策最好。我知道你不会说的。
Annie Duke: 不,我不会说的。整个合伙人团队整体上非常出色,这是我们都知道的。我想告诉你的是,所有合伙人都有强项,也都有弱项,而且它们并不完全重合,这其实是一件非常好的事情。这就是为什么我要说——这就是为什么让不止一个人参与一个决策的价值所在——有些人在评估市场的某个特定维度上非常强,有些人则在评估创始人的某个特定维度上非常出色。他们之间有重叠,但也有 Todd 在某方面特别厉害、Josh 在某方面特别厉害、Bill 在某方面特别厉害的情况。所以这是一件好事——每个人都有强项,每个人都有弱项,而且它们不完全重合。从这里你就能看到,获取多元意见、把决策真正拆解成各个组成部分,能够让你看到多元观点作为决策输入的真正价值。
Lenny Rachitsky: 这个话题我可以聊一辈子。也许让我再问一个问题,因为我实在太好奇了。到目前为止,这个分析有没有得出什么令人惊讶的发现?比如,“哦,也许市场没有我们想的那么重要”,或者”这个人在某方面特别厉害”——
关于决策评分标准的意外发现
Annie Duke: 是的。我觉得总体来说,当你创建最初的决策评分标准时,有些东西是大家会拍桌子极力主张的,他们认为这对做出决策特别重要。我们发现的一件事是,有时候他们的直觉绝对正确——他们拍桌子强调的东西确实非常有预测力,不仅对他们自己如此,对其他合伙人也是一样。但有时候它完全没有预测力,而这些同样都是拍桌子式的强烈主张。也就是说,假设你对某件事拍桌子力主,有时候你拍桌子的那个点对你和对其他合伙人来说确实很有预测力,能很好地预测公司的表现。但有时候,当你真的拍桌子强调某件事时,不仅它对其他合伙人没有预测力,对你自己也没有预测力。
我觉得真正重要的是要理解这一点——这也是为什么在提升决策质量时,把隐性的东西显性化如此关键——我们的直觉有时是对的,有时是错的。不是说直觉一文不值、你的直觉完全错误,显然不可能是这样的,否则我们早就死了,对吧?所以你的直觉有时是对的,但也确实是错的。而如果你不把它显性化,你就不会发现它什么时候是错的。
你不会发现它什么时候偏离了基准,而那就是灾难。所以我觉得真正有趣的就是这一点——你以同等的激烈程度和信心认为某个特定因素真的非常重要,有时候确实如此,有时候则不然。这很令人惊讶,因为我们说的是真正出色的专家级人物,我想我们所有人都会对直觉有一种直觉判断,对吧?你直觉地认为,既然他们这么厉害,那他们对什么东西重要的直觉判断肯定也是好的,但不一定。关键就在这里——不一定。对,我觉得这可能是最令人兴奋的发现。
Lenny Rachitsky: 你还是保持神秘,不过我还是很感谢你分享的这些。
Annie Duke: 嗯,我必须保持神秘。
Lenny Rachitsky: 是的,我理解。
Annie Duke: 我不能泄露商业秘密。
事前验尸与终止标准
Lenny Rachitsky: 我想聊聊另一个框架,实际上我听说很多公司在用,在产品领域经常出现的一个概念是事前验尸,本质上就是提前设想可能会出什么问题。你能谈谈这个吗?因为我觉得它容易实施,非常有效,而且很多人已经在做了。
Annie Duke: 好的。事前验尸很好,但前提是你必须附带一个预先承诺。我想把这一点说得非常清楚。我们发现,事前验尸这个东西——这个工作实际上是我和 Maurice Schweitzer 以及 Linnea Gandhi 一起做的,他们都在宾大——当你让人们做事前验尸时,它通常并不会真正改变他们的计划或行为。我觉得我们有一种感觉,认为如果你做事前验尸,想想事情可能会以哪些方式出错,那就会改变你的计划,但其实未必——除非你专门为此目的使用它,并且说:“好,我们要做这件事,但让我们想想根据这些信息如何调整计划。“但我觉得比这更重要的,是事前验尸能让你设定终止标准。
终止标准就是一组你可能看到的信号,告诉你该转向或该停止了。因为一旦我们真正启动了某个项目,我们在决定放弃时会非常非常迟缓。我相信每个人都有过这种感觉——事情拖得太久了,即使已经超预算、错过了时间节点,而当你最终叫停时,你会意识到自己本应该早得多就停下来。这在各行各业都是如此,原因在于各种认知偏差。最广为人知、可能也是影响最大的,是所谓的沉没成本——那种感觉,“但那样的话我已经投入的所有时间和精力就都浪费了。“也就是说,你会把过去投入的东西纳入考量,以此来决定未来是否继续。所以我们要做的其实就是在这件事上变得更好。要理解的是,当你进入事前验尸流程时,你很可能会启动这个项目——大概率是这样的。
利用事前验尸来设定终止标准。我给你举一个我合作过的销售团队的例子。基本上,我给所有销售代表发了一个提示,内容是:想象你通过 RFP 或 RFI 获得了一个线索,你跟进工作了六个月。现在,六个月过去了,交易黄了。回头看,你意识到当时就有早期信号预示了这个结果,那些信号是什么。这就是事前验尸。你看到了哪些信号,能告诉你这件事会走下坡路?他们想出了各种各样的想法。注意,这不意味着他们不会去跟进这个线索,对吧?但他们识别出了各种各样的信号。我就举其中三个。
RFP/RFI 明显是为某个竞争对手量身定制的——他们觉得这是一个非常糟糕的信号,结果可能不妙。另一个是客户不愿意看产品演示,只想谈价格。显然,这相当不好。还有一个是,前几次会议之后,他们无法让决策者出现在房间里。完整的列表比这长得多,但这三个是代表性的。对于每一个信号,它现在就变成了终止标准——如果我看到了这个信号——然后你给它附加一个行动。在价格那个例子中,他们实际上直接说:“我们应该终止。“如果客户真的不想看演示,只问价格,他们只是想拿我们去压别人降价——我们只是一个走过场的陪标。所以他们在那种情况下直接说:“终止,不再跟进这笔交易。”
这很好,因为销售人员会永远追着交易不放,而领导层会说:“你为什么停止跟进那笔交易?“他们还会因此惹上麻烦。这个方法就能帮助解决这个问题,对吧?在 RFP/RFI 是为竞争对手量身定制的那个例子中,他们也有对应的行动——直接问对方是否在和一家竞争对手合作,以及进展到了什么程度,根据回答决定终止还是继续跟进。在无法让决策者出现在房间里那个例子中,在下次会议中提出安排高层对接。如果对方说好的,那就继续。如果对方说不,就终止。
所以我觉得这才是事前验尸最好的用途——去识别一路上那些表明事情在变糟的信号,然后不是寄希望于看到那些信号时自己会理性行动——这种希望通常不会实现,这就是为什么有很多人会在暴风雪中攀登珠穆朗玛峰,尽管他们根本不应该那么做。利用事前验尸为你识别出的这些信号建立结构,并承诺在看到这些信号时会采取的行动。我觉得这才是事前验尸最好的用法。
让隐性的事物显性化
Lenny Rachitsky: 这真的很有帮助。有趣的是,你的很多例子都可以归结为你经常提到的一个框架——让隐性的东西显性化。First Round 的例子就是一个很好的案例,就是”这是我们所有的假设”,把它们真正显性化地列出来,就能揭示出多少东西。
Annie Duke: 对,然后你就可以审视它们。你可以审视它们,人们可以讨论它们,你可以判断它们是对是错或者别的什么。我想非常明确地说,我不是反对你的直觉或直觉判断。我觉得它有时候可能还挺准的。我只是想让你把它显性化,就这样。
关于放弃
Lenny Rachitsky: 好的。我们没时间深入讨论”放弃”这个话题了,这是你最近的一本书。也许我们以后会专门做一期关于放弃的后续节目,但让我只问一个问题——
Annie Duke: 这是我的错,因为我的回答太长了。
Lenny Rachitsky: 是我的错。
Annie Duke: 向大家道歉。
Lenny Rachitsky: 不需要道歉。希望我们以后还有充裕的时间。那我只问一个问题。我找到了你说过的这样一段话:你应该假设,如果你已经在考虑放弃,那很可能已经过了你本该放弃的时间点。你现在仍然相信这一点吗?这通常是一个好的经验法则吗?就关于放弃这一个话题,有没有什么要点、建议或教训?
Annie Duke: 数据相当有力地表明,到你真正放弃的时候,很可能已经远远超过了你本该放弃的时间点。原因其实很简单——当我们开始做一件事的时候,是在困难的条件下开始的,这种困难来自于启动决策本身的不确定性。当我们开始做一件事时,运气会影响结果,而显然我们无法控制运气,因为运气不在我们的掌控之中。同时还存在隐藏的信息。所以情况是这样的,事后我们会知道自己将会获得新的信息,而这会让开始做事变得非常困难,因为我们想要比实际需要的更高的确定性。这就是为什么贝索斯有那个 70% 法则,试图让人们打消顾虑,愿意接受启动决策中的不确定性。好消息是,当你学到新的东西,而新学到的东西让你觉得”天哪,如果我早知道这些,我就不会开始了”,你通常还有选择放弃的余地。
所以这是好消息。坏消息是,适用于启动决策的那些困难,同样适用于停止决策。换句话说,我们同样是在不确定性下做出这个决定的。所以除非我们继续做已经开始的事情,否则我们无法确定结果到底是好是坏。而除非我们能确定,否则我们就不想放弃。正如 Richard Thaler 所说,大多数人直到这件事实际上已经不再是一个决策的时候才会放弃。换句话说,整个事情已经彻底搞砸了,创业公司已经没钱了,或者你已经登上了珠穆朗玛峰,暴风雪真的已经笼罩着你,你被困在其中。或者正如他所说,直到你已经掉进冰裂缝里了,你才会做出放弃的决定,因为那时候你知道事情真的糟透了。所以人们通常不会辞职,直到觉得别无选择,人际关系、项目、正在开发的产品也是一样,全都适用,因为我们想要确定无疑。
在此基础上,还有沉没成本的问题,就是当我们放弃时,我们会觉得已经投入到正在做的事情中的一切都浪费了。当然,浪费是一个前瞻性问题,而不是回顾性问题,即使我们把它当作回顾性问题来对待,因为它本质上是前瞻性的。如果你今天不会开始这件事,那就意味着你继续投入的一切才是真正的浪费,对吧?但我们一直在这样做。我们继续推进那些不该继续推进的事情,因为我们试图保护过去已经投入其中的资源。然后还有其他问题,比如禀赋效应,即对所建造的东西的拥有感。这在产品领域尤为严重,因为我们一直在构建东西。
一旦我们构建了东西,我们就拥有了它们。一旦拥有了,我们就不想放弃,而且我们实际上比那些我们不拥有的相同东西更看重它们。然后还有内部和外部认同的问题,说到底就是你的身份认同。别人怎么看你?你怎么看自己?你是否觉得自己失败了?这意味着,有太多偏见在阻碍我们停止,以至于到你真正开始考虑放弃的时候,很可能已经过了你本该放弃的时间点。但我们还是会继续下去,直到确信自己别无选择,因为事情是这样的——当你放弃一件事,有人问”嘿,你为什么停了?“你说”哦,我没有选择”,然后把出了问题的所有事情告诉他们,没有人会质疑你。
他们会说,“嗯,听起来你已经尽了最大努力。“但如果你提前放弃,人们会说,“什么?“所以简单讲一个我觉得最好的故事。让我在结束前把它拿出来。Stewart Butterfield 开发了一个叫 Glitch 的产品,Glitch 是一个大型多人在线世界建造合作游戏。在 2000 年代发布的,获得了评论界的高度好评。它是蒙提·派森遇上苏斯博士,非常了不起。他们获得了大量口碑传播和公关报道,没有做任何付费营销。他们有 Andreessen Horowitz 和 Accel 这样的顶级投资者,银行里有 600 万美元,有 5000 个铁杆用户,也就是每周游戏时间超过 20 小时的用户。
问题在于客户获取是个巨大的难题——每有一个每周游戏超过 20 小时的人,就有 95 到 99 个人来了五分钟就走了。显然,这是一个客户获取问题,所有人都知道。所以他们在 2012 年达成一致,开始做付费营销,做了六周。在这六周期间,他们新用户的获取量以每周 6% 到 7% 的速度增长,这非常惊人。六周结束的时候,那是 2012 年 11 月,六周结束后的那个周一早上,Stewart Butterfield 给投资者和联合创始人写了一封信,说”我今天早上醒来,确信 Glitch 已经结束了。”
注意,在这种情况下没有人会做出这样的决定,对吧?但实际发生的事情是,你真正需要看清的是这件事值不值得继续?如果今天重新来过我还会开始吗?这是对未来的预测,对吧?他做了一些粗略的计算,他说,“看,如果我们继续以目前的速度和成本获取客户,需要 31 周才能实现盈亏平衡。“但这是一个荒谬的假设,因为客户获取成本会上升。获取成本必然会上升,因为我们会饱和核心游戏市场,所以它必然会上升。所以他在那一刻意识到,这不是一个能达到风险投资规模的生意,而他要的就是一个能达到风险投资规模的生意。所以即使没有人认为他应该关掉它,他自己看到了他应该关掉它。
放弃的真正代价
Annie Duke: 不仅如此,他还意识到,为了那些以股权作为报酬的员工,他也应该关掉它。他此时已经明白那些股权不值得他们付出的时间,继续做下去对他们不公平。于是他关掉了它。显然,这感觉……谁会这么做呢?对吧?而他实际上会告诉你,他知道自己在做那轮营销推广之前就应该关掉它,但他需要那次推广来向自己证明他确实看清了未来。
这个故事的后续是这样的——仅仅两天后,他就想,“嗯,我是个做创业的人。我想做点新的东西。“而他有一个团队在开发产品时使用的内部通讯工具,所有人都很喜欢。他说,“其实,他们真的很喜欢那个。也许那应该成为下一个产品。“于是他去找投资者谈,他们把资金转投到了新项目上。那个当时连名字都没有的东西,后来得到了一个名字——可搜索的所有公司知识日志(Searchable Log of All Company Knowledge),也就是 Slack。
所以我认为有一点很重要,需要认识到——我们太过纠结于”但我已经投入了那么多怎么办?“而我们忘记的是,当你做一件事情的时候,不仅有不值得做这件事本身的直接成本,还有无法将注意力投入到其他可能对你可用的机会上的成本。以 Stewart Butterfield 的聪明才智,直到他放弃 Glitch,他才看到了 Slack。这才是他如果继续做 Glitch 所要承担的真正代价。如果他继续做 Glitch,Slack 就不会成为我们今天都在使用的产品。
结语与资源
Lenny Rachitsky: 这是一个极其美好的方式来结束我们的对话。我觉得我还有至少十亿个问题想问你,但另一方面,我也觉得我们通过这次对话已经帮助很多人做出了更好的决策,所以我非常感谢你抽出时间来。最后两个问题。如果大家想了解你正在做的事情,或者想与你合作,可以在哪里找到你?听众怎样才能帮到你?
Annie Duke: 如果你对我感兴趣或想合作,可以在 annieduke.com 找到我。我有一个 Substack 叫 Thinking in Bets,欢迎去看看。我在 maven.com 上每学年教两次有效决策的课程。如果你对此感兴趣,可以去 Maven 了解一下。我下一期课程目前定在九月,不过我可能会在五月开一期,还不确定。至于大家能怎么帮我——我联合创办了一个组织叫 The Alliance for Decision Education,我们致力于将改善成人决策的知识带到 K-12 教育中,让世界变得更好。所以我非常希望大家能去看看,如果你感兴趣,帮忙传播一下。
Lenny Rachitsky: 太棒了。我们会在节目备注中附上所有这些链接。Annie,再次非常感谢你来做客。
Annie Duke: 谢谢。非常感谢你,这太有趣了。
Lenny Rachitsky: 我也是。大家再见。非常感谢收听。如果你觉得这期节目有价值,可以在 Apple Podcasts、Spotify 或你喜欢的播客应用上订阅。也请考虑给我们评分或留下评论,因为这真的能帮助其他听众找到这个播客。你可以在 lennyspodcast.com 找到所有往期节目或了解更多关于这个节目的信息。下期见。
术语表
| 原文 | 中文 |
|---|---|
| Accel | Accel(投资机构名称,保留原文) |
| adversarial collaboration | 对抗性合作 |
| Andreessen Horowitz | Andreessen Horowitz(投资机构名称,保留原文) |
| Angus Deaton | 安格斯·迪顿(诺贝尔经济学奖得主) |
| Barb Mellers | Barb Mellers(人名,保留原文) |
| Bezos | 贝索斯(Amazon 创始人) |
| Brett Berson | Brett Berson(人名,保留原文) |
| Daniel Kahneman | 丹尼尔·卡尼曼(诺贝尔经济学奖得主,行为经济学先驱) |
| decision rubric | 决策评分标准 |
| Dr. Becky | Dr. Becky(人名,保留原文) |
| Dr. Seuss | 苏斯博士(美国儿童文学作家) |
| endowed | 被赋予(拥有感) |
| executive alignment | 高层对接 |
| First Round | First Round(投资机构名称,保留原文) |
| focusing effect | 聚焦效应 |
| forced rank | 强制排序 |
| Glitch | Glitch(产品名称,保留原文) |
| hit rate | 命中率 |
| IC | IC(Individual Contributor,销售代表,保留原文缩写) |
| Josh Kopelman | Josh Kopelman(人名,First Round 联合创始人,保留原文) |
| kill criteria | 终止标准 |
| Linnea Gandhi | Linnea Gandhi(人名,保留原文) |
| Matthew Killingsworth | Matthew Killingsworth(人名,保留原文) |
| Maurice Schweitzer | Maurice Schweitzer(人名,保留原文) |
| Maven | Maven(在线教育平台名称,保留原文) |
| mediating judgments | 中介判断 |
| mental time travel | 心理时间旅行 |
| Monty Python | 蒙提·派森(英国喜剧团体) |
| net new ARR | 净新增 ARR |
| nominal group | 名义小组 |
| pre-mortem | 事前验尸 |
| priming | 启动效应 |
| product market fit | 产品市场契合度 |
| Quit: The Power of Knowing When to Walk Away | 《放弃:知道何时离开的力量》(Annie Duke 的著作) |
| Renata Quintini | Renata Quintini(人名,保留原文) |
| Renegade Partners | Renegade Partners(投资机构名称,保留原文) |
| resulting | resulting(Annie Duke 提出的决策术语,指以结果论好坏的思维倾向) |
| RFI | RFI(Request for Information,信息请求,保留原文缩写) |
| RFP | RFP(Request for Proposal,请求提案,保留原文缩写) |
| Richard Thaler | Richard Thaler(行为经济学家,诺贝尔经济学奖得主,保留原文) |
| Rip Van Winkle | Rip Van Winkle(美国童话中沉睡多年的角色,保留原文) |
| Roseanne Wincek | Roseanne Wincek(人名,保留原文) |
| seed | 种子轮 |
| special partner | 特别合伙人 |
| Stewart Butterfield | Stewart Butterfield(Slack 联合创始人,人名保留原文) |
| sunk cost | 沉没成本 |
| The Alliance for Decision Education | The Alliance for Decision Education(组织名称,保留原文) |
| Thinking In Bets | 《对赌》(Annie Duke 的著作,中文译名) |
| Thinking, Fast and Slow | 《思考,快与慢》(Daniel Kahneman 的著作) |
| Todd Jackson | Todd Jackson(人名,保留原文) |
| vintage | 年份基金 |
| World Series of Poker | 世界扑克大赛 |
| 禀赋效应 / endowment | 禀赋效应(拥有感) |
此文档由 AI 分片翻译(translate_long_document)