如何以精益方式招聘、衡量并释放增长 | Crystal Widjaja(Gojek 和 Kumu)
How to scrappily hire for, measure, and unlock growth | Crystal Widjaja, Gojek and Kumu
Gojek’s Early Rapid Growth
Crystal Widjaja: I felt like it was a problem that was very solvable. And we ended up renting a stadium to just hire 60,000 drivers in a couple of weeks. So I think looking back, it was certainly a risk. When I got there it was in a house and I realized I’ve probably made a huge mistake, but we were growing very quickly already, even at that small scale of 4,000 orders per day.
From Investment Banking to Southeast Asian Tech
Lenny: Crystal Widjaja has been leading product and growth teams at some of the largest consumer businesses in Southeast Asia, including Kumu, where she’s currently the chief product officer, and Gojek where she built and led the growth team through the early years of what is now the largest super app in Southeast Asia. To put this in context, Gojek completes more rights per day than Lyft and more food deliveries than GrubHub, Uber Eats and DoorDash combined, and it’s the number one mobile wallet in Indonesia and Southeast Asia. In my opinion, American startups have a lot to learn from startups in Asia and Crystal has been at the ground floor of some of the biggest successes there.
In our conversation, we covered the biggest growth unlocks that Crystal has seen across the companies she’s worked at, what growth investments usually pay off and which often don’t, we dig into growth models, a bunch of tips for accelerating growth, why most analytics efforts fail in how to avoid that, how to hire and structure your growth team, and we also talk about the nonprofit that Crystal started that aims to help young women get into STEM called Generation Girl. Crystal is such a star, and I hope that you enjoy this episode as much as I did. And with that, I bring you Crystal Widjaja. If you’re setting up your analytics stack, but you’re not using Amplitude, what are you doing? Amplitude is the number one most popular analytic solution in the world used by both big companies like Shopify, Instacart, and Atlassian, and also most tech startups.
Amplitude has everything you need, including a powerful and fully self-service analytics product, an experimentation platform, and even an integrated customer data platform to help you understand your users like never before. Give your teams self-service product data to understand your users, drive conversions and increase engagement, growth, and revenue. Get your vanity metrics, trust your data, work smarter and grow your business. Try Amplitude for free. Just visit amplitude.com to get started. Hey, Ashley, head of marketing at Flatfile. How many B2B SaaS companies would you estimate need to import CSV files from their customers?
Ashley: At least 40%?
The Scale of Gojek and Kumu
Lenny: And how many of them screw that up, and what happens when they do?
Ashley: Well, based on our data about a third of people will consider switching to another company after just one bad experience during onboarding. So if your CSV importer doesn’t work right, which is super common, considering customer files are chalked full of unexpected data and formatting, they’ll leave.
Why Join Gojek
Lenny: I am zero percent surprised to hear that. I’ve consistently seen that improving onboarding is one of the highest leverage opportunities for both signup conversion and increasing longterm retention, getting people to your aha moment more quickly and reliably is so incredibly important.
Ashley: Totally. It’s incredible to see how our customers like Square, Spotify and Zora are able to grow their businesses on top of Flatfile. It’s because flawless data onboarding acts like a catalyst to get them and their customers where they need to go, faster.
Why No Super Apps in the US
Lenny:
Crystal Widjaja: We may have crossed paths on Clubhouse and the audio forums or on the Twitterverse. So it’s really cool to see you.
Crazy Growth Tactics
Lenny: Wow. I just remembered that. That is so right. I think we were talking about Reforge and Eppo. Is that right?
Crystal Widjaja: Yes, that’s right. Good times.
Rapid and Scalable Validation
Lenny: Oh my God. Clubhouse days.
Crystal Widjaja: The days when club does was a thing. They have a thing too, to learn about exponential decay.
Experimenting with Small Samples
Lenny: Oh man. Okay. Maybe we’ll get to that. And we’re going to be chatting a lot about consumer growth and a bunch of stuff along those lines. But before you get into that, you have a fairly unusual path and also geography as compared to many of my other guests. And so just to set a little context, could you just walk us through your career path and journey from, I think as an investment banker initially, and then currently as chief product officer at Kumu and then living in Singapore also? So yeah, tell us all about your path.
Finding the Step Before Conversion
Crystal Widjaja: Yeah. I think my path was certainly a nonstandard one. While I grew up in San Jose, the Bay Area, you could see companies like Lyft emerging around the year that I was graduating college. But really it was how do I graduate college as quickly as possible because this is very boring. So I took a Poli Sci major. I am not a math or a computer science major. I didn’t know what a consultant was because I was just trying to get out of college. I didn’t realize people start looking for jobs before they’d graduate. So the last two weeks of school I was looking on Craigslist, because I was like, “Craigslist is how everyone gets a job.” I’m a first generation American student, so my parents could not help me at all with like, “You should look into this company called McKinsey,” or, “Here are all of the life paths that you have ahead of you.” So I ended up taking an investment banking research job.
And my job there was to figure out how to call startups and analyze their potential for VC financing or M&A advisory. And I barely knew what those words meant at the time. I ended up owning a huge Excel database of 130,000 rows, 60 plus columns. And because again, I am very impatient, I was like, “This is a terrible experience. How would I create a customer database?” And so I ended up Google fooling all of the work needed to build a MySQL database, I presented a plan, and investment banking, surprise, surprise, is not very tech forward. So they looked at my plans and they’re like, “What is this MySQL thing? Isn’t that super expensive? What is open source?” So I ended up leaving that job because I realized that if I wanted to get into something more tech, it would probably not be at a investment bank. So I did took the investment banking strategies that I had learned there and applied the same pattern matching to companies in Southeast Asia.
So my family originally is from Indonesia. I thought I have a kind of safety net, I must speak Indonesian really well just by birth, so maybe that’s a great country for me to look at. So I took the approach of let’s find a company that makes a lot of sense, that I feel I resonate with. And I literally cold called emails some companies. So Gojek being on that list. I literally emailed someone after Googling HR at Gojek and said, “I’m willing to move to Indonesia, take a bet on me.” And they actually did. So I got extremely lucky. Five years fly by insanely fast. I went through building out the data team from scratch. When you have all of the data, you know how much fraud you have in the system. So then I ended up building out the fraud and risk team, picked up performance marketing, and then it was like, okay, now we’re ready to grow. So you have all of this data now take on growth.
Growth Investment Best Practices and Pitfalls
Lenny: Got it. You were very modest about Gojek and the success of that company, and also Kumu where you work now. So just to set a little context for folks that aren’t familiar with these companies, can you share how big they are and how big of a deal they are in Southeast Asia?
Crystal Widjaja: Yeah. They are pretty massive. So Gojek is now called GoTo, they just merged with the largest eCommerce platform in Indonesia. So across Southeast Asia, we had about 170 million users. Southeast Asia has scale. If you ever wanted to work at scale, you would go to Southeast Asia. We had 20 plus different services from transportation to food, shopping, medicine delivery, bill pay, movie tickets. So it was like all of the startups in America in one app, all being built at the same time with the same user base. And so everything was tremendously layered, because you could fill all of these opportunity gaps in the market where a single app would probably not be as sustainable.
So Gojek is massive across Indonesia, Singapore, Thailand, Vietnam. And then Kumu is kind of a super app for social. So Gojek was very transactional. It was like, “Here’s a job to be done. I want to pay for something and someone delivers it to me.” And with Kumu, it’s more so of a, “I want to do clubhouse, Zoom, Google Hangouts gather around all in one app.” So we cover social feeds, audio, video, multi seats. There’s a ton of different use cases that we serve on Kumu. And Kumu is primarily in the Philippines, but ranks top 10 and a bunch of countries as a top grossing mobile app.
Empirical Retention Rate Benchmarks
Lenny: So with Kumu you joined when they’re already doing fairly well. But Gojek, as you said, you joined very early. What did you see in that company that helped you decide to join such a risky, early stage company? For folks that are maybe thinking about joining a startup, what kind of things did you take away of what to look for?
Growth Strategy Modeling Frameworks
Crystal Widjaja: Honestly, it’s probably a lot of luck. But also at that age I realized I have very little to lose. So with Gojek I think I felt like it was the right company because I was able to really clearly understand the value prop. Traffic in Indonesia is crazy. It takes you two hours to go 20 kilometers. So of course you want to take a motorcycle taxi to beat that traffic. Of course, you don’t want to go out and get food and then have to come back this long pathway of two hours. So I think taking that Warren Buffet approach, I knew that the product made sense. The market made sense as well. So drivers, there were already a thing, but it was very hard to connect them to the consumer.
It was painful to haggle prices. There were lots of restaurants scattered across Indonesia. So the value prop and the market made sense and the channel by which you would do it through this mobile app made a little bit less sense at the time because most drivers didn’t have a mobile app, but I felt like it was a problem that was very solvable. And we ended up renting a stadium to just hire 60,000 drivers in a couple of weeks. So I think looking back, it was certainly a risk when I got there, it was in a house and I realized I’ve probably made a huge mistake. But we were growing very quickly already, even at that small scale of 4,000 orders per day.
Lenny: I want to spend a lot of time talking about what you learned, driving growth at these companies. But one quick question. So, Gojek’s the super app where you do a lot of stuff in one app. Do you have any insights into why a super app hasn’t emerged in the US?
GoPay Growth Lever: Drivers as Salespeople
Crystal Widjaja: Yeah. I think the sentimentality of a conglomerate is very different in Southeast Asia. So we’ve grown up with a specific conglomerate owning, not just the mall that you go to, but also the apartment building that you live in, and the school that you go to. And so they’re very well integrated and there’s this sense of trust in a conglomerate. Whereas in America we already shy away from, does Google know too much about me? There’s also, I think, the second aspect of it, which is that in Asia, we’ve kind of leapfrogged to the computer era. So everyone has a phone, but you may not even have a computer in the entire household. And so when your phone is full, are you going to delete a photo of your kid or are you going to delete this app? You’re probably going to delete the app. So for anyone to really survive, it has to be part of this super app concept.
The Physics of Kumu’s Growth
Lenny: Oh wow. I’ve never thought of it that way. That you don’t have a lot of space on your phone and so you want one app to do a lot of things.
Crystal Widjaja: That’s right. So there’s a decision factor that you don’t really have in the US because the cloud storage and device capacity there is a little bit bigger.
Key Unlocks for Early Growth
Lenny: Interesting. So in the US you can have different apps to be… Basically a super app doesn’t have to be the best at everything. The fact that it does enough and everything good enough. Wow, it’s fascinating.
Crystal Widjaja: You just need to get the job done.
Beer Subscriptions and the Pause Feature
Lenny: Amazing. Okay. That’s super interesting. Okay. So transitioning a bit to growth and things you’ve learned along the way. So you talked about how, I think Gojek, you said hired tens of thousands of drivers really quickly. Are there things that startups in Asia do that you think companies in the US should do and can learn from in terms of growth?
Why Most Analytics Work Fails
Crystal Widjaja: Yeah, so we did crazy things. If someone told you, in the US, that they were going to rent out a stadium, pre-load a bunch of mobile devices, market that drivers should come here in mass for a job fair. They’re going to give them a phone and send them on their way, some people would say, “No. That’s crazy. Won’t we get in trouble.” And to an extent, maybe that’s true. So maybe there are some limitations there, but this concept of doing things that are somewhat crazy, but validate a point, doing stuff that don’t scale, especially I think is really the bread and butter of what we did at Gojek. We were insanely scrappy.
We would do things as simple as wanting to test a subscription feature, which was just released in Singapore a couple weeks ago. We ended up saying, “We have this voucher system that we can distribute vouchers in the back end. We obviously know our driver’s phone numbers. Why don’t we just add them to a WhatsApp group?” We’ll add a hundred drivers randomly to a WhatsApp group. We’ll tell them, “Every time you are on a ride with a customer, try to sell them this pitch. You are the only driver who can sell a subscription package. Have the customer give you $10. Text us when they say yes. Someone will be sitting by this phone all day, every day.
We’ll look up the customer that you were on a ride with in the backend, we’ll give them the vouchers in the back end, and then we’ll deduct $10 from your balance.” It works. It’s really this Wizard of Oz experience. We don’t have to build anything. I coordinated with a bunch of interns and we were able to validate some of the value prop and conversion rates that we would expect in a subscription service. When we wanted to do a new onboarding screen, but turns out we have lots of engineering work to do, we took a screenshot of the screen as is, and we just had our designer put what the onboarding flow might look like if we had to overlay it on top of the screen.
And we just sent that as an in-app message. And then eventually I think finding stuff that does scale intuitively. We knew that we were sending out lots of fake features through things like Typeform surveys. Things like a personality quiz can be very easily done through Typeform. And we realized that if we built in the in-app webpage and we made it easier for us to do a website deployment on our backend side, we wouldn’t have to wait for a mobile app release to test some of these new features out that could be done on web. So it’s really just like, what is the user experience that we want to create? How do we manifest that as quickly as possible? Let’s just try that first.
How to Spot Data Tracking Issues
Lenny: Going back to the stadium example. I know you said that you hired a stadium full of people, I didn’t realize it was actually a stadium that you-
Crystal Widjaja: That it was literally a stadium that we rented, like a football field, a couple football fields if I’m not wrong. It was long lines, boxes of phones and SIM cards. So it was a lot of just doing really hard work to get to that scale.
Event Tracking and Taxonomy
Lenny: Wow. I know you do a lot of advising too. Do you advise startups to be more scrappy and do things that don’t scale? I imagine because in the US the culture is a little different.
Crystal Widjaja: The only thing better than knowing… If you have data of what your customers are doing, that is the best data you could ever get. And so if you don’t have a tested hypothesis, if you can’t think of a way to run an experiment, then honestly that idea is pretty useless. Maybe it makes sense to the market, to the model, but you could have weird consumer sentiments. Not everyone is a rational actor. So testing the actual experience and seeing how people respond to it, that’s the best possible data.
Recommended Data Tool Stack
Lenny: Pulling that thread a little bit, for startups, experiments are often hard because there’s just not enough data and enough users. How do you think startups should approach that? Can you run experiments when you’re really, really early?
Crystal Widjaja: You should. Even if you have a sample size of 30, the data you get back, generally, does not change but its precision will. So mathematically speaking, you’re going to get the same level of trends, but the precision at which you understand those trends will become more deep if you have more data. But the underlying information that you’re getting out of that won’t be very different at larger scales. So what’s better than having 30 data points? Certainly having 100. But what’s better than having zero is definitely 30.
Building a Growth Team
Lenny: Fascinating. So contrarian. Running experiments at 30 people. I love that.
Crystal Widjaja: You have to. Every idea is so cheap at that scale. You could do things that don’t scale dramatically better with 30 people than at 100 if you’re testing.
Hiring Early Growth Talent
Lenny: Just to pull on that a little bit, when you’re running an experiment with 30 people, what do you look for? You’re looking for 20 of them to do something, a large percentage of that group does something?
Crystal Widjaja: So everyone wants to go on retention. They want to see that users are doing this thing, and they want to get from step zero to 100 really quickly, but they don’t realize that users make decisions based on succeeding events. So what’s one step before the user makes that decision? What are the things that they have to do, the things that have to be done? So we’re always looking for what is a specific reason that this user might have converted? For things like GoFood it would be things like when does a user try a new merchant if what people are ordering right now or just food that they already trust and know. If you need to have trust in order to purchase food from a merchant, how do we generate that trust? So we actually hacked it by connecting people’s Facebook connect login.
So we had already had permission to look at who they had connected with on Facebook. We actually looked at the food that their friends had purchased and used that as a data set of, “Hey, here’s food that Lenny purchased and liked. Maybe you would like it too.” And so that was one way to hack the trust factor. And we did find that when we told people, “This friend purchased from this merchant,” you would be twice as likely to purchase from a brand new restaurant then users who did not have this feature. And that increases GMV, that eventually gets you to the conversion rate that you wanted, but it solved a different problem. Before how do I convert, it was how do I solve for trust? How do I break the barrier of facilitating that decision making process, that aha moment, by fixing the setup moment, which was trust?
Experiment Design Interview Questions
Lenny: And that’s just a general rule of thumb you have. Don’t use retention as a goal. I know you wrote about this somewhere. Is that a rough rule of thumb you use?
Generation Girl: Getting Girls into STEM
Crystal Widjaja: I think a lot of people thought that I had meant retention sucks, don’t care about it at all. But in reality it was really when you think about retention, that’s just not specific enough. So there is this mental model that I use from made to stick where they’ll tell you like, “Lenny, think of everything in the world that is orange.” And you’re like, “An orange. What else?” And then if you change that structure with sandbox to think of everything orange that’s in a construction site, then you really start to realize and grasp at concrete concepts, and can actually action on them in real life.
Lenny: Got it. Speaking of retention, where have you found products and companies have the most success increasing retention?
Crystal Widjaja: It’s usually the step right before conversion. So if they aren’t sure why the user opens the app or they aren’t sure why the user got to this checkout page, it’s often some copy or the path has been ineffective in some way. I’d like to see founders think about the user psych model that Darius Contractor often talks about. So you need some momentum in that user journey to get them over the hump of some of these very painful user processes like typing in a credit card. That’s a lot of work. How do you lower that friction? And being able to sequence the right steps effectively and just moving around screens actually can do a lot.
Lenny: Going even deeper there. So the companies you’ve worked at, the companies you’ve advised, you’re on the boards of a couple companies I noticed, what have you found to be really good uses of time in terms of growth investments, things that often work? And then a second question, what do you find is rarely successful or people invest a lot of time and ends up not being really useful for growth?
Crystal Widjaja: Yeah. I think I see a lot of founders grasping at straws. So there’ll be this brand new feature that does something different from what people are already doing on our app, like this will make things work. But they don’t have any Wizard of Oz test, they haven’t proven that people want to do that, they don’t have any data of users currently trying to do that. And that’s a sign of why this, instead of literally anything else that you could be doing. I do find if you have a lot of people landing on a webpage or an app and then not doing anything, then it’s probably copy.
They haven’t even experienced the product, it’s clearly not the product that’s wrong. So how can you change the copy and resonate with the pain point rather than the solution you are offering so that users understand how to fit themselves into the use case? So copy is a big one if I see conversion rates aren’t landing between app launch to some first action. But if there is conversion and they’re just not as frequent, I try to look at what the most painfully long conversion events are. So users who eventually check out or eventually completed the aha moment, what are the user paths, and what is the longest one that seems like it’s the most painful? Are there enough people trying to do that?
And how do we shorten that cycle? So for Kumu things like users wanted to sign up and find their friends on Kumu. And so they were using search frequently, search was underutilized API, it was slow. We sped that up. Conversion rates go from 60% to 90% over the course of a few weeks of just optimizing that and putting more content there. So looking at where are people doing things and then failing, you already know this percent of people would convert if you fixed this, that’s a definite potential win. So we try to layer these definite wins with crazy bets of brand new feature with no data. At least run an experiment if you can. But I always try to layer in these sure wins.
Lenny: When you talk about conversion being good and bad, do you have a rule of thumb or of a mental model of here’s a rough range of this is good and we should not really spend a lot of time on this and this is bad and we should optimize?
Crystal Widjaja: So assuming that the frequency is correct, so you have a weekly frequency, if users are coming back, if it’s a free product, 60%. It has to be at least 60%. If it’s a free product, we go over a week. If it’s a paid product, I usually look at that more as maybe 20 to 30%.
Lenny: And this is retention, people coming back the next week?
Crystal Widjaja: Exactly. Coming back in the second week or month or whenever your frequency ratio is. And this is at scale. So if you are much smaller, your friends and family that better be near close to 80% no matter what, because if you can’t even convince the people who care about you to use the product, it probably isn’t going to solve the job for anyone else.
Lenny: Very handy. Very concrete numbers. And then your point is that when you’re a startup, it’s only going to go down because your early adopters that are more excited and they’ll be more excited by coming back. So you want to start really high.
Crystal Widjaja: Don’t make the same mistake that Netflix and Spotify have made, which I guess is when they’ve launched, they’ve started international expansion and they see this very small percentage of users start to sign up for Spotify or Netflix. There are very few people though in Southeast Asia or internationally that have the types of credit cards that Spotify or Netflix would accept. And so when they launch in these markets and they see a ton of uptick in the first week, they’re like, “This is only going to get better.” When in reality it’s like you just pulled forward everyone who could have possibly subscribed to you, now you’re going to have to work a lot harder to get everyone else.
Lenny: The 60% number. So you’re saying it’s then every week, 60% of the previous week come back, roughly. Is it just a rule of thumb?
Crystal Widjaja: Yeah, exactly.
Lenny: Is that how you think about it versus say cohort retention? Is that just because it’s easier is just a simple rule of thumb?
Crystal Widjaja: Am actually thinking of it as cohorts. So 60% should be your week one, and then it should flatten. I think I usually give teams two to three weeks or frequency periods to see things flatten, but it better flatten around 60% for a free product. That’s actually what we saw at Gojek. Early days it was like 60, 70% retention rates because people were using this product that really solved a huge problem for them. And I think that’s when I knew we were going to be fine. If people keep coming back, the product just needs to work.
Lenny: Wow. So week one, 40% of people drop off week two and beyond basically nobody drops off is what you look for.
Crystal Widjaja: Yeah.
Lenny: Wow. What a high bar. But I like that, because-
Crystal Widjaja: Yeah, well Gojek is a decacorn.
Lenny: Okay. There we go. If you want to be a decacorn, there’s your new benchmark. Amazing. Okay. There’s a bunch of other stuff I want to dig into. One is just data modeling and thinking about growth strategy as a founder. So say a startup is just trying to think about, how do we drive growth, where do we invest, do you have a framework or a process? I know this might be a really big question, but just for founders to think about how their growth works, what their drivers might be, how would a founder approach that problem?
Crystal Widjaja: For sure. So I thought that this was not an obvious process. It wasn’t an explicit process until I worked with Reforge to build my data four PMs program. Got to get that plug there.
Lenny: Go Reforge.
Crystal Widjaja: I basically talked with the Reforge folks about here’s what I would do in all of these scenarios. And they’re like, “Oh, so you mean you’re doing this step one, step two?” And I was like, “Yes, actually. How did you figure that out?” So I don’t really think in-frame works, this is just a logical process to me. But I think what I’ve figured out is, it’s step one, you have constraints. Similar to our sandbox example of everything in the world that’s orange versus everything in a construction site, you have to think about the physics of the current market, the product, the model and the channels that you’re using. So to use Gojek as an example, it would be market of Indonesia.
Here are the consumers in this market, the driver’s side supply side in this market. Here is the product, mobile app. We’re able to connect drivers and consumers. There is a allocation that we create model. We charge per order, channel. We are able to do this through push notifications or in acquiring new users. It might be through Facebook ads, or, and this was a really big insight for us, it’s the real world. There was a physical conception of a driver in a jacket driving around the city who was marketing Gojek for us. And word of mouth actually was primarily driven by, “I saw a driver on the street, so I knew Gojek was here.”
And that actually was a huge driver of all of Gojek’s growth as it expanded to new cities. So step one is, what are the physics? Step two is when you think about loops and growth funnels and the quantitative inputs to each loop, does that fit into these physics or do you have to change four or five different things? So we were very careful about changing too many parameters and making too many bets on too many variables going our way. So we would always change one small thing at a time and make sure that it fit into the model.
Lenny:
Crystal Widjaja: I think it’s both figure out how you’re growing and also the elements that you have at your disposal. What are the levers that you have that maybe you’ve never tried using? When we looked at our model this way, we actually realized we had underutilized the driver’s capacity to drive our growth. Pun definitely intended. So in looking at the model this way, we had thought through what is our goal. We want GoPay to be much bigger than it really is. It’s a E-wallet service, users are able to get access to this digital balance. How do we drive adoption? And so when we looked at the lever of we have a driver, we actually created an incentive model. So we built a very small service that would check when a driver got allocated to a customer, again, the product and the model, we would then check in the database, has this customer ever used our GoPay product before?
Did they have a digital balance? And if the answer was no, we would message the driver immediately, “Hey, this customer hasn’t done a GoPay top up before. If you get them to give you cash and we deposit it into their virtual wallet, we’ll give you extra money.” So using them as the salesperson. You wouldn’t believe how great of a salesperson someone can be when you were literally trapped in a car with them going somewhere. And so you have this captive audience, captive attention, you have someone who has the incentive to cross pay or cross sell someone into GoPay. And customers were able to feel the benefit because the driver was explaining it to them directly. There was no change to the physics, it was a lever usage.
Lenny: What a devious strategy.
Crystal Widjaja: It was huge. It was 60% of acquisition once we released that.
Lenny: Oh my God. So for thinking through your potential levers and physics of your growth, do you think about it bottoms up, here’s all the things that are going on and here’s areas we can invest? Or do you have a menu of options top down of here’s the 10 things it could be, it’s looking like for Gojek it’s these four and let’s focus on that?
Crystal Widjaja: Yeah. I think you always have to start from the fact that we are not wizards. It’s very hard to move the physics of a universe when you are trying these new things. So start with what currently works and currently exists and where you think the biggest constraint is or the best lever is, and then fix that one piece because the entire universe isn’t exploding. The world isn’t changing so dramatically that your physics change. So I think rooted in reality is very important.
Lenny: Got it. Okay. So it’s see what’s working, find the constraints. And then step two is basically what can you do to the product to optimize the funnel/loop to make it go even faster?
Crystal Widjaja: Exactly.
Lenny: Love that. Maybe as another example, if something comes to mind, with Kumu, how do you think of Kumu through this lens?
Crystal Widjaja: Yeah. I’m always very hesitant to talk about Kumu, because there’s so much competition right now and we are on the cusp of some very interesting things. But I think for Kumu, it’s actually very complex because there’s a lot of human emotion that is involved. With Gojek you knew if you got the job done. You made a transaction. With Kumu, how do you know if a consumer made a friend, felt like they had a genuine friendship? So you almost have to create more friction to identify users who really got past that barrier and aren’t explicit with the activity that they did. So we have features that tell us if a user is really searching for this job to be done, if they really want to be part of a community, how do they fill out this?
Do they fill out the form? Do they fill out a questionnaire of many questions? Do they go through this friction just to get access to a community? So we almost create this artificial friction to help differentiate how deeply a user wants something or needs something. And if the user doesn’t fill out that questionnaire, maybe they’re actually looking for something else. They were looking for entertainment. They were looking for content or short form content. And so creating almost like hand razor approaches for a user to say, “I wanted this thing.” We leave a lot of breadcrumbs in the app to be able to identify those paths.
Lenny: Awesome. While we’re on the topic of these two companies, just maybe for inspiration to founders who are thinking of ways to drive growth, what were a couple of the bigger unlocks growth wise for these two companies or even any other company that you’ve worked with that’s interesting?
Crystal Widjaja: Yeah. Definitely, in the early days it was copy. So I think if your product does something that’s not super familiar, you have to tie it to something that is. So I talked about using drivers to sell GoPay. Before that, one thing that we did was to actually take someone’s virtual account number and put it onto a picture of a credit card. You know what a credit card is, that’s familiar to you. A lot of people didn’t know what a digital wallet was. And so when they looked at this like, “Oh, okay. I have this virtual thing that acts like a credit card. It works like my debit account.” Then they understood the concept a lot better.
And we actually saw top ups increase based on us, literally just sending that picture with someone’s virtual account number there. So they could go to an ATM and they would just type in the card number as they would a regular debit account. And they realized that they could top up through that channel. Because that was something that was pretty interesting to us, with just how do we tie the familiarity loop back into the consumer mental model of the product and drive acquisition that way?
Lenny: And that was at Gojek?
Crystal Widjaja: Yeah.
Lenny: Is there anything else maybe, since you don’t want to talk too much about Kumu, any other advisorships or companies, examples of something that ended up working really well to help them accelerate growth?
Crystal Widjaja: Things that have worked really well. So for one of the companies I work with, AB&B, they run a lot of their D2C brands in South America and globally. So one of the features that we were looking at was how do we ensure that subscriptions don’t actually become a canceling point for a user. So in the app you could cancel or you could resume your subscription, but you couldn’t pause it. So when we looked at the cancellation reasons and we saw that their number one reason was, I still have too much fear, we actually decided, well, let’s just add a pause button then.
Because canceling the subscription is a permanent solution to having too much fear. How do you make a temporary solution that solves the actual problem? Adding in a pause button actually helped alleviate a lot of the churn that was becoming very hard to reacquire back. So that was one fix where we looked at the, again, physics of the model. We’re not going to create new changes to the product or create one time buys or reactivation emails. We’ll just solve the problem at that small constraint where everyone drops off.
Lenny: Wait, so can you order beer as subscription? Is that a thing? Is this a consumer product or is this-
Crystal Widjaja: It was a thing, yeah.
Lenny: Cool. Okay. This also reminds me, at Airbnb, this was actually one of the biggest wins, is adding a snooze feature to your listing. Exactly the same thing. Yeah. All right there we go. Awesome. Tip for folks that have churn problems, snooze/pause. I want to shift a little bit to a post that you wrote that maybe is one of your more popular posts you wrote on the Reforge blog called Why Most Analytics Efforts Fail. And I’d love to hear your broad overview of why do most analytics efforts fail and then how do teams avoid this? Maybe what are two to three things they can do?
Crystal Widjaja: Yeah, I’m actually pretty surprised at how much noise that has generated because I guess it came from a place of frustration where I kept telling people like, “You are doing this wrong. Here’s how you should probably be doing it.” But I think it resonated a lot with folks because they recognize all of those symptoms, but they weren’t sure why it was happening. So to say, oh, this is the thing, instrumentation is what’s wrong, I think it’s a very actionable thing. It’s probably one of the most solvable problems out there. It just takes some time and mental model shifts to do it well. So a lot of people look at tracking data as how do I track my OKR? How do I know if I’m going up or down? But they don’t use it to track or identify insights. So I will use the example of using Twitter for “news” when in reality they’re actually using Twitter for entertainment.
Do not treat metric gathering as entertainment. It’s not there for you to be like, “Oh, that’s interesting, how novel,” and then not act on it. So real news is information that changes what you do in the real world. And if you don’t change what you’re doing, what you are doing is just getting entertainment. So let’s use that as the premise. The next step in instrumentation is to look at the fact that measurements do not equate to insights. A measurement would be an observation. It’s a data point in your database. So the example being power users do four times more bookings, is an all observed fact because your transactional database obviously says that is the case, but it’s on an insight because it doesn’t have context. It doesn’t give you information that lets you act on it and better understand the problem.
So another example would be if I see my girlfriend hanging out with a guy I don’t know, that is an observed fact that you see in the real world. Your hypothesis could be that your girlfriend is cheating on you, but the insight, the actual fact might be that she’s not cheating on you, it’s her cousin. And now your insight is, I am paranoid and I need to change my behavior to be less crazy. So the insight will provide value when you have this, why answered? Why is this person doing this thing? Here’s why. And then you are going to act differently. So for our purposes, if we look at a GoFood user will transact and is more likely to use a voucher, that’s a fact, that’s an observation, but it’s not an insight. An insight would be something like GoFood users who are power users are more likely to use a free shipping discount on a high GMV basket versus non-power users.
And that actually tells you how to change your marketing approach. It tells you in what circumstances does someone do this. When it’s a high GMV basket, give power users the ability to get a free discount, but do not do this for non-powered users because they won’t convert any better than they normally would. So, that helps you change your marketing spend. It helps you understand the decision points of power users versus non-power users. The insight is instrumenting properties into an event so that you can segment who is doing what behavior and make some hypotheses on that observation. Test that hypothesis, and then you get some causal representation of whether or not that hypothesis was right.
Lenny: So it sounds like a lot of the root of the issue is setting up the wrong metrics, the wrong… I guess there’s the tracking element of just capturing the right information. And then also just not focusing on insights versus just having a bunch of information.
Crystal Widjaja: Exactly.
Lenny: What are signs that you’re doing this? Say someone’s going to go load up their dashboard and they’re like, “Am I failing or not?” What should they be looking for?
Crystal Widjaja: So I already know if a team is good at instrumentation or not just by looking at the instrumentation spec. The symptom of a bad data tracking approach is you have a ton of rows with a ton of events, but every event has one property or no property being tracked. So an example with Gojek would be when a user lands on the map to select a drop off point, the event would be drop off or map loaded, let’s say. And the properties there should be things like how many drivers do they see on the screen? What is the pickup location? What city is it in? What latitude and longitude is it? Is there surge pricing? What is the current minimum fare? Do they have a voucher code?
All of these characteristics of the experience and the context that can help you look at hey, when a user only sees two drivers on the screen, they’re much less likely to convert than a user who sees five drivers on a screen. Now we can look at in what cities and in what latitude and longitudes do we mostly only see two drivers versus five drivers. Being able to do the second layer approach of the why and not just stop at, “That’s weird. When you have two drivers you are less likely to book.” But then you never ask why. That drives me crazy. Or the inability to even know that there were only two drivers on the screen. You’re missing so much context of the user’s experience that you’re unable to make assumptions about why the user didn’t convert.
Lenny: I love this. Maybe your course is probably going to be the answer, but for folks that want to figure out how to do this taxonomy and events well, how do they go about doing that?
Crystal Widjaja: So I think it’s important to just go through examples. Yes, every product is different, but everyone has the same signup flow for the most part. So look at the signup flow examples that I have in the blog post or in, I believe Amplitude actually has a pretty good long-winded documentation on this, on how to do a event tracking. But it’s really a matter of sitting down and thinking really deeply. If I were to press this button, why would I and why would I not? And am I tracking that in my user properties? So it’s really just sitting down and mapping out the experience.
Lenny: Speaking of Amplitude and other data tools, do you have a default recommended metrics stack for our founders just to start with and maybe a few other things as they evolve?
Crystal Widjaja: That really depends on how early they are. So if they have a single data warehouse with all of their transactional data, usually I say, you can probably get by with Google Data Studio. It’s free usually with whatever you’re using. If not Metabase has a great open source free tool. If you have someone who can write SQL or if you have multiple databases, then Metabase is great. If you need in app mobile device event tracking, I usually recommend CleverTap because Mixpanel has unfortunately failed me a lot. And Amplitude doesn’t have the CRM components that I would need all in one space.
If I am much bigger and I need more analytics juice, maybe Amplitude makes sense on top of this, or something that helps me pipe data into more dashboards and do less ETL for me. Then I would get into Segment. And then once you get into experimentation, obviously I have to shout out to Eppo. I think they’ve really instrumented a lot of the dashboards that I would’ve normally had to do in experimentation projects. So I usually look at something like Eppo to just automate the decision making flow.
Lenny: Awesome. I think we’re both small investors in Eppo, big fans, a little bit of bias, but yeah’s it’s an excellent Airbnb team that built it, so it’s cool. Shifting a bit from metrics and data to just growth teams in general, maybe first question is just, how do you recommend companies set up a growth team in the early days and then over time?
Crystal Widjaja: Yeah. So I can talk about how growth was set up at Gojek as an example, which I think is probably the best practice. So we didn’t really know what growth was at that time, but we knew there were obvious gaps to fill. So because we had grown so quickly, the core product team was still making the core product features. As simple as phone number masking. That wasn’t a thing yet. You had access to your driver’s phone number. It’s probably not a great thing. It’s probably part of the core functionality and we need to fill that gap. At the same time, growth was still necessary because you had all of these users trying to use a product that aren’t quite getting there.
So things like figuring out what SMS provider we should use to send the OTP to this user who is signing up from this telco provider. That was a growth objective that isn’t necessarily core feature work, but was a gap to fill given the onboarding and SMS success delivery rates. Things like telling the driver if this was a brand new customer, because at this point in time, drivers had taken thousands of rides and they assumed every single customer knew how Gojek worked, when maybe they didn’t. And so we knew that the protocol was that a power user would know they would make an order and they would just wait. They would wait somewhere, they would keep an eye out for a driver and then they would get on the motorcycle and go.
But for a brand new user, are you supposed to walk to the driver? Are you supposed to find them? It’s unclear to this brand new, uneducated new user how to use the product. And so first time user experience could have been a terrible one where they went and walked off and then the driver came to the pickup point and they couldn’t find them. So it was all of these small acquisition, adoption and engagement use cases that growth was filling the gap on. And eventually we embedded our growth, I would say product managers at the time, into these teams and they ended up synthesizing what growth was as a full-time role. Eventually becoming PMs who own specific parts of the product stack.
Lenny: So in your experience, and I hear this a lot, is your first growth person shouldn’t just come in and figure out what to work on. You should understand here’s where we need growth help, let’s find somebody to tackle it, versus come help us figure out what to do to drive growth. Is that how you’ve seen it?
Crystal Widjaja: Exactly. I think it’s just setting the bar too high to expect someone to come in and model everything. Again, there are physics in place it’s very hard to move everything. So it’s really about having someone who already has all of this data knows where the biggest gaps are. Doesn’t have to start from scratch and figure this out and then just picks some small space to work on that they know is workable.
Lenny: Do you have strong opinions about growth being integrated? The way that you described where growth PM basically has a cross functional team basically is the PM versus a separate growth team that’s off to the side.
Crystal Widjaja: Yeah. I think it can work as a separate growth team to the side if the company is truly head over heels, tripping on insane product market fit, if there’s insane, product market fit and you are really scrambling to do core feature stacks, then maybe a growth team to come and be clean up is fine. We’re the cleanup crew. We pick up the pieces that were left behind, we connect the dots. You forgot to plug this in, we’ll plug it in for you. But we were a team of lots of stats heavy people. So a lot of my team were statistics graduates. We cared a lot about looking at numbers and odds and probabilities because it really is a numbers game at that scale. You could work on anything and everything would probably do something. But what was the thing that would make the most impact now and unlock us for the future.
Lenny: I was going to ask you folks to look for when they’re hiring an early growth person, is that what you find, just stats, data kind of person?
Crystal Widjaja: You have to have someone who knows how to run the numbers. If you’re looking at ratios of conversion rates, but you don’t realize that this ratio is of a much smaller base size, you’re going to make the wrong decision. So someone who is intuitively good at statistics, they know how to do sampling appropriately. They know what selection bias is. The worst possible thing is to have a growth person who thinks they are doing the right thing and is measuring things wrong and then focusing on the wrong areas.
Lenny: Do you find that it’s often easier or better to hire a young up and coming person or find someone that’s got a bunch of experience for your first growth hire?
Crystal Widjaja: I would hire someone who is willing to take intro to statistics course. And it doesn’t matter if they’ve had the experience to go wild or not. I think it really is, can they focus on the right opportunity rather than the most flashy thing? And I think both profiles can come under that.
Lenny: Got it. And then what do you do in a hiring process for someone like this? What kind of things do you suggest founders look for?
Crystal Widjaja: Yeah. I actually look for that first principle bias. So I’ll give people case studies of here’s what we see, how do you know that this is true? And then I have them set up an experiment design. I want to see that they are sampling randomly. Not that they’re like, “I’m going to build this feature and launch it, and of course it’s going to work.” I want to see that they’re taking a measured deliberate approach to considering why someone might do this or what tools are available. A growth team can go terribly wrong when they just try to onboard a bunch of brand new tools that don’t integrate well and it takes six months to integrate fully, and then they get nothing done for six months. Everything in growth is an opportunity cost of time, trade off with what you could have been doing to the product in that time.
So we biased towards really quick hacky things. Like in the early days of Gojek growth, I think our first real growth experiment, we were actually still the data team at this time, was to connect a quick Python script to the Twilio API that we had access to. And we SMSd a bunch of drivers through a CSV that we uploaded that said like, “Hey, your acceptance rate is really low. You’re not supposed to do that. Please accept all the rides that you are getting.” And that actually increased acceptance rates by 2% across the board. And when we looked deeper into that data, it did even more so for brand new drivers. And so we then worked with the data driver onboarding team so that they could better facilitate the onboarding experience for their drivers.
Lenny: For the interview question that you described, an experiment design question, do you give that as a project where they have time to work on it or is it a live thing?
Crystal Widjaja: Yes. Yeah. I don’t think live works really well for these case studies. I want to see people put in the time and the work to do something to the best of their ability. And of course we ask them like, “Hey, you have five days. We expect you to spend probably four hours on this, so if you don’t have four hours within these five days, let us know.” So we’re pretty careful about giving them the appropriate amount of time to do it at the level of quality that we would’ve expected if they were to work here full-time. So give them those four hours, we want to see do they Google. If they can’t figure it out right now, let’s see them Google it. We’ll ask them what approaches they took, how did they figure this out. And we like to hear people say that they literally had to Google this and read a bunch of white papers. I do that as well.
Lenny: For people trying to design one of these for themselves, do you have a question that you’ve retired that you could share or something that would help somebody design their own prompt?
Crystal Widjaja: Yeah. I can give you a template after this call.
Lenny: Amazing. We’ll include that in the show notes. Easy peasy. Amazing. Okay. A last topic that I wanted to cover is a very cool thing that you were involved in. It’s a nonprofit that you started called Generation Girl. And I think the mission is to help women and young girls get into STEM. So I’d love to hear about this program, how you got into it, what it’s all about, and then also just how listeners can help support what you’re doing.
Crystal Widjaja: Absolutely. Yes. Generation Girl is very near and dear to my heart. So I co-founded this with a couple of amazing women who were also at Gojek, but are now full-time at Generation Girl. So this really stemmed from us repeatedly getting annoying comments about working in STEM. So things like, “You can’t possibly be the engineer on this project. You look like you like makeup and stuff.” And we were like, “Yes, I absolutely love makeup, but I also am badass at writing SWIFT code, so step aside.” So having experienced a lot of the misrepresentation of what an engineer should look like or should like, I think we really look to Legally Blonde, is one of my favorite movies that represents you can take the powers that you have, whether you like engineering or design or data, and you can be whoever you want and still kick ass at it.
So a lot of the women that we support, we’re actually happy if they go into one of our classes and they say, “Actually, I don’t like engineering.” That’s great. That’s agency and empowerment that they got to make that decision for themselves without any cultural biases or social pressure telling them that they should feel this way. And so we offer free classes for girls 12 to 17. We have college classes. We partner with teachers about how to teach STEM topics, especially in areas where they don’t have laptops for every student. How do you teach how to use Figma and things like that? So people can definitely support us and reach out to us. We have a PayPal on our website, take a look.
Lenny: Can you share some of the impact that you’ve seen from this? Are there numbers you can share or anything that you can share around what the organizations have done.
Crystal Widjaja: So we’ve already had several thousand students go through Generation Girl, summer clubs and programs and classes. So we have an event every week. We have a full summer club that’s every single day for two weeks, every summer and every winter. We have partnerships with some of the biggest tech companies in Indonesia, where we partner students with engineers and they work on projects together. And most recently we’re part of the MIT solve program with our new initiative Class. So Class, we’re creating a free to use site for teachers.
So right now we have partnered with a handful of universities in Indonesia, both in rural and city of Jakarta where teachers can now have the knowledge and material to explain newer concepts that maybe they’re less familiar with, because startup world changes rapidly, how you develop changes rapidly. So this is one thing that we are most excited about because every teacher impacts thousands of students a year. And being able to teach the teachers and give them the resources that they need is something that’s really important.
Lenny: That’s incredible. It’s currently just in Southeast Asia, is that right?
Crystal Widjaja: Only in Indonesia, because frankly, this is where everyone needs the most support. Globally STEM is not well received or welcoming at all to women. I think it’s gotten worse over the past few decades. Below 18% of college graduates are women in computer science. So we’re really trying to reach the youngest generation because that’s when you are told or informed that computer science is for specific types of people.
Lenny: It’s really sad to hear that it’s heading in the wrong direction. What do you think is contributing to that?
Crystal Widjaja: I think there is still a lot of this mental model of what a computer scientist is able to do and how much support they’re given. So it’s been shown in studies that at the youngest generation middle school, high school, you are more likely to be given introductory STEM classes as a male than as a female. So women just aren’t targeted for STEM at that younger age. And so when they enter the high school or college classes for computer science, they’re way behind. And that does not feel good. No one likes to be the worst in the class. And so it’s more likely that you’ll drop out. We’ve seen studies at Carnegie Mellon that actually would create introductory computer science classes before the college class starts. And for the women who did join those classes, they actually graduated at similar rates as their male counterparts. So it’s really setting them up for success.
Lenny: If folks want to help. You said that there’s a PayPal page. Is there any other sort of action people can take?
Crystal Widjaja: Yes. Enterprise software. We love to teach iOS development, licensed software. We have hundreds of students a year, so let us know.
Lenny: Awesome. And they can reach you on generationgirl.com?
Crystal Widjaja: Generationgirl.org.
Lenny: Crystal, thank you so much for being here. I’ve taken enough of your time. Two last quick questions. Where can folks find you online if they want to reach out? And then other than the Generation Girl chat we just had, is there any other way folks can be helpful to you?
Crystal Widjaja: Yes. Please find me at crystalwidjaja.com. You can reach out to me and my email is there. Listeners, please do instrumentation correctly. Please don’t track your KPIs. Please track your user journeys and experiences. We’ll have much funner things to talk about if you do that.
Lenny: Amazing PSA. Thank you so much, Crystal.
Crystal Widjaja: Thanks Lenny. This was a blast.
Lenny: Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify or your favorite podcast app. Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.
Glossary
| English | 中文 |
|---|---|
| AB&B | AB&B(公司名,保留原文) |
| adoption | 采纳(指用户对产品功能的接受和使用) |
| aha moment | ”aha moment”(顿悟时刻,保留原文) |
| allocation model | 分配模型 |
| bread and butter | 看家本领 |
| churn | 流失 |
| cohort retention | 同期群留存分析 |
| conglomerate | 综合性企业集团 |
| Crystal Widjaja | Crystal Widjaja(嘉宾,保留原文) |
| D2C | D2C(Direct-to-Consumer,直接面向消费者,保留原文) |
| Darius Contractor | Darius Contractor(产品与增长领域从业者,保留原文) |
| decacorn | decacorn(估值超过百亿美元的初创企业,保留原文) |
| engagement | 参与(指用户与产品的互动活跃度) |
| Facebook Connect | Facebook Connect(保留原文) |
| Generation Girl | Generation Girl(非营利组织,保留原文) |
| GMV | GMV(成交总额,保留原文) |
| GoFood | GoFood(Gojek 旗下外卖服务,保留原文) |
| Gojek | Gojek(公司名,保留原文) |
| GoPay | GoPay(Gojek 旗下电子钱包服务,保留原文) |
| instrumentation | 埋点(指在产品中植入数据采集代码以追踪用户行为) |
| KPIs | KPI(关键绩效指标,保留原文) |
| Kumu | Kumu(公司名,保留原文) |
| leapfrog | 直接跳过(技术发展阶段) |
| Legally Blonde | 《律政俏佳人》(电影名) |
| Lenny | Lenny(主持人,即 Lenny Rachitsky,保留原文) |
| Made to Stick | 《让创意更有黏性》(Chip Heath & Dan Heath 著作) |
| MIT Solve | MIT Solve(麻省理工学院的社会影响力项目,保留原文) |
| OKRs | 目标与关键结果 |
| onboarding flow | 新手引导流程 |
| opportunity cost | 机会成本 |
| OTP | OTP(一次性密码/验证码,保留原文) |
| performance marketing | 效果营销 |
| product market fit | 产品市场契合度 |
| psych model | 心理模型 |
| Reforge | Reforge(增长领域专业培训平台,保留原文) |
| scrappy | 精益(在资源有限条件下灵活高效地推进) |
| selection bias | 选择偏差 |
| super app | 超级应用 |
| SWIFT | SWIFT(苹果编程语言,保留原文) |
| top up | 充值 |
| Typeform | Typeform(在线问卷工具,保留原文) |
| value prop | 价值主张 |
| white papers | 白皮书 |
| Wizard of Oz | ”绿野仙踪”式体验(指用人工模拟自动化功能的 MVP 验证方法) |
Reformatted by reformat_english.py
如何以精益方式招聘、衡量并释放增长 | Crystal Widjaja(Gojek 和 Kumu)
文字记录
早期 Gojek 的疯狂增长
Crystal Widjaja: 我觉得那是一个非常可解的问题。我们最后租了一个体育场,几周之内就招了六万名司机。回想起来,这当然是有风险的。我刚到的时候,公司就在一栋房子里,我意识到自己可能犯了一个巨大的错误。但即便在那个日订单量只有四千单的小规模下,我们已经在快速增长了。
Lenny: Crystal Widjaja 一直在东南亚一些最大的消费者企业中领导产品和增长团队,包括她目前担任首席产品官的 Kumu,以及 Gojek——她在那里组建并领导了增长团队,经历了这家如今已成为东南亚最大超级应用的早期岁月。作为参照,Gojek 每天完成的打车订单超过 Lyft,外卖配送量超过 GrubHub、Uber Eats 和 DoorDash 的总和,它也是印度尼西亚和东南亚排名第一的移动钱包。在我看来,美国初创公司有很多可以向亚洲初创公司学习的地方,而 Crystal 正站在其中一些最成功案例的起点。
这次对话中,我们聊到了 Crystal 在她工作过的公司中所见的最大增长突破点,哪些增长投资通常能产生回报、哪些往往不能;我们深入探讨了增长模型,分享了诸多加速增长的技巧,分析了为什么大多数数据分析工作会失败以及如何避免;还聊到了如何招聘和搭建增长团队,以及 Crystal 创立的一个旨在帮助年轻女性进入 STEM 领域的非营利组织 Generation Girl。Crystal 是一颗真正的明星,希望你能和我一样喜欢这期节目。那么,有请 Crystal Widjaja。
Lenny: Crystal,非常感谢你能来。我在网上读过你不少文章,我们通过邮件和推文交流过很多次,但这是第一次真正坐下来聊,所以我非常期待向你学习,也让大家了解你。
Crystal Widjaja: 我们可能在 Clubhouse、音频论坛或者推特上打过照面。所以很高兴能见到你。
Lenny: 哇,我刚想起来了。确实如此。我记得我们在聊 Reforge 和 Eppo,对吧?
Crystal Widjaja: 对,没错。那段时光真不错。
Lenny: 天哪,Clubhouse 的日子。
Crystal Widjaja: Clubhouse 还风光的那段日子。那段日子也让人学到了什么叫指数衰减。
从投资银行到东南亚科技
Lenny: 哎呀。好吧,也许我们会聊到那个。接下来我们会聊很多消费者增长及相关话题。但在此之前,你的职业路径和地域经历,和我许多其他嘉宾相比都相当不同寻常。为了给大家一些背景,能不能讲讲你的职业历程——从最初做投资银行,到现在担任 Kumu 的首席产品官,还有在新加坡的生活?请跟我们分享你的经历。
Crystal Widjaja: 好的。我的路径确实不太常规。我在湾区圣何塞长大,在我大学毕业那年前后,可以看到 Lyft 这样的公司正在崛起。但说实话,我当时想的就是怎么尽快毕业,因为觉得大学生活很无聊。所以我选了政治学。我不是数学或计算机科学专业的。我甚至不知道咨询顾问是什么,因为我只想赶紧毕业走人。我不知道人们在毕业前就开始找工作了。所以毕业前的最后两周我才在 Craigslist 上搜,因为我觉得”Craigslist 就是大家找工作的方式”。我是第一代美国移民学生,父母完全没办法指导我——比如”你应该看看麦肯锡这家公司”,或者”你面前有哪些职业路径可以选择”。于是我最后找了一份投资银行研究的工作。
我的工作就是打电话给初创公司,分析它们获得风险投资融资或并购顾问服务的潜力。当时我几乎不知道这些词是什么意思。最后我负责管理一个庞大的 Excel 数据库——13万行、60多列。因为我又是一个非常没有耐心的人,我就想:“这个体验太糟糕了,我怎么做一个客户数据库出来?“于是我通过 Google 搜索学会了搭建 MySQL 数据库所需的所有知识,做了一份方案。而投资银行——不出所料——在技术方面并不先进。他们看了我的方案说:“这个 MySQL 是什么东西?是不是特别贵?什么是开源?“所以我离开了那份工作,因为我意识到如果我想进入更技术化的领域,投资银行大概不是合适的地方。于是我把在投行学到的策略,用同样的模式识别方法应用到了东南亚的公司上。
我的家庭原本来自印度尼西亚。我想,我有一个安全网——凭血缘我一定天生就说印尼语吧——所以也许印度尼西亚是一个值得我关注的国家。于是我的策略是:找一家我觉得很有意义、能产生共鸣的公司。我真的直接发了冷邮件联系了一些公司,Gojek 就在名单上。我就是 Google 了”Gojek HR”然后给一个人发了邮件,说”我愿意搬到印度尼西亚,赌我一把”。他们居然真的赌了。我运气极好。五年转眼就过去了。我从零开始组建了数据团队。当你拥有了所有的数据,你就知道系统里有多少欺诈行为。所以后来我又组建了欺诈与风险团队,接手了效果营销,然后就是——好了,现在我们准备好增长了。你现在有了所有这些数据,去搞定增长吧。
Gojek 与 Kumu 的规模
Lenny: 明白了。你对 Gojek 以及那家公司取得的成功,还有你现在工作的 Kumu,都说得很谦虚。那为了给不太熟悉这些公司的听众补充一些背景,能分享一下它们的规模有多大,以及在东南亚有多重要吗?
Crystal Widjaja: 好的。它们的规模确实很大。Gojek 现在叫 GoTo,刚和印度尼西亚最大的电商平台合并了。在整个东南亚,我们大概有 1.7 亿用户。东南亚是有规模的。如果你想在大规模下做产品,你就去东南亚。我们有 20 多种不同的服务,从交通出行到外卖、购物、送药、缴费、电影票,应有尽有。基本上相当于美国所有的初创公司都塞进了一个应用里,同时服务于同一批用户群在开发。所以一切都非常层层叠加,因为你可以填补市场上各种机会缺口,而单个应用可能没法那么可持续。
Gojek 的版图横跨印度尼西亚、新加坡、泰国、越南。而 Kumu 则更像一个社交领域的超级应用。Gojek 偏交易型——“我有个任务要完成,我想付钱让人把东西送过来”。而 Kumu 更像是”我想把 Clubhouse、Zoom、Google Hangouts 全部聚在一个应用里”。所以我们覆盖了社交信息流、音频、视频、多席位等功能。Kumu 上有大量不同的使用场景。Kumu 主要在菲律宾运营,但在多个国家的移动应用畅销榜上排名前十。
为什么选择加入 Gojek
Lenny: 那么 Kumu 你加入的时候他们已经做得不错了。但 Gojek,如你所说,你加入得非常早。你在那家公司里看到了什么,让你决定加入这样一家高风险的早期公司?对于那些可能正在考虑加入初创公司的人,你有什么经验总结,关于应该看哪些东西?
Crystal Widjaja: 说实话,可能很大程度是运气。但那个年纪我也意识到自己没什么可失去的。对于 Gojek,我觉得它是对的公司,因为我能非常清楚地理解它的价值主张。印度尼西亚的交通状况很疯狂。20 公里的路要开两个小时。所以你当然想坐摩托出租车来避开堵车。你当然不想出门买饭然后再花两个小时回来。所以我用那种巴菲特式的方法——我知道这个产品是说得通的。市场也说得通。司机本来就存在,但要把他们和消费者连接起来非常困难。
砍价很痛苦。印度尼西亚各地分布着大量的餐厅。所以价值主张和市场都说得通,通过移动应用来做这件事的渠道在当时看起来不太说得通,因为大多数司机没有手机应用,但我觉得这是一个非常可解决的问题。后来我们租了一个体育场,几周之内就招募了 6 万名司机。回首看来,我刚到的时候确实有风险,公司还在一栋房子里运营,我心想我大概犯了一个巨大的错误。但我们的增长已经非常快了,即便在那个小规模下每天也有 4000 单。
为什么美国没有出现超级应用
Lenny: 我想花很多时间聊聊你在这些公司推动增长中学到的东西。不过先快速问一个问题。Gojek 是一个超级应用,在一个应用里做很多事情。你对为什么美国没有出现超级应用有什么见解吗?
Crystal Widjaja: 我觉得东南亚对综合性企业集团的情感认知很不一样。我们从小到大,一个特定的企业集团不仅拥有你去逛的商场,还拥有你住的公寓楼,你上的学校。所以它们整合得非常好,人们对企业集团有一种信任感。而在美国,人们已经开始回避了——Google 是不是对我了解太多了?我觉得第二个方面是,在亚洲,我们直接跳过了个人电脑时代进入了移动互联网时代。所以每个人都有手机,但整个家庭里可能连一台电脑都没有。那么当你的手机满了的时候,你是删掉你孩子的照片,还是删掉这个应用?你大概会删掉应用。所以任何想要真正存活下来的应用,就必须成为超级应用概念的一部分。
Lenny: 哇,我从来没从这个角度想过。因为你手机空间不多,所以你想要一个应用做很多事情。
Crystal Widjaja: 对。所以存在一个决策因素,而这在美国并不那么明显,因为那边的云存储和设备容量都更大一些。
Lenny: 有意思。所以在美国,你可以有不同的应用各做各的……基本上超级应用不需要在每件事上都是最好的,关键在于它什么都做得足够好。哇,太有意思了。
Crystal Widjaja: 只需要把事办成就行。
增长中的疯狂做法
Lenny: 太精彩了。好的,这个话题超级有意思。好,我们来转到增长和你一路上学到的东西。你刚才提到,我记得 Gojek,你说在很短时间内雇佣了数以万计的司机。有没有一些亚洲初创公司做的、你认为美国公司也应该做、可以在增长方面学到的事情?
Crystal Widjaja: 有的,我们做了很多疯狂的事。如果有人在美国告诉你,他们打算租一个体育场,预先装载一批移动设备,宣传让司机大规模来这里参加招聘会,给他们发一部手机然后让他们上路——有些人会说:“不行,这太疯狂了。我们会不会惹上麻烦。“在某种程度上,也许确实如此。也许确实存在一些限制,但这种做稍微疯狂的事、但能验证一个观点的做法,做那些不可扩展的事情,尤其是我认为这确实是我们在 Gojek 的看家本领。我们极其精益。
比如我们想测试一个订阅功能,也就是几周前刚在新加坡上线的那个。我们最终的做法是:“我们有一个可以在后端分发代金券的代金券系统。我们显然知道司机的手机号。要不我们就把他们加到一个 WhatsApp 群里?” 我们随机拉 100 个司机进一个 WhatsApp 群。告诉他们:“每次你载客的时候,试着向客户推销。你是唯一能销售订阅套餐的司机。让客户给你 10 美元。客户说好的时候你就发短信告诉我们。会有人全天候守在这部手机旁边。”
“我们会在后端查到你当时载的那位客户,在后台给他们发代金券,然后从你的余额里扣除 10 美元。” 这招管用。这真的是一种”绿野仙踪”式的体验——我们不需要开发任何东西。我跟一群实习生协调配合,就验证了订阅服务中一些价值主张和预期转化率。当我们想做一个新的新手引导页面时,但发现我们有大量的工程工作要做,我们就截了一张当前页面的截图,让设计师把我们想要的新手引导流程直接叠在截图上面,模拟出如果开发出来它会是什么样子。
用可扩展的方式快速验证
我们就以应用内消息的形式发了出去。到后来,我觉得直觉上我们开始找到一些可以规模化复制的方式了。我们心里清楚,自己一直在通过 Typeform 问卷之类的方式投放大量”假功能”。像性格测试这类东西,用 Typeform 就能轻松实现。后来我们意识到,如果在应用里嵌入网页视图,让后端的网页部署变得更便捷,就不必等移动端发版才能测试那些其实可以用网页实现的新功能。所以核心思路就是:我们想创造什么样的用户体验?怎么以最快的速度把它呈现出来?先动手试再说。
Lenny: 回到体育场的例子。我知道你说过雇了”一个体育场的人”,我之前没意识到你们是真的租了一个体育场——
Crystal Widjaja: 是真的租了一个体育场,像一个足球场那么大,如果没记错的话可能有几个足球场那么大。排着长队,一箱一箱的手机和 SIM 卡。所以其实就是靠大量艰苦的体力活来达到那个规模。
Lenny: 哇。我知道你也做了很多顾问工作。你会建议初创公司更加精益、去做那些不可规模化的事吗?我能想象美国的创业文化会有些不同。
Crystal Widjaja: 有一件事比什么都强——如果你能拿到用户行为的真实数据,那就是你能获得的最好数据。所以如果你没有一个经过验证的假设,如果你想不到怎么跑一个实验,那坦白说这个想法就没什么用。也许它在逻辑上对市场成立、对商业模式成立,但消费者的心理可能是很奇怪的。不是每个人都是理性决策者。所以测试真实的体验,观察人们的反应,那才是最好的数据。
小样本也能跑实验
Lenny: 顺着这条线再拉一下——对初创公司来说,实验往往很难,因为数据和用户量都不够。你觉得初创公司应该怎么看待这个问题?在非常非常早期的时候能跑实验吗?
Crystal Widjaja: 应该跑。即使样本量只有 30,你拿回来的数据大体上不会变,变的是精度。从数学上讲,你会看到同样水平的趋势,只是你对这些趋势理解的精度会随着数据量增加而变得更深。但你从底层获取的信息,在更大规模上不会有多大差异。30 个数据点比什么都没有强得多,当然有 100 个更好。但比零更好的,肯定是 30。
Lenny: 太有意思了,真的很反常识。30 个人就跑实验。我喜欢这个。
Crystal Widjaja: 你必须这样做。在那个规模下,每个想法的成本都极低。你用 30 个人可以做远比 100 人时更极致的不可规模化实验。
Lenny: 再追问一下,当你用 30 个人跑实验的时候,你看什么?你希望其中 20 个人做了某件事吗?也就是这群人中很大比例的人做了某件事?
寻找转化发生的前一步
Crystal Widjaja: 所有人都想直接看留存。他们想看到用户在做这件事,想从零直接跳到一百,但他们没意识到,用户的决策是基于一系列前置事件做出的。那么在用户做出那个决策的前一步是什么?他们需要做什么,哪些条件必须满足?所以我们一直在寻找的是:这个用户之所以转化,有没有某个具体的原因?以 GoFood 为例,问题可能是:用户什么时候会尝试一家新商家,还是他们目前点的外卖只是自己已经信任和熟悉的那些?如果用户需要对商家建立信任才愿意下单,那我们怎么建立这种信任?我们的做法其实是走了条捷径——接入用户的 Facebook Connect 登录。
我们已经有权限查看他们在 Facebook 上连接了哪些好友。我们真的去看了他们的朋友买过什么食物,然后把这些作为一个数据集,告诉用户:“嘿,这是 Lenny 买过并喜欢的食物,也许你也会喜欢。“这就是我们建立信任因素的一种方式。我们确实发现,当我们告诉用户”你的这位好友从这家商家买过东西”时,他们从一家全新餐厅下单的概率是没有该功能用户的两倍。这拉高了 GMV,最终帮你达到想要的转化率,但它解决的其实是一个不同的问题。不是”怎么转化”,而是”怎么解决信任问题”。怎么打破阻碍决策过程的壁垒,让用户到达那个”顿悟时刻”——而我们的做法是修复更前面的那个前置条件,也就是信任。
Lenny: 这是一个你的通用经验法则吗?不要把留存作为目标。我记得你在某个地方写过这个观点。这是你大致的经验法则吗?
Crystal Widjaja: 我觉得很多人以为我的意思是”留存没用,完全不用管它”。但实际上我想说的是,当你思考留存的时候,它本身不够具体。我有一个思维模型来自《让创意更有黏性》这本书。他们会跟你说:“Lenny,想想世界上所有橙色的东西。“你就想:“橘子。还有呢?“然后如果你换个框架,加一个限定条件——想想建筑工地上所有橙色的东西——你就会真正开始抓住具体的概念,并且能在现实中真正对它们采取行动。
Lenny: 明白了。说到留存,你发现产品和公司在提升留存方面最成功的地方通常在哪里?
Crystal Widjaja: 通常就在转化发生前的那一步。如果他们不确定用户为什么打开了应用,或者不确定用户为什么到了这个结账页面,那问题往往是某些文案,或者用户路径在某处失效了。我希望创始人能思考一下 Darius Contractor 经常谈到的用户心理模型。在用户旅程中你需要积累一些势能,才能推动用户跨过那些非常痛苦的环节,比如输入信用卡号——那可费劲了。你怎么降低这种摩擦?把正确的步骤有效地排列组合,甚至只是调换一下页面的顺序,其实就能产生很大效果。
增长投资中的有效做法与常见误区
Lenny: 再往深处聊聊。你在你工作过的公司、你顾问过的公司——我注意到你还在好几家公司做董事——你觉得在增长投资方面,哪些时间的使用特别值得、效果往往很好?第二个问题,哪些事情你觉得很少能成功,或者大家投入了大量时间但最后对增长其实没什么用?
Crystal Widjaja: 我经常看到很多创始人在抓救命稻草。比如会有一个全新的功能,做的和用户目前在应用里做的事情不一样,大家觉得”这个东西会让一切好起来”。但他们没有任何”绿野仙踪”式测试,没有证明用户想做这件事,也没有用户当前正在尝试做这件事的任何数据。这就是一个信号:为什么偏偏要做这个,而不是其他任何你能做的事?我确实发现,如果很多人到达了网页或应用但什么都没做,那问题多半出在文案上。
他们甚至还没有体验过产品,那显然不是产品本身的问题。所以你要做的,是调整文案,让它与用户的痛点产生共鸣,而不是只展示你提供的解决方案,这样用户才能理解自己如何融入这个使用场景。所以,如果在应用启动到第一次操作之间转化率不理想,文案往往是一个关键因素。但如果转化本身是有的,只是频率不够高,我就会去看那些最漫长、最痛苦的转化事件。也就是那些最终完成购买或最终到达”aha moment”的用户,他们的路径是什么样的,哪条路径最长、看起来最痛苦?有足够多的人在尝试走这条路吗?
以及如何缩短这个周期?以 Kumu 为例,用户想要注册后在平台上找到自己的朋友,所以他们频繁使用搜索功能。但搜索这个 API 本来就利用率不足,而且速度慢。我们对此做了加速优化,并增加了更多内容。仅仅几周的优化,转化率就从 60% 提升到了 90%。所以关注用户在哪里做事却失败了——你已经知道如果修好这个问题,会有百分之多少的人转化——这就是一个确定可以拿下的胜利。我们会把这些确定的胜利和那些没有数据支撑的全新功能的疯狂赌注搭配起来。至少如果能跑实验的话,先跑一个实验。但我总是会把这些确定性收益叠加上去。
留存率的经验基准
Lenny: 你谈到转化率的好坏时,有没有一个经验法则或者心理模型,比如某个大致范围——这个算好的,我们不用在上面花太多时间;这个算差的,我们需要优化?
Crystal Widjaja: 假设使用频率是正确的,也就是说你有周频率,用户会回来——如果是免费产品,60%。至少要达到 60%。如果是免费产品,我们按周来看。如果是付费产品,我通常看的大概是 20% 到 30%。
Lenny: 这说的是留存,也就是第二周还在回来的人?
Crystal Widjaja: 没错。第二周或第二个月回来,或者你的频率周期对应的那个时间点。而且这是在规模化之后的数字。如果你规模还很小,你的朋友和家人不管怎样最好接近 80%,因为如果你连在乎你的人都说服不了来使用这个产品,那它大概也解决不了其他任何人的问题。
Lenny: 非常实用,非常具体的数字。而且你的观点是,当你还是初创公司时,这个数字只会往下走,因为早期采用者更兴奋、更愿意回来。所以你一开始就得非常高。
Crystal Widjaja: 不要犯 Netflix 和 Spotify 犯过的错误。他们上线的时候开始国际扩张,看到很小比例的用户注册了 Spotify 或 Netflix。但在东南亚或国际市场上,能被 Spotify 或 Netflix 接受的那种信用卡持有者本来就非常少。所以当他们在这些市场上线、看到第一周有大量增长时,他们会说”这只会越来越好”。但现实是,你只是把所有有可能订阅你服务的人提前拉过来了,接下来你得花大得多的力气去获取其他人。
Lenny: 60% 这个数字,你是说每周,上一周用户的 60% 大概会回来。这只是一个经验法则吗?
Crystal Widjaja: 对,没错。
Lenny: 你是这么想的,那相对于同期群留存分析(cohort retention)呢?只是因为它更简单、就是一个简单的经验法则吗?
Crystal Widjaja: 我其实就是在用同期群的思路。60% 应该是你第一周的数据,然后它应该趋于平缓。我通常给团队两到三个星期或频率周期来看曲线趋于平缓,但对于免费产品,它最好平缓在 60% 左右。我们在 Gojek 看到的就是这样的。早期就是 60%、70% 的留存率,因为人们在使用一个真正为他们解决了重大问题的产品。我想就是那时候我知道我们会没事的。如果用户不断回来,产品只需要能正常工作就行。
Lenny: 哇。所以第一周 40% 的人流失,第二周及之后基本上不再流失,这就是你要找的形态。
Crystal Widjaja: 对。
Lenny: 哇。这个标准真高。但我喜欢这一点,因为——
Crystal Widjaja: 是啊,毕竟 Gojek 是一家 decacorn(百亿美元级别的独角兽)。
Lenny: 好,这就对了。如果你想成为 decacorn,这就是你的新基准。太棒了。好,我还有一堆其他内容想深入聊。其中一个就是数据建模,以及作为创始人如何思考增长策略。假设一家初创公司正在思考:我们怎么驱动增长、在哪里投入——你有没有一个框架或流程?我知道这可能是一个很大的问题,但就是帮助创始人思考增长如何运作、驱动因素可能是什么,一个创始人应该怎么切入这个问题?
增长策略的建模框架
Crystal Widjaja: 当然。我以前觉得这不是一个显而易见的流程,也不是一个明确的流程,直到我和 Reforge 合作开发我的”数据型产品经理”课程。得给这个打个广告。
Lenny: Reforge 加油。
Crystal Widjaja: 我基本上和 Reforge 的人聊了在各种场景下我会怎么做。然后他们说,“哦,所以你的意思是你在做第一步、第二步?“我说,“对,就是这样。你们怎么总结出来的?“所以我不太用框架来思考,这对我来说就是一个合乎逻辑的流程。但我认为我梳理出来的是——第一步,你有一些约束条件。就像我们之前那个沙盘的例子,把世界上所有橙色的东西和建筑工地上所有东西做对比。你必须考虑当前市场、产品、模型和渠道的”物理定律”。以 Gojek 为例,市场就是印度尼西亚。
这里的消费者是什么样的,司机端供给是什么样的,产品是什么——移动应用,能够连接司机和消费者。有一个我们创建的分配模型,我们按订单收费。渠道方面,我们通过推送通知来触达用户,或者获取新用户时可能通过 Facebook 广告。还有一个对我们来说非常重要的洞察——那就是真实世界。有一个穿着夹克在城市里骑行的司机的物理存在,他在替我们做 Gojek 的市场推广。而口碑传播的主要驱动力其实是”我在街上看到了一个司机,所以我知道 Gojek 来了”。这实际上是 Gojek 扩展到新城市时所有增长的最大驱动力。所以第一步是,物理定律是什么?第二步是,当你思考增长闭环和增长漏斗以及每个闭环的量化输入时,这些是否符合这些物理定律,还是你需要同时改变四五个不同的东西?我们在同时改变太多参数、对太多变量下注方面非常谨慎。所以我们总是每次只改变一个小东西,确保它能融入整个模型。
GoPay 的增长杠杆:让司机成为推销员
Crystal Widjaja: 第一步,总结一下,就是搞清楚你是怎么增长的。以 Gojek 为例,部分原因是真实世界的存在——人们就是在街上看到 Gojek 骑手骑来骑去。
我觉得既要搞清楚你是怎么增长的,也要搞清楚你手头有哪些可用的要素。你有哪些杠杆可能从来没有尝试过?当我们这样审视我们的模型时,我们实际上意识到我们低估了驱动增长中司机的能力。这个双关语绝对是故意的。用这种方式审视模型后,我们理清了目标是什么——我们希望 GoPay 的规模远比现在大得多。它是一个电子钱包服务,用户可以获得数字余额。那我们怎么推动采用率呢?当我们审视”我们拥有司机”这个杠杆时,我们实际上创建了一个激励模型。我们搭建了一个非常小的服务,当司机被分配给一个客户时——再次强调,这就是产品和模型——我们会去数据库查一下,这个客户以前有没有用过我们的 GoPay 产品?
他们有没有数字余额?如果答案是否定的,我们会立刻给司机发消息:“嘿,这个客户从来没有做过 GoPay 充值。如果你让他们把现金给你,我们把它存入他们的虚拟钱包,我们就额外给你一笔钱。“所以我们是把司机当成了推销员。你不会相信,当一个人真的被困在车里和你一起去某个地方时,他能成为多么出色的推销员。所以你拥有了一个被锁定的受众、被锁定的注意力,拥有了一个有动机去交叉推销 GoPay 的人。而客户能切实感受到好处,因为司机在直接向他们解释。物理定律没有改变,这只是对杠杆的运用。
Lenny: 真是个狡猾的策略。
Crystal Widjaja: 效果非常惊人。我们上线这个之后,它占了新客获取的 60%。
Lenny: 天哪。那么在梳理你潜在的增长杠杆和增长物理定律时,你是自下而上地思考——这里正在发生所有这些事情,这里是我们可以投资的领域?还是自上而下地有一份选项清单——这里有 10 种可能,看起来对 Gojek 来说是这 4 种,我们就专注这些?
Crystal Widjaja: 对。我觉得你始终必须从一个事实出发——我们不是巫师。当你尝试这些新东西时,要改变一个宇宙的物理定律是非常困难的。所以从当前有效的东西、当前存在的东西出发,找到你认为最大的约束或最好的杠杆在哪里,然后去修复那一小块,因为整个宇宙并没有在爆炸。世界并没有剧烈到让你的物理定律都变了。所以我认为扎根于现实非常重要。
Lenny: 明白了。好的。所以就是看什么在起作用,找到约束。然后第二步基本上就是你能对产品做什么来优化漏斗/闭环,让它转得更快?
Crystal Widjaja: 完全正确。
Kumu 的增长物理定律
Lenny: 很好。也许再举一个例子,如果想到什么的话——以 Kumu 为例,你怎么透过这个视角来看 Kumu?
Crystal Widjaja: 嗯。我一直对谈论 Kumu 很谨慎,因为现在竞争非常激烈,而且我们正处于一些非常有意思的节点上。不过我觉得对 Kumu 来说,情况其实非常复杂,因为其中涉及大量人类情感。在 Gojek 你知道任务是否完成了一一你完成了一笔交易。但在 Kumu,你怎么知道一个用户交到了朋友、感觉到了真正的友谊?所以你几乎必须创造更多摩擦,来识别那些真正跨越了那道门槛的用户,而他们并不会明确表达自己做了什么活动。所以我们有一些功能能告诉我们一个用户是否真的在寻找这个待完成的任务,他们是否真的想成为社区的一部分——他们怎么填写这个?
他们会填完那个表格吗?会填完一份有很多问题的问卷吗?他们会经历这些摩擦仅仅是为了获得一个社区的准入吗?所以我们几乎是在创造一种人为的摩擦,来区分一个用户有多想要、多需要某个东西。如果用户没有填完那份问卷,也许他们其实是在找别的东西——他们是在找娱乐,找内容或短视频内容。所以几乎像创造”举手示意”式的方法,让用户表达”我想要这个东西”。我们在 App 里留下了很多面包屑,来识别这些路径。
早期增长的关键解锁
Lenny: 很棒。既然我们在聊这两家公司,也许可以给正在想办法驱动增长的创始人一些灵感——这两家公司中有哪些比较大的增长解锁?或者你合作过的其他任何公司有什么有趣的例子?
Crystal Widjaja: 有的。在早期,绝对是从文案入手。我觉得如果你的产品做的是一些大家不太熟悉的事情,你必须把它和人们熟悉的东西关联起来。我刚才讲了利用司机来推销 GoPay,在那之前,我们做的一件事是把某人的虚拟账号号码放到一张信用卡的图片上。你知道信用卡是什么,那对你来说是熟悉的。很多人不知道数字钱包是什么。所以当他们看到这个时会想,“哦,好的,我有这个虚拟的东西,它用起来像信用卡,运作方式像我的借记卡账户。“然后他们就更好地理解了这个概念。
我们确实看到充值量增加了——原因仅仅是我们发了一张把虚拟账号号码印在上面的图片。这样他们可以去 ATM 机,像输入普通借记卡账号一样输入那张卡号。然后他们意识到可以通过那个渠道充值。这让我们觉得很有意思——如何将熟悉度闭环连接回消费者对产品的心理模型中,并以此驱动获客?
Lenny: 这是在 Gojek 的?
Crystal Widjaja: 对。
Lenny: 既然你不想过多谈论 Kumu,有没有其他顾问项目或公司的例子——有什么最终效果非常好、帮助他们加速增长的?
Crystal Widjaja: 有一些效果很好的做法。比如我合作的一家公司 AB&B,他们在南美和全球运营大量 D2C 品牌。我们当时关注的一个功能是,如何确保订阅不会真正成为用户流失的节点。在 App 里你可以取消或者恢复订阅,但不能暂停。当我们查看取消原因时,发现排名第一的原因是”我还是太害怕了”,于是我们决定——那就加一个暂停按钮吧。
因为取消订阅是对”太害怕”这个问题的永久性解决方案。你怎么提供一个临时性的解决方案来解决真正的问题?加入暂停按钮实际上帮助缓解了大量流失,而这些流失用户之前很难重新获取。这就是一个修复案例——我们再次审视了模型的物理定律。我们不会去创建新的产品变更、做一次性购买活动或重新激活邮件,只是在那个小小的约束点——也就是所有人流失的地方——解决问题。
啤酒订阅与”暂停”功能的妙用
Lenny: 等等,所以啤酒可以按订阅下单吗?这真的有?这是消费级产品还是——
Crystal Widjaja: 是真的有过,对。
Lenny: 酷。好的。这也让我想到,在 Airbnb,实际上最大的胜利之一就是在房源列表中加了一个”小睡”功能。完全一样的事情。对,就是这个。太棒了。给那些有流失问题的朋友一个建议——加暂停/小睡按钮。我想稍微换个话题,聊聊你写的一篇文章。也许是你在 Reforge 博客上最受欢迎的帖子之一,叫《为什么大多数分析工作会失败》。我很想听你概括讲讲,为什么大多数分析工作会失败?团队又该如何避免?能举两三件具体可以做的事吗?
为什么大多数分析工作会失败
Crystal Widjaja: 好的,其实那篇文章引发的反响让我挺意外的。因为它最初是出于一种沮丧——我一直在跟人们说”你们这样做是错的,大概应该这样做”。但我想它之所以引起了很多人共鸣,是因为他们认出了所有那些症状,却不确定为什么会出现。所以说”哦,原来问题出在埋点(instrumentation)上”,这是一个非常可操作的诊断。这大概是最可解决的问题之一了,只是需要花些时间、转变思维模型才能做好。
很多人看待追踪数据的方式是:我怎么追踪我的目标与关键结果?我怎么知道指标是往上走还是往下走?但他们不会用数据来追踪或发现洞察。我用一个例子来说明——就像有些人用 Twitter 看”新闻”,但实际上他们是在用 Twitter 来娱乐。
不要把指标采集当成娱乐。它不是让你觉得”哦,真有意思,好新鲜”,然后什么都不做的。真正的”新闻”是那些会改变你在现实世界中行动的信息。如果你没有因此改变行动,那你做的就只是在获取娱乐。
基于这个前提,埋点的下一步是要认识到:测量不等于洞察。测量是一个观察结果,是你数据库中的一个数据点。举个例子,“重度用户完成的预订量是普通用户的四倍”,这是一个观察到的事实,因为你的交易数据库显然证明了这一点。但这不是一个洞察,因为它没有上下文,不能给你可以据此行动的信息来更好地理解问题。
再举个例子,如果我看到我女朋友和一个我不认识的男人在一起,这是你在现实中观察到的一个事实。你的假设可能是女朋友在出轨,但真正的洞察、真正的事实可能是她没有出轨,那个人是她的表哥。于是你的洞察变成了:我太偏执了,我需要改变自己的行为,别那么神经质。洞察之所以有价值,是因为它回答了”为什么”——这个人为什么做这件事?原因是这样的。然后你会据此改变行动。
回到我们的语境,如果我们看到”GoFood 用户更可能使用代金券”,这是一个事实、一个观察结果,但不是洞察。洞察应该是类似这样的:GoFood 的重度用户在 GMV 较高的购物篮中,比非重度用户更可能使用免运费折扣。这才真正告诉你如何改变营销策略。它告诉你,在什么情况下某人会做这件事——当购物篮 GMV 高时,给重度用户提供免运费折扣,但不要对非重度用户这样做,因为他们不会比平时转化得更好。这能帮你优化营销支出,帮你理解重度用户和非重度用户的决策差异。洞察的本质是:在事件上埋点足够多的属性,这样你才能细分谁在做什么行为,并基于这个观察提出假设,然后验证假设,最终得到该假设是否正确的因果表征。
如何判断你的数据追踪是否出了问题
Lenny: 所以听起来,问题的根源很大程度上在于设置了错误的指标,错误的……我想还有一个是追踪层面的问题,就是没有捕获到正确的信息。还有就是只关注数据本身而不是洞察,只是一堆信息堆在那里。
Crystal Widjaja: 完全正确。
Lenny: 有什么迹象表明你在犯这些错误?比如有人打开自己的数据看板,心想”我到底是做对了还是做错了?“他们应该看什么?
Crystal Widjaja: 我只要看一眼埋点规格说明,就已经能判断一个团队擅不擅长做埋点了。糟糕的数据追踪方式有一个典型症状:你有一大堆事件,行数很多,但每个事件只有一个属性,或者根本没有属性。以 Gojek 为例,当用户在地图上选择下车点时,事件可能是 drop off 或 map loaded。而属性应该包括:用户在屏幕上看到了多少个司机?上车地点在哪里?在哪个城市?经纬度是多少?有没有动态加价?当前最低票价是多少?用户有没有代金券?
所有这些体验特征和上下文信息,才能帮你分析——嘿,当用户只看到两个司机时,他们的转化率比看到五个司机的用户低得多。然后我们可以进一步看,在哪些城市、哪些经纬度位置,大多只能看到两个司机而不是五个。这样才能做到第二层分析——问”为什么”,而不是停在”真奇怪,只有两个司机时下单概率更低”就完了,却从不追问原因。这种不追问的做法让我抓狂。或者更糟——你甚至不知道当时屏幕上只有两个司机。你缺失了用户体验的大量上下文,根本无法对用户为什么没有转化做出合理的假设。
事件埋点与分类法
Lenny: 我很喜欢这些。也许你的课程就是答案,但对于那些想搞清楚怎么做事件分类和埋点的人,具体应该怎么入手?
Crystal Widjaja: 我觉得关键是多看实例。确实,每个产品都不一样,但大多数人的注册流程基本相同。所以可以看看我在博文里的注册流程示例,或者我记得 Amplitude 实际上有一份相当详尽的文档,专门讲怎么做事件追踪。但归根结底,这件事其实就是坐下来深入思考:如果我按下这个按钮,我为什么会按?我为什么不会按?我在用户属性里有没有追踪到这些?本质上就是坐下来,把用户体验完整地梳理出来。
推荐的数据工具栈
Lenny: 说到 Amplitude 和其他数据工具,你对创始人有没有一个默认推荐的指标工具栈,先从最基础的开始,然后随着发展再逐步补充?
Crystal Widjaja: 这很大程度上取决于他们处于多早期的阶段。如果他们只有一个数据仓库,里面放着所有交易数据,我通常会说,用 Google Data Studio 就够了。它基本是免费的,搭配你现有工具就能用。如果没有的话,Metabase 有一个很好的开源免费工具。如果你有能写 SQL 的人,或者你有多个数据库,那 Metabase 很合适。如果你需要 App 内移动端事件追踪,我通常推荐 CleverTap,因为 Mixpanel 可惜让我失望了很多次。而 Amplitude 又没有我需要的那种一体化 CRM 组件。
如果我的规模已经大得多,需要更强的分析能力,那在这个基础上加 Amplitude 可能是合理的,或者用一些能把数据输送到更多看板、减少 ETL 工作量的工具。这时我会引入 Segment。而一旦进入实验阶段,当然要提一下 Eppo。他们确实把很多我以前在实验项目中需要手动搭建的看板都自动化了。所以我通常会考虑用 Eppo 这样的工具来把决策流程自动化。
增长团队的搭建
Lenny: 太好了。我想我们在 Eppo 都有小额投资,都是忠实用户,多少有点偏心,但确实是之前 Airbnb 的团队做的,很酷。把话题从指标和数据稍微转向增长团队这个更一般性的话题,也许第一个问题是,你建议公司在早期以及后续发展中如何搭建增长团队?
Crystal Widjaja: 好的,我可以以 Gojek 增长团队的搭建为例来说明,我觉得这可能是比较最佳实践的做法。当时我们其实不太清楚增长到底是什么,但我们知道有明显需要填补的空缺。因为我们的增长速度太快了,核心产品团队还在做核心产品功能。比如简单的电话号码遮罩,当时还没有这个功能。你可以直接看到司机的手机号,这大概不是什么好事,这应该属于核心功能的一部分,我们需要填补这个缺口。与此同时,增长工作仍然必不可少,因为有大量用户在尝试使用产品,却无法顺利完成。
比如要搞清楚应该用哪个短信服务商来给从某个运营商注册的用户发送 OTP(一次性验证码)。这是一个增长目标,不算是核心功能开发,但鉴于新手引导流程和短信送达率的考虑,这是一个需要填补的缺口。再比如告知司机这位是不是全新客户——因为到那个时候,司机已经跑了几千单,他们默认每位乘客都知道 Gojek 是怎么运作的,但事实可能并非如此。我们知道熟练用户的行为模式是:下单,然后就在原地等着,留意司机的位置,然后上摩托车出发。
但对于一个全新用户来说,你应该是走过去找司机吗?还是该让司机来找你?对于这个完全不了解产品的新用户来说,怎么使用产品根本不清楚。所以首次使用体验可能会非常糟糕——用户走开了,司机到了上车点却找不到人。增长团队要填补的就是所有这些小的获客、 adoption(采纳)和 engagement(参与)方面的用例。最终我们将增长产品经理——当时的产品经理——嵌入到各个团队中,他们逐渐整合出增长作为全职角色的定义,最终成为负责产品技术栈中特定部分的 PM。
Lenny: 根据你的经验——这一点我也经常听到——你的第一个增长人员不应该只是进来然后自己想该做什么。而是你先想清楚哪里需要增长方面的帮助,再去找人来解决,而不是让人来帮你弄清楚该做什么来驱动增长。这是你的观察吗?
Crystal Widjaja: 完全正确。我觉得期望一个人进来然后把所有东西都建模好,标准设得太高了。说到底,有一些固有的”物理规律”在那里,要同时推动所有东西是非常困难的。所以关键是要有一个已经掌握了所有这些数据、知道最大缺口在哪里的人。他们不需要从零开始摸索,只需要挑选一个他们知道可操作的小领域来推进。
Lenny: 对于增长是嵌入式这种模式,你有很强的倾向吗?就是你刚才描述的那种——增长 PM 基本上有一个跨职能团队,PM 本身就是团队一员,还是说单独设一个增长团队放在旁边也可以?
Crystal Widjaja: 我觉得如果公司真的是完全沉浸在令人疯狂的产品市场契合度中,忙到不可开交,那增长团队作为独立团队在旁边是可以的。如果有极强的产品市场契合度,你正拼命赶着做核心功能,那来一个增长团队做收尾工作没问题。我们是收尾小分队,捡起被遗漏的部分,把各种点连接起来。你忘了接这个,我们来帮你接上。但我们这个团队里有很多统计背景很强的人。我的团队中很多是统计学专业毕业的,我们非常看重数字、概率和赔率,因为在那样的规模下,增长确实是一个数字游戏。你做什么都可能会有点效果,但什么是当下能产生最大影响、并为未来解锁可能性的那个东西?
早期增长人才的招聘
Lenny: 我正想问你,大家在招早期增长人才时应该看重什么——就是你说的那种吗,偏统计、数据型人才?
Crystal Widjaja: 你必须找一个会算数的人。如果你在看转化率的比率,但没意识到这个比率的基础样本量要小得多,你就会做出错误的决策。所以需要一个对统计学有直觉的人——他们知道如何正确抽样,知道什么是选择偏差。最糟糕的事情莫过于一个增长人员自以为在做正确的事,结果度量方式是错的,然后把精力集中在了错误的方向上。
Lenny: 你觉得招一个年轻有潜力的人更容易还是更好,还是找一个经验丰富的人来做第一个增长招聘?
Crystal Widjaja: 我会招一个愿意去上统计学入门课程的人。不管他们之前有没有放开手脚大干一场的经验。我觉得关键是他们能否聚焦在正确的机会上,而不是最花哨的事情上。我觉得两种背景的人都可能做到这一点。
Lenny: 明白了。那在招聘这类人的过程中你会怎么做?你建议创始人考察哪些方面?
Crystal Widjaja: 我实际上会考察那种第一性原理思维的倾向。我会给候选人案例研究——这是我们观察到的现象,你怎么知道这是真的?然后让他们设计一个实验方案。我想看到他们在做随机抽样,而不是那种”我要做这个功能然后上线,当然会成功”的态度。我想看到他们采取一种审慎的、有条理的方法来思考——为什么用户会这样做?有哪些工具可用?增长团队如果搞砸了,很多时候就是因为他们试图引入一堆新的工具,但这些工具之间整合得不好,花了六个月才完全集成好,然后六个月什么都没干成。增长中的一切都是时间的机会成本,与你在那段时间本可以对产品做的事情做权衡。
所以我们偏向于非常快速的 hack 方式。比如在 Gojek 增长团队的早期,我觉得我们第一个真正的增长实验——当时我们其实还是数据团队——就是把一个简单的 Python 脚本连接到我们有权限的 Twilio API 上。我们通过上传的 CSV 给一批司机发了短信,内容大概是:“嘿,你的接单率很低,不应该这样,请接受你收到的所有订单。“这个举措实际上让整体接单率提高了 2%。当我们深入分析数据后发现,对全新司机的效果更加显著。于是我们与司机 onboarding 团队合作,帮助他们优化司机的 onboarding 体验。
面试中的实验设计题目
Lenny: 你刚才提到的面试问题,那个实验设计题,你是给他们作为一个项目让他们有时间去做,还是现场实时完成的?
Crystal Widjaja: 对,是带回去做的。我不觉得现场做这类案例研究效果很好。我想看到人们投入时间和精力,尽自己最大能力把事情做好。当然我们会跟他们说,“嘿,你有五天时间,我们预计你大概花四个小时在这上面,所以如果你在这五天内抽不出四个小时,请告诉我们。“所以我们在给予他们适当的时间方面比较谨慎,让他们能够以我们期望的、相当于全职工作时的质量水准来完成。给他们这四个小时,我们想看到他们会不会去 Google。如果他们当下想不出来,让我们看到他们会去搜索。我们会问他们采取了什么方法,是怎么解决这个问题的。我们很乐意听到有人说他们确实去 Google 了,读了一堆白皮书。我自己也会这样做。
Lenny: 对于想为自己设计这类题目的人,你有没有一个已经不再使用的题目可以分享,或者能帮助别人设计自己题目的内容?
Crystal Widjaja: 有的。通话结束后我可以给你一个模板。
Lenny: 太好了。我们会把它放在节目备注里。轻轻松松。好的。我想聊的最后一个话题是你参与的一件非常酷的事情。你创办了一个叫 Generation Girl 的非营利组织。我记得它的使命是帮助女性和年轻女孩进入 STEM 领域。所以我很想听听这个项目,你是怎么参与进来的,它的内容是什么,以及听众如何支持你们正在做的事。
Generation Girl:让更多女孩走进 STEM
Crystal Widjaja: 当然。Generation Girl 对我来说非常亲近和珍贵。我和几位非常了不起的女性共同创办了这个组织,她们当时也在 Gojek,但现在全职在 Generation Girl 工作。这个项目的缘起,是因为我们反复听到一些令人恼火的评论,关于女性在 STEM 领域工作。比如,“你不可能是这个项目的工程师。你看起来像是喜欢化妆之类的。“我们的反应是,“没错,我绝对热爱化妆,但我写 SWIFT 代码也同样厉害,请让开。“正是因为经历了大量关于工程师应该长什么样、应该喜欢什么的偏见和误解,我觉得我们真的可以从《律政俏佳人》中找到共鸣——这是我最喜欢的电影之一,它传递的是你可以利用你所拥有的能力,无论你热爱的是工程、设计还是数据,你都可以成为你想成为的人,并且同样出色。
所以我们支持的很多女孩,如果她们上了我们的课后说,“其实我不喜欢工程。“我们反而会很高兴。这就是自主性和赋能——她们能够在没有文化偏见或社会压力告诉她们应该如何感受的情况下,自己做出这个决定。所以我们为 12 到 17 岁的女孩提供免费课程。我们也有大学课程。我们与教师合作,探讨如何教授 STEM 主题,特别是在那些不是每个学生都有笔记本电脑的地区。比如你怎么教他们使用 Figma 之类的工具。大家完全可以通过我们的网站支持我们、联系我们,我们的网站上有 PayPal,欢迎去看看。
Lenny: 你能分享一下你看到的成效吗?有没有可以分享的数据,或者关于这个组织做了什么的任何信息?
Crystal Widjaja: 我们已经有数千名学生参加过 Generation Girl 的夏令营、项目和课程。我们每周都有活动。我们有一个完整的夏令营,每年夏天和冬天各举办一次,连续两周每天都有。我们与印尼一些最大的科技公司建立了合作关系,把学生和工程师配对,一起做项目。最近我们还凭借新项目 Class 入选了 MIT Solve 项目。Class 是我们正在打造的一个供教师免费使用的网站。
目前我们已经与印尼的几所大学建立了合作,既有农村地区的,也有雅加达市的,教师们现在可以获得相关的知识和教学材料,来讲解一些他们可能不太熟悉的新概念,因为创业世界变化很快,开发方式也在快速变化。这是目前最让我们兴奋的事情,因为每位教师每年影响数千名学生。能够培训教师、给他们提供所需的资源,是一件非常重要的事情。
Lenny: 太厉害了。目前只在东南亚,对吗?
Crystal Widjaja: 只在印尼,因为坦率地说,这里是人们最需要支持的地方。全球范围内,STEM 对女性来说并不友好,也不够欢迎。我觉得过去几十年情况反而变得更糟了。计算机科学专业的大学毕业生中,女性占比不到 18%。所以我们真的在努力触达最年轻的一代,因为正是在那个年纪,你会被告知计算机科学只适合特定类型的人。
Lenny: 听到情况在朝错误的方向发展,真的很令人难过。你认为是什么原因造成的?
Crystal Widjaja: 我认为关于计算机科学家能做什么、应该得到多少支持,仍然存在很多固有的心理模型。研究表明,在最年轻的一代中——初中、高中阶段——男生比女生更有可能被安排上 STEM 入门课程。所以女生在那个较年轻的年龄段根本没有被纳入 STEM 的目标群体。因此当她们进入高中或大学的计算机科学课程时,她们远远落后。那种感觉很不好。没有人喜欢做班上最差的学生。所以她们更可能退出。我们在卡内基梅隆大学的研究中看到,他们在大学课程开始之前开设了计算机科学入门课程,而参加了这些课程的女性,毕业率与男性同学基本持平。所以关键在于提前为她们搭建成功的条件。
Lenny: 如果有人想帮忙的话,你刚才说有 PayPal 页面。还有其他可以采取的行动吗?
Crystal Widjaja: 有的。企业级软件。我们喜欢教授 iOS 开发,需要正版软件授权。我们每年有数百名学生,所以如果有资源可以联系我们。
Lenny: 太棒了。大家可以在 generationgirl.com 上找到你们吗?
Crystal Widjaja: 是 generationgirl.org。
Lenny: Crystal,非常感谢你来参加节目。已经占了你不少时间了。最后两个快问快答。大家如果想联系你,在网上哪里可以找到你?然后除了刚才聊的 Generation Girl,还有其他方式大家可以帮到你吗?
Crystal Widjaja: 好的。请到 crystalwidjaja.com 找我。你可以联系我,我的邮箱在上面。听众们,请正确做好埋点。请不要只追踪你的 KPI。请追踪你的用户旅程和体验。如果你们这样做,我们聊起来会更有意思。
Lenny: 很棒的公益提醒。非常感谢你,Crystal。
Crystal Widjaja: 谢谢 Lenny。这次聊得非常开心。
Lenny: 非常感谢你的收听。如果你觉得这期节目有价值,可以在 Apple Podcasts、Spotify 或你喜欢的播客应用上订阅。也请考虑给我们评分或留下评价,这真的能帮助其他听众找到这个播客。你可以在 lennyspodcast.com 找到所有往期节目或了解更多关于这个节目的信息。下期再见。
术语表
| 原文 | 中文 |
|---|---|
| AB&B | AB&B(公司名,保留原文) |
| adoption | 采纳(指用户对产品功能的接受和使用) |
| aha moment | ”aha moment”(顿悟时刻,保留原文) |
| allocation model | 分配模型 |
| bread and butter | 看家本领 |
| churn | 流失 |
| cohort retention | 同期群留存分析 |
| conglomerate | 综合性企业集团 |
| Crystal Widjaja | Crystal Widjaja(嘉宾,保留原文) |
| D2C | D2C(Direct-to-Consumer,直接面向消费者,保留原文) |
| Darius Contractor | Darius Contractor(产品与增长领域从业者,保留原文) |
| decacorn | decacorn(估值超过百亿美元的初创企业,保留原文) |
| engagement | 参与(指用户与产品的互动活跃度) |
| Facebook Connect | Facebook Connect(保留原文) |
| Generation Girl | Generation Girl(非营利组织,保留原文) |
| GMV | GMV(成交总额,保留原文) |
| GoFood | GoFood(Gojek 旗下外卖服务,保留原文) |
| Gojek | Gojek(公司名,保留原文) |
| GoPay | GoPay(Gojek 旗下电子钱包服务,保留原文) |
| instrumentation | 埋点(指在产品中植入数据采集代码以追踪用户行为) |
| KPIs | KPI(关键绩效指标,保留原文) |
| Kumu | Kumu(公司名,保留原文) |
| leapfrog | 直接跳过(技术发展阶段) |
| Legally Blonde | 《律政俏佳人》(电影名) |
| Lenny | Lenny(主持人,即 Lenny Rachitsky,保留原文) |
| Made to Stick | 《让创意更有黏性》(Chip Heath & Dan Heath 著作) |
| MIT Solve | MIT Solve(麻省理工学院的社会影响力项目,保留原文) |
| OKRs | 目标与关键结果 |
| onboarding flow | 新手引导流程 |
| opportunity cost | 机会成本 |
| OTP | OTP(一次性密码/验证码,保留原文) |
| performance marketing | 效果营销 |
| product market fit | 产品市场契合度 |
| psych model | 心理模型 |
| Reforge | Reforge(增长领域专业培训平台,保留原文) |
| scrappy | 精益(在资源有限条件下灵活高效地推进) |
| selection bias | 选择偏差 |
| super app | 超级应用 |
| SWIFT | SWIFT(苹果编程语言,保留原文) |
| top up | 充值 |
| Typeform | Typeform(在线问卷工具,保留原文) |
| value prop | 价值主张 |
| white papers | 白皮书 |
| Wizard of Oz | ”绿野仙踪”式体验(指用人工模拟自动化功能的 MVP 验证方法) |
此文档由 AI 分片翻译(translate_long_document)