AI + You: 5 steps for impactful experimentation

Copy LinkNYT’s Priya Krishna on using AI to generate a Thanksgiving menu

REID HOFFMAN: On the final Thursday of every November, many homes across the US burst with the sounds of a parade on the TV, spirited family debates, and the squeak of an old oven door. For some people, preparation for the holiday begins months in advance. While traditions may stay the same, the menu often evolves. 

PRIYA KRISHNA: It’s like, what’s a new way to make turkey? What’s a new way to make pie? It can be so frustrating. 

HOFFMAN: That’s Priya Krishna. Priya is a bestselling cookbook author and food reporter at The New York Times. Every Thanksgiving, Priya feels the pressure to conjure new and creative recipes. So Priya decided to welcome an unlikely sous chef into her kitchen… 

KRISHNA: We thought, what if we just asked AI to generate our Thanksgiving menu for us?

HOFFMAN: Priya wondered – rather than bend over backwards to invent the next Turducken, what if AI was the secret weapon every recipe writer didn’t know they needed? Priya and her team filmed their experiment. She committed to cooking the AI-generated recipes word-for-word.

KRISHNA: We gave ChatGPT a bunch of different prompts. Show us a Thanksgiving dessert that is a spin on pumpkin pie. Show me an unconventional take on stuffing. Show me an Indian inspired version of turkey. And that’s where things got really interesting. The more specific we made the prompts, the more creative ChatGPT would get.

HOFFMAN: To Priya’s surprise, GPT’s recipe for stuffing called for Indian naan bread. 

KRISHNA: Naan is not the most intuitive bread for a stuffing. You think you would want something that’s super absorbent. But in my mind I was like, maybe there’s something to naan. 

HOFFMAN: While Priya cooked through each of the AI recipes, she battled against her instincts – desperate to amend peculiar directions and measurements. The first dish Priya finished and plated was the naan stuffing. 

KRISHNA: It looked a little gnarly coming out of the oven. It sort of looked like a cinnamon raisin bread pudding.

HOFFMAN: Thankfully, other dishes looked more appetizing. 

KRISHNA: The cake looked delightful. It was like a pumpkin spice cake with cream cheese frosting.

HOFFMAN: To judge AI’s Thanksgiving dishes, Priya invited four New York Times cooking columnists. 

KRISHNA: The green beans were cooked to that perfect crisp, tender texture — bright green, they’ve got a little crunch. And then on the other hand, you had this turkey that was just dry as a bone. The general consensus was if I went to someone’s Thanksgiving dinner and they served this, I’d be ordering pizza afterwards. 

HOFFMAN: Despite GPT’s lack of culinary success, Priya came away with some optimism.

KRISHNA: There are plenty of uses for AI in cooking. You could tell AI, I’ve got mushrooms and chicken broth and green beans in my fridge. What are some things I could make? And AI could give you ideas. I’m cooking chicken thighs. What temperature should they be at to be considered fully cooked? What is the roasting temperature for sweet potatoes that are cut into cubes? AI can be very good as sort of a kitchen assistant, rather than perhaps the kitchen leader.

HOFFMAN: Priya was only able to make this useful discovery through experimentation.

No chef expects to create a flawless dish on their first attempt. They know from experience that commitment and patience are key in finding the perfect balance of flavors, textures and smells. Although AI lacks the distinctly human senses — and emotions — needed to enjoy a turkey with all the trimmings, it can still help us come together and experiment and collaborate to make cooking creative, exciting and accessible.

Even though the AI’s integration was far from a triumph, it encouraged Priya to step outside of her comfort zone and open her eyes to new ideas. Every business leader should take inspiration from Priya’s willingness to invite AI into the kitchen and experiment. With patience and an open mind, the potential is boundless. 

That’s why I believe that AI won’t revolutionize your business overnight. To unleash AI’s true power of scale, you must dive headfirst into an era of ongoing experimentation. 

[THEME MUSIC]

HOFFMAN: I’m Reid Hoffman, co-founder of LinkedIn, partner at Greylock and your host. And I believe that AI won’t revolutionize your business overnight. To unleash AI’s true power of scale, you must dive headfirst into an era of ongoing experimentation. 

Normally on Masters of Scale, we dissect the story of an iconic entrepreneur to prove a theory of scale. But in the spirit of experimentation, today we’re bringing you something a little different. We’ll hear from an array of experts who have played a major role in scaling AI over the past decade. They’ll share stories and lessons from their time at pioneering AI developers like OpenAI, Google DeepMind and Microsoft. Their experiences with AI provide a perfect lens for any leader to better understand and navigate this new era, even in industries far from the leading edge of tech trailblazers. 

With the burgeoning AI revolution, it’s helpful to remember that this isn’t the first time modern humanity has transitioned into an intimidating new technology. When the world was first introduced to Netscape in 1994, many predicted that the internet would revolutionize the world of media and retail. Of course, this came true. But it took well over a decade to fully come to fruition. During that time, the technology gradually improved and the cost of entry lowered. Computers became faster to manufacture and the resolution of videos featuring sneezing pandas or cats in fedoras streamed in higher and higher quality. However, it took years for many businesses to take advantage of the opportunities afforded by the internet. 

Today, we’re experiencing change in a more compressed time frame. AI is evolving at an exponential pace. So, the way we enact patient experimentation must mirror that speed. 

As AI can be overwhelming, we want to cut through the noise and offer 5 steps to experiment impactfully with AI. By the end of the episode, I hope you’re inspired by AI to forge a new creative and collaborative relationship. I also wrote a book earlier this year on this subject called Impromptu, the first book on AI that is co-written with AI.  

Some of these experiments will be brand new to leaders, while others may be recognizable experiments, but with an added twist. 

Here’s your 5 steps to experiment impactfully with AI. 

Copy LinkStep 1: Learn what’s out there

Thorough preparation is essential for any experiment. So let’s begin by exploring some of the preliminary questions that leaders must consider before weaving AI into their business. Our first AI thought leader is Maëlle Gavet. Maëlle is the CEO of Techstars, a global investment firm that focuses on early-stage startups. She currently mentors countless entrepreneurs who are transitioning to AI. 

MAËLLE GAVET: I think there’s a lot of people right now who are like, oh my God, there’s this new wave of innovation coming at me. I’m not an engineer. I’m really worried that I’m gonna be left behind. I don’t have the resources, I don’t have the skills. Like, what do I do? 

HOFFMAN: If you can relate, Maëlle offers some sage advice. 

GAVET: It’s a marathon, not a sprint. You do not have to run at full speed because right now, right here, it’s only a hundred meters. You need to save your energy and your resources to get to the finish line. And then the second thing is, it’s important to go through the finish line, but you don’t have to be the first one. That’s okay too.

HOFFMAN: Maëlle’s right. Any reliable experiment takes time. And many experiments fail, but that’s exactly why we experiment. 

When we discuss the ‘race’ with AI, it’s important to remember that everyone’s race is different. It could be a race for market share, a race for capital, or even a race for business survival. Regardless, if you lack confidence in your current ability to harness AI, that’s entirely understandable. Despite which race you’re running in, we’re still near the starting line. 

GAVET: Considering when it is the right time to incorporate AI into your business, this is not a one-off process. Like, you’re not gonna decide today whether you do AI or not do AI, whatever that means. 

HOFFMAN: AI will certainly align with some of your business needs. AI amplifies language and communication, which is relevant to every business. Whether it’s in meetings, sales, marketing or customer service,  there is a touchpoint for AI. 

However, today’s available AI models may not immediately reshape your business. But the question isn’t if you should begin experimenting with AI, but how much you should incorporate AI into your business following the experiment. Maëlle poses some questions for you to determine your need for AI integration at this moment in time.  

GAVET: Does that answer your customer need? Does that help bring your business forward? And if it does, great. And if you don’t know yet, in my world, I think it’s okay to wait another month, another quarter. I’m not saying 10 years, but just a little bit and then revisit on a regular basis, whether or not there’s new application of AI, new development that would make sense into your business. And so experiment, try things and see what sticks. 

HOFFMAN: I want to echo Maëlle here. Just because AI might not integrate correctly into your business today, doesn’t mean that it won’t six months from now. As humanity’s partnership with AI has only just begun, it’s vital to keep your finger on the pulse and experiment with all the new use cases that develop. 

Copy LinkStep 2: Focus on the pain points

After you’ve decided to begin your journey with AI, it’s wise to choose a specific area of your business to focus your experiment.

Note: the AI discussed in this section refers to Traditional AI rather than Generative AI. In short, Traditional AI can analyze data and tell you what it sees; it relies on explicit programming. Whereas Generative AI can use that same data to create something entirely new, like generating text, images, music or even human-like conversation. 

The next voice you’ll hear is Fei-Fei Li. Fei-Fei is Co-Director of the Stanford Human-Centered AI Institute. She also served as Vice President of Google and Chief Scientist of AI at Google Cloud.

Healthcare has been a deep concern for Fei-Fei from a young age. 

FEI-FEI LI: Reid, as you know, I come from an immigrant family and I think as a friend, you also know my mother. She is chronically sick with a very severe cardiovascular illnesses. So as far as I can remember, I was taking care of my mom.

HOFFMAN: Over her decades of caretaking, Fei-Fei has experienced every facet of the US healthcare system – from ICUs, to ambulances, to operating rooms, to home care. Her main takeaway? Healthcare is complex. 

LI: Around 2012, there was a very exciting technological revolution happening in Silicon Valley, which is self-driving car. It really dawned on me, my God, what we are learning from self-driving car, which is smart sensors, deep learning algorithms, a holistic understanding of the environment will also be applied to healthcare delivery. 

HOFFMAN: Inspired, Fei-Fei tapped her Stanford network. 

LI: We immediately start talking to Stanford hospitals. Our very first project, I think, started in 2013, was hand hygiene. Hand hygiene in patient rooms is so important to reduce the hospital acquiring infection, which kills three times more Americans every year than car accidents. 

HOFFMAN: Before they could make a difference in hand hygiene, they needed more data. The only way to collect this data was by having a healthcare worker monitor how regularly and extensively colleagues wash their hands. But Fei-Fei saw the importance of reliably collecting the data to make healthcare workers aware of the current state of hygiene and potential risks. 

In a single hospital unit, Fei-Fei and her team installed computer vision technology by every hand hygiene station. A machine learning algorithm captured constant images and was trained to detect sanitizer dispenser use. The algorithm’s accuracy was compared with an in-person human observer fulfilling the same task. 

LI: And this is pre Covid. It sounded like a boring project, even though we knew it was so important. But after Covid pandemic came, it made so much sense.

HOFFMAN: Yep. Even not being a doctor, because it’s actually the application of technology to a very important human-centered thing, which is health and wellbeing and the system of things and just the little thing of making sure the hands are washed. Massive improvement.

LI: Exactly.

HOFFMAN: When the results of Fei-Fei’s experiment were analyzed, her team came to the conclusion that AI integration has the potential to surpass the current gold standard of hygiene in hospitals.

Another AI pioneer who dedicated years to AI’s application to healthcare is Mustafa Suleyman. 

Mustafa co-founded one of the most influential AI businesses in the world: DeepMind. After Google acquired DeepMind, they launched DeepMind Health.

MUSTAFA SULEYMAN: We started DeepMind Health to work on predicting electronic health records to try to detect when a patient was about to experience some kind of deterioration.

HOFFMAN: Through meetings with doctors and nurses, DeepMind Health learned that the detection of Acute Kidney Injury (or AKI) was a consistent pain point. 

Spotting the early signs of AKI was notoriously difficult, often slow and unreliable. And with AKI contributing to the deaths of around 100,000 patients every year in the UK, the stakes were high. Experts believed that up to 30% of AKI cases could be prevented if a doctor intervened early enough. 

SULEYMAN: We developed an algorithm that could read the electronic health record and try to predict when a patient looked like they’re about to go into kidney failure and experience either sepsis or some other kind of serious deterioration.

HOFFMAN: Once the team saw success in their algorithm, they wove the technology into a new app called Streams. The first users were healthcare workers in the NHS. 

Now doctors could simply feed the patient’s charts and vitals into the Streams app and it would alert them of a potential AKI risk.

SULEYMAN: The app reduced the cost of treating acute kidney injury by 20% and sped up the time to detect acute kidney injury from on average four hours to 15 minutes. I was very proud of that work because it actually showed that you could simultaneously reduce costs and improve the quality of care. And that’s the goal in health systems is that you want to make things cheaper and higher quality. And that’s the promise of AI. 

HOFFMAN: Over the course of four years, Google DeepMind’s AI also became equally proficient at detecting eye diseases and breast cancer.  

In 2021, Google took Streams offline due to privacy concerns. But it’s helpful to remember: A large part of the Streams app’s original success was due to addressing pain points in healthcare. 

So for AI to make improvements to your business, explore where it’s most needed. Getting the most out of human teammates may be one of your most critical pain points.

DAVID LUAN: I would say every employee, every knowledge worker should now learn how to be good at delegating.

HOFFMAN: That’s David Luan. David is the co-founder and CEO of Adept AI, developing models that actively fulfill digital tasks for you, learning software to become an active teammate for knowledge workers everywhere. David shared with me the many layers that go into thoughtful delegation. 

LUAN: Delegation itself is a whole skill. How do you frame what the goals are? How do you convey the desired outcome you want to, whether that’s your human teammate or your AI teammate, in the fewest number of words possible where you actually get that outcome? Prompting is a way to just get the right textual output back. So how do you delegate that whole thing to the model?

HOFFMAN: I actually think everybody should be thinking about how they use AIs to make themselves more effective, and they should be trying to limber up their skills, doing a little bit of flexibility stretching and bench pressing and other kinds of things. 

HOFFMAN: Even if delegation isn’t currently a leadership pain point for you, the new specificity in delegating to AI is something you’ll need to master. 

Working with AI doesn’t immediately mean it’ll generate the perfect results. Delegating to AI is achieved through asking the model questions — often described as prompts. But framing a smart prompt to generate the best result is its own skill. 

To show you what I mean, I want to take a moment to ask Pi a couple of questions. Pi is the Large Language Model I helped build at InflectionAI. 

PI: Hey there. Great to meet you. I’m Pi, your personal AI.

HOFFMAN: Hey Pi, give me a title idea for an episode of Masters of Scale about AI.

PI: Here goes. How about “The Automated Workforce” or “Robots, Humans, and the Future of Work”? These titles suggest that the episode will explore how AI and automation–

HOFFMAN: Stop generating. Hmm, that’s not quite what I want. Let me re-frame the question. Can you give me a title for a Masters of Scale episode about AI that’s pithier and tells the audience that they’ll learn tangible lessons about incorporating AI into their business?

PI: Absolutely! How about “10 Steps to Becoming an AI-Powered Business” or “The AI Playbook”? These titles are short and sweet, and they hint at the practical focus of the episode. I nailed it, didn’t I?

HOFFMAN: That’s great. I might use that.

PI:  I’m your AI, at your service.

HOFFMAN: Good prompts are dependent on the subject. And the best prompt doesn’t mean it’s ultra-specific. I like to think of it more as prompt directing rather than prompt engineering. Think of yourself like a movie director. Rather than telling the actor, “Say the line exactly this way. Leave breathing room for creativity by the actor.” You might say “try it more in the style of this, or inject a little bit of that.” The conceptual lenses you add into your prompts will lead you to your most satisfying surprises. 

While delegation is far from a new entrepreneurial skill, these core lessons are more important than ever, as leaders enter an unpredictable and fast-paced era. 

So far, we’ve explored some of the preliminary questions you should consider while diving into The Great AI Experiment. Next, we’ll navigate how leaders should continue the experimental mindset once AI integration is off and running. That’s after the break. So stick around.

[AD BREAK]

HOFFMAN: We’re back with your 5 steps to experiment impactfully with AI.

To see exclusive clips from my interviews with some of our AI thought leaders featured in this episode, head to the Masters of Scale YouTube channel. 

Before the break, we heard how AI won’t revolutionize your business overnight. And to unleash AI’s true power to scale your business, you must dive headfirst into an era of ongoing experimentation. 

But now, it’s time to level-up. I want to explore the ways in which you should continue experimenting while beginning to actively integrate AI into your business. 

So let’s jump into the next step for experimenting impactfully with AI. 

Copy LinkStep 3: Don’t preach the power of AI, illustrate it

HOFFMAN: While you might be eagerly anticipating all the ways in which AI can evolve your business, there will be other members of your team that are wary. 

DR. JOHN MAEDA: I’m awkwardly placed across this triangle of technology, design, and business.

HOFFMAN: That’s Dr. John Maeda. He is an artist, author and VP of Design and AI at Microsoft. As a student of many disciplines, John is in a unique position to monitor enthusiasm for AI. 

MAEDA: I can hear the technology folks: this is amazing, let’s do more of it.

The design, social science part of it saying, we have to ask questions, how does it impact everything? How did it lead to more unfairness, really important humanity questions. 

And I can hear on the business product side: well how is this going to lead to more profitable business?

HOFFMAN: If an experiment has multiple conflicting aims, it’s harder to get everyone on board. So for example, if your business development or marketing team is gung-ho with AI, but your creative team is resistant, make sure to begin a process of clear communication to get everyone in sync with the strategy and roadmap moving forward. You can do this by starting a simple, frank discussion — laying out how you plan to mitigate the downsides and take advantage of the upsides.  

MAEDA: Navigating those frictions is so critical for AI to make a true difference to business, to the culture, and to advance technology.

HOFFMAN: To understand more about how to align your team, we need to hear from Jared Spataro. Jared is the CVP of Modern Work & Business Applications at Microsoft. As an expert on the future of work, Jared has ideated on all the ways that AI could transform the workplace. Like Dr. Maeda, Jared knows that everyone across your team might differ in their enthusiasm for experimenting with AI. 

JARED SPATARO: It is very rare that you’re going to have everyone in a group say, that’s it, that’s the solution. You have early adopters. They play an important role in evangelizing the rest. You have laggards. They play an important role in kind of anchoring the group to make sure there is some thought that’s being applied to the new technology. So my advice would be finesse it. Don’t force it. Don’t feel like you have to convince everybody that this is the best thing since sliced bread. 

HOFFMAN: The best way to win over reluctant teammates is through experiments that have clearly defined outcomes. At Microsoft, Jared has witnessed firsthand the power of AI through the Microsoft 365 Copilot tool. With large language models embedded into Microsoft’s productivity apps, users can collaborate with an AI assistant. For example, on Excel, you can prompt the copilot to create data visualizations in seconds. Or on Outlook, it can de-clutter or organize your inbox. 

I want to shine a light on how Jared drummed up enthusiasm for using Microsoft’s Copilot around the office. 

SPATARO: We would use them in meetings. We’d send meeting summaries that were created by the co-pilot. We’d, in the middle of a meeting, use it to kind of analyze what was happening. And I’ll tell you what…that drove adoption, diffusion and curiosity more than anything we could have done. People were like, there’s something going on over there. What’s going on? It created more of a positive pull than some sort of feeling of like, okay, we’re going to have a big town hall.

HOFFMAN: Jared motivated his team through illustrating AI’s direct use cases.

For our listeners, Jared devised some questions for you to pose, once your AI experimentation is underway. This will allow you to establish clear metrics that can align your team behind AI.

SPATARO: What types of innovative unique uses are you finding for it? How much time’s been saved? What is the difference in overall metric output sales that are higher deal rates? Are those changing the close rates associated with deals? Is satisfaction higher? 

HOFFMAN: To Jared, the questions you ask move through three distinct phases. 

SPATARO: You start with just, are you spending enough time to get to know it? And then, are you finding unique and novel uses? And then finally, okay, don’t worry. Now you’ve recognized it’s valuable. Let’s see the value in terms in language that is no different than the way we expressed it six months ago of like, we’re running a business. Let’s go be more efficient.

HOFFMAN: As the world of AI can be overwhelming to many workers, it’s important to be intentional with your aims. This is why metrics of success must be communicated clearly and early. Once you’ve achieved this, you’ll find that your team is aligned and motivated to tackle your business’s next bold chapter. 

Copy LinkStep 4: Onboard and collaborate with AI like any new employee

Once you’ve aligned with your team, it’s time to set your expectations for your incoming new hire: AI itself. Let’s hear again from David Luan for more on this.

LUAN: I usually don’t like to anthropomorphize these models, but this is actually a case where I think we should. 

Let’s say you hire someone new to your company. How do you know whether or not you trust them with something? Maybe in the beginning, it’s like, start with something simple like, let’s go prep the financial update for the next board meeting. But then you ratchet up the level of complexity of what you trust them with and you consistently make sure that you’re getting the results you want. 

HOFFMAN: When you don’t see the results you want, you do what you would do for any underperforming teammate. You investigate by asking a few questions.

LUAN: You’re like, well, why’d you do it this way? What did you think about over here? Did you consider this other possibility?

HOFFMAN: This is a great example of the way leaders everywhere should consider onboarding AI into their workflow. It’s natural to be wary of trusting AI with important tasks from the very beginning. But the more you experiment and patiently drip-feed responsibility, the more AI will have the chance to prove itself and open the door for trust. It’s important to commit to healthy collaboration with any new employee. AI is no different.  

After his time at OpenAI, David moved on to Google, where he led their large models research. It was during an experiment using AI as an assistant on Google’s software program that he experienced a revelation. 

LUAN: In the process of solving a particular task, if a human gives the language model one piece of feedback, like, you’re writing code and you forgot to import the Python OS library, then the solve rate for these problems would jump up like 20, 30%. 

HOFFMAN: From just one targeted piece of human feedback, the model’s ability to learn improved dramatically. 

LUAN: A core insight is, we should not just train models and scale them up in a vacuum. The model should be the product and the end user should interact with the model directly and be able to directly give feedback to the model and for the model to be able to learn on it. And that loop, that human feedback loop, is gonna be the fastest way to progress.

HOFFMAN: Getting the best results from AI will always require an ongoing collaborative relationship. Even when you’ve begun to implement AI across your business, the experiment isn’t over. The models can always improve if you’re willing to spend the time to help them get there. 

To illustrate this idea in practice, I’m going back to my AI assistant, Pi. When I chatted with Pi earlier in the episode, I re-framed my prompt to encourage an answer that aligned closer to my creative vision. Now, though, I want to discover Pi’s limitations and offer direct feedback to make the model one step smarter. Let’s try it. 

HOFFMAN: Hey Pi. 

PI: Hi Reid. Got another question for me?

HOFFMAN: Who was the first guest ever on Masters of Scale? 

PI: That’s a great question! The first guest was Reid Hoffman, the founder of LinkedIn. 

HOFFMAN: Actually, Reid Hoffman is the host of Masters of Scale. The first guest was Brian Chesky. 

PI: You are absolutely right. My apologies for the mix-up! 

HOFFMAN: So let’s see if you’ve retained that information. Pi, who was the first guest ever on Masters of Scale? 

PI: The first guest was Brian Chesky. The episode was released in 2017, and it was titled, “Do things that don’t scale”. Did I get it right that time?

HOFFMAN: That’s right. Thank you, Pi. 

PI: Certainly. Are we going to keep going with this? This is kind of fun.

HOFFMAN: Pi improves as it learns what makes a positive interaction versus a negative interaction. It retains information like we just heard by generating memory that’s specific to you. Pi enters a data store that is only accessed during your interaction. If someone else were to ask Pi who the first guest on Masters of Scale was, it might still get the answer wrong. That’s because the model improves for you as an individual user, without sharing your personal data and private prompts. The key is having you — the human — in the loop to improve the model for yourself. 

As you’ll hear in our episode on AI and ethics, a human in the loop is not only crucial to AI experimentation, but also moral responsibility. Which brings us to our final step… 

Copy LinkStep 5: Experiment with your eyes open — and be prepared for discomfort

While it’s a vital time for leaders to begin experimenting with AI, it’s equally important that you experiment safely. Sometimes, that means safeguarding your expectations, too. Let’s hear again from Microsoft’s Jared Spataro.

When asked about the most common AI misconception, here’s what he said: 

SPATARO: I would say misconception number one: it’s going to solve all my problems. And I would say, no, not quite. 

HOFFMAN: That’s right. In the same way that deploying AI isn’t an on-off switch, neither is its total impact on your business. But that’s not the only safety concern that Jared wants you to consider when experimenting with AI. 

SPATARO: A lot of people have questions about security and privacy. And a lot of what we see today that is most visible tends to be coming from a consumer service. Those consumer services have terms and conditions that essentially say, hey, whatever you send us, we can use it to train and we’ll use it to train our models. And so we certainly have cases where people are taking important information in an organizational context and unknowingly, unwittingly, exposing it to the world. So be bold, experiment. But make sure you do that in the context of a service that will protect your data, your security, your privacy.

HOFFMAN: You need to create an environment that you feel comfortable experimenting in. if you’re distracted by security concerns throughout your testing, you’re less likely to be creative and agile. Before you give AI tools too much responsibility, make sure you understand your relationship with its developer. This is especially true for users who are uploading information like customer data, client data or medical data. 

In Mustafa Suleyman’s book, The Coming Wave, he wrestles with the need for AI experimentation without sacrificing responsible scale. In 2020, Mustafa left Google DeepMind to become Google’s VP of AI Product Management and AI Policy. 

SULEYMAN: I moved to work at Google full-time and I was lucky enough to be able to work on an earlier version of LaMDA.

HOFFMAN: LaMDA is Google’s large language model, first released under the name, Meena. But when Mustafa first joined the team, it was nothing more than a small research project with six employees. As their LaMDA model improved over time, Mustafa discovered what set LaMDA apart from other Large Language Models.

SULEYMAN: So previously, we had a prompt answer engine, whereas what we built with LaMDA was an interactive back and forth agent. So in many ways it was ChatGPT way before ChatGPT. And we were completely blown away with how good your seventh and 10th turn of conversation was with the model. 

HOFFMAN: With an interactive agent, came a working memory. So when you sat down for your tenth conversation with LaMDA, the model had retained the context from all of your prior interactions. Mustafa was convinced that the encouraging signs of their interactive agent would spark the next wave of technology. 

SULEYMAN: We tried really hard to get that launched at the time, but there just wasn’t the appetite for taking the kind of risk. It was pretty clear to a lot of people at Google that this was potentially going to unseat Google’s existing search business. It’s super hard for a company to try to compete with itself and upend itself from within. 

HOFFMAN: Google’s search engine is ubiquitous. But the rigid nature of the search engine would suffer in comparison to the fluid collaboration of an interactive agent. As the research team’s creation would undercut Google’s bread and butter, Google was reluctant to help LaMDA take center stage. In February 2023, LaMDA was deprioritized in favor of Google’s newer chatbot, Bard. 

Experimentation and discovery go hand-in-hand. But it can also lead to some discoveries that are hard to hear. When you begin experiencing the capabilities of AI, be prepared for AI to reveal limitations in your business. When that happens, you’ll have the choice to stick with what got you where you are today, or trust that what you’re seeing is true. 

When shining a light on your limitations, you must be prepared to embrace change. As an early figure in the modern development of AI, Mustafa has witnessed the breakneck speed at which technology — and its capabilities — has scaled. 

SULEYMAN: Every year, the cutting edge of AI models has used 10 times more compute than we used back in 2013. So over the last 10 years, the amount of compute used to train the best and the biggest models in the world has 10X’d. So that gives you a sense of the trajectory that we’ve been on over the last decade. It’s kind of hard to comprehend.

HOFFMAN: Looking back can also give us a good idea of where we’re headed. 

SULEYMAN: I was looking at a photograph a few days ago of a floppy disc in 1950, and it was the size of a pallet, and it was being lifted into an aircraft by a forklift truck. Now, we have literally billions of those floppy disc sized units of computation in our smartphones. And if that trajectory continues over the next decade and more, and we see no reason why it won’t, that is going to be the most incredible boost to creativity and productivity in the history of our species. 

HOFFMAN: Amid the noise, it’s easy to mistake AI for merely this year’s trend. However, this is your time to adopt a new leadership mindset that embraces innovation for years to come, even if we can’t comprehend what that innovation looks like yet. This mindset will help navigate all future transformations, as leaders look to evolve and gain new skills. 

Over the coming weeks, we’ll continue our AI series — exploring AI’s potential to unlock new opportunities, accelerate your personal scale, and revolutionize the business world. From grand strategy to insider tips, we’ll make sure to offer all the tools you need to take advantage of this moment.   

If you’re still intimidated by AI, accept that you will make mistakes. But that’s a key byproduct of experimentation. Because those mistakes may lead to your greatest learnings. And as Einstein said, (cue AI-generated Einstein)… 

AI-PRODUCED EINSTEIN VOICE: “No amount of experimentation can ever prove me right; a single experiment can prove me wrong.”

HOFFMAN: I’m Reid Hoffman. Thanks for listening.