In this episode, we sit down with Karl Ulrich, CIBC Endowed Professor at the Wharton School, as we dissect his recent research experiment comparing startup ideas from Penn Students from those generated by ChatGPT. We discuss the experiment’s goals, key takeaways, and the game-changing speed of AI in idea generation.
Explore the motivation behind this experiment and the collaborative dynamics between AI and humans. Karl highlights the teamwork aspect, envisioning a future where both AI and humans play crucial roles in the ideation process.
Don’t miss this exploration of AI’s impact on creativity and the evolving landscape of collaboration between machines and humans.
Transcription: [00:00:00] Sean Ammirati: In this week’s podcast, we have an academic thought leader, and a trailblazer in the field of innovation, entrepreneurship, and design. I’m incredibly excited to have Karl Ulrich, the CIBC-endowed professor at the Wharton School. His books include Product Design and Development, Innovation Tournament Handbook, and Winning in China.
[00:01:39] Sean Ammirati: Today, our conversation with Professor Ulrich will really pivot around the research that many of you may have seen recently covered in the Wall Street Journal, where he had ChatGPT generate ideas and then compared those to MBA students ideas in his class. And I think it really it got a lot of publicity.
[00:02:00] Sean Ammirati: I think it also illustrates some really important points and I’m incredibly excited for this week’s conversation on Agile Giants.
[00:02:06] Karl Ulrich: Yeah, Sean, great to join you. It’s a real pleasure.
[00:02:10] Sean Ammirati: Thank you so much. And Karl, maybe as a way to get started here, my guess is most people did see the Wall Street Journal coverage, but maybe just as a refresher.
[00:02:19] Sean Ammirati: And for those who didn’t. Could you talk a little bit about the experiment you did and maybe what the goals of that experiment were? Sure.
[00:02:26] Karl Ulrich: First of all, let me credit my colleagues. So I had three great co-authors. Christian Terwiesch, who’s my longtime collaborator. Karan Girotra, who’s now a distinguished faculty member at Cornell Tech, but was our doctoral student.
[00:02:41] Karl Ulrich: And then a current MBA student, Lennart Meincke who helped us with a lot of the data collection. And so the four of us looked at the question of how GPT 4, one of the leading LLMs, performs relative to an elite group of Penn students in terms of generating new product ideas. So that was the basic study that we performed.
[00:03:07] Sean Ammirati: If you could summarize the key takeaways from it, so I think the Wall Street Journal did sort of the the high level, so how would you summarize what you learned from that question?
[00:03:17] Karl Ulrich: I should first say that as you know, and as many academics know, many writers know, we don’t write our headlines.
[00:03:26] Karl Ulrich: The publication writes the headlines and they framed it as pitting MBA students against GPT. The reality was the majority of our students were not MBA students. They were Engineering and Wharton as business undergraduate students at the University of Pennsylvania, and some of the criticism of the paper pointed out that, “hey, wait a second, is that a low bar comparing GPT’s performance relative to undergraduates.”
[00:03:58] Karl Ulrich: And I think that’s fairly ridiculous as a criticism, given that these are some of the smartest people on the planet. Our alumni include Elon Musk, for example, and many founders of very successful businesses like Warby Parker, Snapdeal, and others. So these are super talented students. And what we did was we took a sample of ideas generated in my product design class prior to the widespread introduction of LLM.
[00:04:28] Karl Ulrich: So that’s prior to November, I guess it would have been November 2022. Right. That’s right. And, prior to November 2022, we had taught this course many times. So we took a sample of product ideas generated in that course, a sample of 200 randomly selected. And we compared those ideas to the ideas generated by GPT 4.
[00:04:52] Karl Ulrich: In response to the identical prompt, the prompt was, create an idea, create an opportunity, identify an opportunity for a new product that could be sold to college students, the key result was that the ideas that the LLM generated that ChatGPT 4 generated were better as measured by consumer purchase intent.
[00:05:22] Karl Ulrich: They were better than the ideas generated by the students, both in terms of the mean quality, the idea, but also in terms of the extreme values, the best ideas generated by the cohort.
[00:05:33] Sean Ammirati: Yeah. Awesome. And so I wanna get into sort of what this means for the future and, and things like that in, in a little bit.
[00:05:40] Sean Ammirati: But I guess maybe to start with. What was sort of the, like, you guys are sitting around with your three collaborators and saying like, “Hey, this would be an interesting experiment to run.” What made you want to run this test?
[00:05:52] Karl Ulrich: In some ways it seemed like the obvious question to ask if you’re thinking about LLMs and AI in terms of assisting innovation, you could think about the development process and the various phases of the development process.
[00:06:10] Karl Ulrich: And the first phase is generating or recognizing or sensing the opportunity. And then there are subsequent phases, but, but we felt that. This first phase would be the most challenging after all, we normally think of computers as being best at tasks that we think of as being more algorithmic and generating the idea itself is the ultimate creative task.
[00:06:36] Karl Ulrich: And so we thought that was a pretty high bar to set. To see how the machine would perform in doing the most creative part of the ideation process or of the product development process. And so that’s why we picked that particular challenge. It also, as with many academic studies, we had a convenient data set available. So we were, to some extent, guided by the data that we had at our disposal to do the analysis.
[00:07:02] Sean Ammirati: Yeah, I think that makes a ton of sense. I think to me, like, and you talk about this in your paper as well. We’ll get to kind of human and computer together collaboration. But to me, like the other part that’s relevant here is the speed at which those ideas were generated, right?
[00:07:20] Sean Ammirati: Like how super talented students at Penn versus Chat GPT and the relative time to generate those is different as well. And how do you think that impacts kind of where we’re heading with human-computer creativity together?
[00:07:36] Karl Ulrich: Yeah. First, let me just give the headline on the difference in speed. We do measure speed, but probably the better metric would be the cost of generating the ideas. There’s an old saying that ideas are a dime, a dozen, and I don’t believe that saying has been adjusted for inflation, and it’s probably a hundred years old, so it could be ideas are a dollar, a dozen, something like that, if adjusted for inflation.
[00:08:04] Karl Ulrich: We found that ideas were actually about a dollar a dozen as generated by GPT4, so incredibly inexpensive. When I have done the estimates of the cost of generating ideas with executive groups, ideas typically cost hundreds of dollars per idea to generate, so more than a thousand dollars per dozen ideas.
[00:08:31] Karl Ulrich: So this is something like a thousand times less expensive. As a method for generating ideas. So that’s a really striking finding. Yeah.
[00:08:42] Sean Ammirati: Okay. So, but what do you think, how should an executive listening to this interpret that? Are they going to evaluate a thousand more? Are they going to? What do you think this means for them in their workplace?
[00:08:56] Karl Ulrich: First let me say, I am not an economist. I’ve actually never taken an economics class despite being a business school professor, but let me channel my inner economist and say that really, if you reduce the cost of supply, then that is naturally going to increase the quantity produced, basically.
[00:09:17] Karl Ulrich: So, if ideas become a thousand times cheaper to produce, then it’s going to be optimal for any ideating entity to generate a lot more ideas. And, and so, of course, in some prior work with Christian Terwiesch and our work on Innovation Tournaments, we found that as you would expect, the quality of the best idea in a sample of ideas increases in the number of ideas you generate.
[00:09:49] Karl Ulrich: So that should be relatively obvious. If you generate 200 ideas, the best of the 200 ideas will be better than if you only generate 100 ideas or even 10 ideas. So that should be relatively obvious. So basically what happens is, you generate ideas until you don’t expect to find one that’s much better.
[00:10:11] Karl Ulrich: And for most people, most groups and individuals, they’re generating dozens of ideas, maybe 5, 6, 7, or maybe the very best ideators and designers are generating 20 or 30 ideas. But given the dramatically lower costs of generating ideas, the LLMs assisting the idea generation task make it feasible, even optimal, to be looking at more like 200, 300 ideas.
[00:10:40] Sean Ammirati: Yeah, I think that’s exactly right. But you can also do it systematically in ways that would have been hard to do before as well. So like, rather than just 200 ideas, you could say, you know, 200 ideas in each of these different brainstorming approaches you have, and then do some evaluation of each of the different approaches, which would be hard to do before it was possible to make these kind of steps before.
[00:11:05] Karl Ulrich: Well, Sean, let me build on that. for just a second. A lot of times when I’m teaching executives about innovation, I suggest that in this phase of the process, what you’re really trying to do is build a map of the landscape. You’re trying to understand what the landscape of possibilities is. And what I would recommend for ideators to do is to first sketch the landscape, think about what the various dimensions of the landscape would be.
[00:11:33] Karl Ulrich: Just top of mind using your human capabilities and then, use the LLM to elaborate various points in that landscape. So say for instance, you’re developing a new food delivery service. You might say. Let’s go look at all the alternative modes of delivery. That might be an area of the landscape to look at.
[00:11:57] Karl Ulrich: Another area of the landscape would be, let’s look at all the menu types that we might look at. Let’s look at all the ordering methods that might be possible here. So you can use the rough map, which the human could generate, and then go and elaborate the various points on that map, the various territories on that map.. Using the LLM. And you could do that very quickly and quite exhaustively.
[00:12:21] Sean Ammirati: Yep. I think that’s exactly right. And we’ve been doing some kind of similar experiments in the lab at CMU and it’s like you can also have it generate things that you might forget. So maybe it generates those four and it’s like, here’s three more.
[00:12:32] Sean Ammirati: But then once you have those, you can use like a methodology, like TRIZ to actually systematically look at that problem 40 different ways as well, which to me gets, again, these are things that you just, to your kind of prediction machines level point about when things get cheaper, you can do like, that just would not have been realistic, you know, two years ago and today it is, which is super exciting to me.
[00:12:58] Karl Ulrich: But the other, just as you mentioned TRIZ, it stimulates another thought. One of the heuristics I teach students is to take the perspective of a particular organization in generating ideas or a particular approach. How would Apple approach this problem? How would Google approach this problem?
[00:13:20] Karl Ulrich: You could even say, what would the TRIZ approach be to this problem? The LLMs are particularly good at taking a perspective. So you could say, how would Da Vinci approach this problem? Or how would Newton approach this problem? Or how would Elon Musk approach this problem? And I think it works as a heuristic, even for the computer to generate it better, to generate better ideas.
[00:13:44] Sean Ammirati: 100%. I think that’s exactly right. And as I started getting to the other thing I was really curious to get your take on, which is you, you sort of point to like the future is really more collaboration versus competition or teamwork here, where it’s like human and computers co-creating together.
[00:14:03] Sean Ammirati: And you would not know this because it hasn’t published yet. We’re recording before either of these is published, but the episode right before you is the Google DeepMind team that created the Dramatron system. So, co-writing screenplays and scripts, and so sort of a very, very different lens at this co-creativity thing. I’m curious with what you’ve been doing looking at this ideation process, but like what is the the human plus computer framework here? Like how should people be thinking about this?
[00:14:31] Karl Ulrich: I have lots of thoughts on that question I think one question you could ask is if you think about the temporal or logical sequence of tasks in creating product or in realizing an innovation, there’s probably a way to decompose those tasks and assign them to the entity that’s best capable, most capable of completing them.
[00:14:55] Karl Ulrich: So as it stands, the LLMs are particularly well suited to the early stage where ideas can be quite well described as using natural language, using, you know, English or sentences and so forth, which, you know, just worth noting, that’s a little surprising, I guess, but given that humans are often the ones evaluating the ideas, it’s perhaps no surprise that natural language is a pretty good way to communicate ideas, but LLMs, they are language models.
[00:15:29] Karl Ulrich: And so when language can be used to describe the opportunity, they work pretty well. At the other extreme, you have tools like Midjourney that work on pixels and images. And, humans are less good, I think, at generating images, and so when you’re going to realize a scene or a visual representation of a concept, the LLM, or the generative AI tools can also be very powerful. Where I see a gap still is in the more detailed design, particularly for technical artifacts, where you’re trying to describe a geometry or, well, particularly a physical geometry. And we don’t really yet have AI tools that are adept at generating geometry where the geometry itself has mathematical meaning. So, for example, solid models, those kind of computer models of geometry.
[00:16:35] Karl Ulrich: But real design of technological artifacts, that’s critically important. When you go to design the winding in a electric motor for an electric vehicle, you have to work with real physics and real geometry. And as yet, a generative AI tool has not proven to be yet developed for those applications.
[00:17:03] Karl Ulrich: I think that’s only a matter of time, but that’s an area where it still feels like the human designer is still playing a really critical role.
[00:17:12] Sean Ammirati: I could not agree more. I also think other machine learning approaches there, kind of namely, you know, sort of optimization type, sort of other purchase optimization may be part of the equation in that as well.
[00:17:26] Sean Ammirati: Like the systems may be less siloed two years from now versus now, but this, the point is though, let’s say we solve that engineering problem. Like what’s the role for humans and the human-machine co-creativity going forward then?
[00:17:44] Karl Ulrich: I think the human for sure remains the pilot and the architect, in that the end of the day, we’re typically trying to respond to a human impulse or a human need.
[00:18:01] Karl Ulrich: So you can’t take the human out of that and that it’s wired into the goal is we’re doing something for humans. So in terms of the direction, the approach, and maybe some of the key building blocks of the solution. I think the human probably remains in the cockpit for generating, for doing innovation.
[00:18:25] Karl Ulrich: Of course, even that you could take the AI and use it at a meta level and say, “Hey, I’d like to generate more revenue for my business. What would be some opportunities for growth?” And in that sense, you could even generate ideas for the basic direction that innovation could take, but at the end of the day, at least until the overlords take over, we, humans are the purpose, or we are the ultimate purpose of using the LLM. So, so we remain in the cockpit.
[00:18:58] Sean Ammirati: Yeah, I like that. I think that the co-pilot analogy, I know you referenced this, but it’s a helpful one where Microsoft got the branding right. I’ve been in the space of entrepreneurship talking about it as like, you still need a CEO, but this might be a really good co-founder for you.
[00:19:15] Sean Ammirati: But I like how you framed that a lot, Karl. I want to step up a level for my last two questions. You know, really value the things you’ve written and the work you’ve done for a long time, but wearing your Wharton hat, and I realized you teach students who aren’t at Wharton as well, obviously per your opening, but how do you think about the future of business education? Given the things that you’re talking about in, you know, humans and AI collaborating on innovation.
[00:19:49] Karl Ulrich: Yeah, good question. So, let’s focus on graduate education for now and switch gears a little bit. Although before I do that, I think the general point I want to make is that you really have to think about the purpose of higher education, and in the words of Christensen, you know, what’s the job to be done?
[00:20:11] Karl Ulrich: Like, what, what is the purpose? Why are people hiring universities to do a job? And for undergraduates, I think it is primarily a four-year period of maturation that is socially sanctioned and supported by your family and, so it, you know, if I think about the four years I spent as an undergraduate and how many of those hours those four years have happened, were required to learn mechanical engineering, which was my undergraduate degree.
[00:20:43] Karl Ulrich: I don’t know, it’s probably 20 percent of the hours spent in that experience. Now if you switch to graduate education, it’s probably even lower than 20%. It might be 10 percent of the hours you spend in the two years of an MBA is about learning about net present value and generally accepted accounting principles and so forth.
[00:21:04] Karl Ulrich: So why do people do MBAs? I think they do them to construct a social network of peers basically that are going to serve them throughout their career. I think it’s a signal to employers of competence. I think it’s an access to a well-lubricated career path and primarily to entry-level jobs in the elite firms, the Amazons, Googles, McKinsey’s of the world. I do think, of course, you do learn some things. And then, as with undergraduate, I think for a lot of students, the MBA is a two year period. Again, socially sanctioned, acceptable to your family, in which you can really reflect on what you want to do with yourself and that might be the most important job to be done for many of our students.
[00:22:04] Karl Ulrich: So if we go back and then think about. Use that lens to look at AI. I don’t think AI disrupts many of those jobs to be done. I think the job to be done remains. Those jobs are very human and very personal and very connected to the human experience. I don’t think they’re displaced or disrupted by technology.
[00:22:32] Karl Ulrich: The question I think, is the ultimate jobs these MBAs go and get, do those go away? Do those become different in the future? And does a graduate degree, does a business education program become less essential to accessing those jobs? My gut instinct is that it doesn’t change.
[00:23:00] Karl Ulrich: I mean, I think, my guess is the specifics of jobs will change the way we do some of those jobs, but it’s pretty hard for me to imagine that there will not be roles for the, you know, plus one sigma plus two sigma performers in administrative skill that seems like something that will persist indefinitely as a need in society.
[00:23:28] Karl Ulrich: So my guess is that business schools and business education remain relevant for the foreseeable future. The job itself I think can change, may change quite a bit. And also it may be the case that the variance in performance starts to be diminished as the lower performers can at least get to mean performance levels pretty easily with the use of tools.
[00:23:54] Sean Ammirati: You’re smarter than me on this stuff. So I am sure you’re right. I do wonder if the jobs they go into change more than you’re saying there. And if there are other ways to get some of those same jobs to be done that may be more efficient than a two-year MBA program. But I mean, I obviously teach MBA students as well. So I hope you’re right. Just sitting where we sit today, like, you know, Y Combinator is another way to get some credentialing, build a really good network, have lubricants to other places, you know, other jobs after your startup experience as a sort of entrepreneurship education comparison.
[00:24:36] Karl Ulrich: Sean, first of all, I agree with you. I would not argue the job isn’t going to change. I think the job is going to change. So I agree with you 100 percent on that. On the question of, if we just take the job you alluded to, which is this filtering mechanism or selection mechanism, and you cite Y Combinator as an alternative selection mechanism, those selection mechanisms strike me as very important in society, and very useful in society.
[00:25:13] Karl Ulrich: And something that obviously people are willing to pay a lot of money for because they give access. I think they remain relevant. And then I think it’s really an interesting question as to whether business schools remain the best way to do it. Y Combinator it’s worth noting is, it’s harder to get into Y Combinator than it is to get into, you know, Stanford Undergraduate. So, you know, Y Combinator is an extremely, effective selection mechanism. And so, you know, I don’t think business schools threaten Y Combinator or vice versa, given the numbers. I do wonder if you look, for example, at coding, there have emerged hacker, hackathons, coding weekends, where people can demonstrate what they’re capable of doing in a very realistic, incredible fashion by actually showing performance.
[00:26:11] Karl Ulrich: And I think that has lessened the critical importance of an undergraduate computer science degree in accessing some technical jobs. I think that is possible in business. I mean, imagine a week-long experience in which you actually had to work in teams and you got peer evaluation and you had to demonstrate performance in terms of getting something done administratively or in a managerial task that could displace a business education in my opinion. But I don’t see that as an AI isn’t as, as something that AI will do.
[00:26:48] Sean Ammirati: Good clarification .Let’s come back to the AI point though. Cause given the audience for this podcast, probably even like, I’m very interested in the business school question, but I’m guessing the questions that the executives listening to this really want me to ask you is like, okay.
[00:27:04] Sean Ammirati: What about for my companies? How do you think companies should think about changing how they’re structured, how they’re organized, going forward, given this shift in technology that we’re experiencing?
[00:27:19] Karl Ulrich: I guess the first thing to say is I would do my best to bring a huge dose of humility to this question because I don’t think we have any clarity at all on this.
[00:27:35] Karl Ulrich: And so what do you do when you face a very cloudy future? It seems to me, what you do is you run lots of parallel experiments.You try to embrace a more relaxed culture, one that embraces greater levels of experimentation. And you work to enhance your sensing capabilities to try to find what interesting experiments are happening within your organization, on the edges of your organization.
[00:28:09] Karl Ulrich: And you look for ways to reinforce those things that are working particularly well. So I honestly have no idea. I just think, personally, despite having thought a lot about these questions. I just feel really feeble in my ability to forecast what’s actually going to happen, to work. So that’s why I just suggest you want to put on your best agnostic problem-solving hat, as we embark on this future.
[00:28:43] Karl Ulrich: Having said that, I do think that it’s critically important that senior executives get some first-hand experience. So I think, if you haven’t yet had the aha moment where you ask yourself, “wait a second, how is that even possible? How could this possibly work?” If you haven’t had that experience yet, you need to get start playing around with MidJourney or GPT 4 and, and really get a sense of that yourself.
[00:29:15] Karl Ulrich: That even goes to the level, I mean, just this last week, I’ve been using GPT 4 to do data analysis. And had this amazing revelation that I don’t need to know SQL and that I can literally use English to query a database for that matter. I don’t even have to upload the data. I can just point the LLM to a data file on the web and in natural language, give it a query.
[00:29:50] Karl Ulrich: And it writes Python, it runs the code, it returns the results. And it’s just phenomenally useful and it’s something, you know, I’ve been putting off for years, really learning how to program, to be a database programmer, just, you know, for my general needs. And I realized I don’t ever need to learn to do that.
[00:30:14] Karl Ulrich: These tools are just so good. So that’s an example of something that I would argue, senior executives need to get their hands dirty with to get a better sense of what kind of transformative forcest hese AI tools are going to have on work.
[00:30:31] Sean Ammirati: I could not agree more with everything you just said, and starting with the learn by doing, but the spirit of humility is, I mean, everything you just said, I think, is exactly right.
[00:30:44] Sean Ammirati: And I think, you talked at the beginning about sort of mapping out where you’re heading. I think in some ways, the spirit of experimentation allows you to sort of in real time, continue to push the map forward there. There was a paper that’s from some Stanford professors where they talked about how to teach data science given large language models.
[00:31:05] Sean Ammirati: And the analogy they drew is it was sort of a pivot from data engineering to product management. And I think like, as you were talking, that analogy kind of came back to me. It’s like, look, how do you become a great product manager? Where you kind of understand the art of the possible. You ask the right questions, you put yourself in the right sort of humble attitude towards it. And you kind of figure it out as you go by, by doing, not just by sitting in your office and thinking.
[00:31:24] Sean Ammirati: That was just, I mean, I started talking about you as a academic thought leader, Karl. It’s insights like that are amazing. I really appreciate you sharing those with our audience today. If people want to stay aware of you and what you’re doing, we’ll include links to whatever social media stuff you ask us to in the show notes, but is there any one place you’d encourage them to follow you, subscribe, whatever, to your content?
[00:31:58] Karl Ulrich: I try to put. I put everything on my website, which is just KTUlrich.com. And that’s also my Twitter handle. And I’m a modestly active X-er.
[00:32:19] Sean Ammirati: Amazing. Well, thank you so much for taking your time to contribute to this conversation today.