Lessons from Corporate Innovators

Moderna’s AI Journey into Generative AI with Andrew Giessel, Executive Director of Data Science and Artificial Intelligence

Play episode

Episode description:
In the season finale of Agile Giants, we’re thrilled to welcome Andrew Giessel, Executive Director of Data Science and Artificial Intelligence at Moderna.

I first crossed paths with Andrew when Moderna’s AI Academy was launched in collaboration with CMU.  I can attest to the incredible work he’s been doing for years at Moderna.  Andrew was the first member of Moderna’s Data Science team and he has been at the forefront of Data Science and AI at Moderna for years.

His involvement hasn’t stopped with this current Generative AI revolution. He played a crucial role in quickly helping all of Moderna start incorporating it into their workflows through a tool he conceived and launched called mChat.

We talk about all of this and where AI is heading in healthcare in this season finale. 

Show links:
LinkedIn: https://www.linkedin.com/in/giessel/
X: https://twitter.com/giessel
AlphaGo Paper:  https://www.pnas.org/doi/epdf/10.1073/pnas.2214840120 

Transcript:
[00:00:00] Sean Ammirati: In this season finale of Agile Giants, I am thrilled to welcome Andrew Giessel, Executive Director of Data Science and Artificial Intelligence at Moderna. Andrew and I first crossed paths when Moderna’s AI Academy was launched in collaboration with CMU. I was one of the co-directors of that program from the Carnegie Mellon side, and I can attest to the incredible work he’s been doing at Moderna.

[00:00:29] Sean Ammirati: Andrew has been at the forefront of data science and AI for quite some time, and his involvement hasn’t stopped with this current generative AI revolution. He played a critical role in quickly helping all of Moderna start incorporating generative AI into their workflows through a tool he conceived and launched called mChat. I’m excited for you to join us in this engaging discussion on Agile Giants. Enjoy.

[00:00:50] Sean Ammirati: So, Andrew, thank you so much for joining me today. Before we get to Moderna, it’s probably worth giving just a little bit of your background and your context. Could you talk just a little bit about your academic and professional background before coming to Moderna?

[00:01:08] Andrew Giessel: Yeah, sure, and thanks for having me. So I’ve been at Moderna for almost eight years. It’ll be eight years on Leap Day, which I joked with someone. And does that mean I’ve been there for two years or eight and it’s like, definitely been eight years. So before that, um, I worked at a friend’s tech startup.

[00:01:34] Andrew Giessel: It was very small as me and two other folks basically. And that was on like mobile phone sensors. And then before that I was a postdoc, an academic, so I did my Ph. D. and my postdoc in neurobiology at Harvard Medical School. Worked in mice mostly, did experiments that were like a combination of electrical recordings and videos through a microscope.

[00:01:58] Andrew Giessel: So I always had to use computers to analyze my data. And that’s kind of, I guess, where I started turning to a data scientist. Before grad school, I was a computer science and biochemistry major. I did both. I went into college as a computer science major and I’m studying biology as well.  So I’ve kind of done both.

[00:02:23] Andrew Giessel:And when I got Moderna, I felt like it was a great, great place for me because there’s this mix where I knew my tools really well, but I could also have a lot of deep and meaningful conversations at a technical level with a lot of my collaborators that were biologists or chemists or whatever else.

[00:02:47] Sean Ammirati: Right. So Moderna today, but certainly probably just for people remembering eight years ago. That’s a very small company that had not gained brand recognition. So, you’ve been there. What employee number were you at Moderna?

[00:03:02] Andrew Giessel: I, well, my class contained employee 300. At that time, there were like handfuls of people joining every week.

[00:03:12] Andrew Giessel: In general, the company is not exactly doubling every year, but almost. By the time we got to the pandemic, we were around 700 people. That’s how big we were when the pandemic started, which is kind of crazy when you compare us to, say, Pfizer, who has 100,000 employees. It’s like a really different sort of game.

[00:03:37] Andrew Giessel: So yeah, I think the early stages of Moderna were really, there was a lot of excitement about the technology. I joined because of the basic premise of Moderna, which is that if you can get mRNA to work once, you basically can get it to work over and over again because the information is basically the drug and not the entire product. The product is basically physically the same, but the information is different. Which is really compelling. 

[00:04:12] Sean Ammirati: Yeah. It’s an amazing vision. And I think it kind of leads to the other part of context setting before we get into what you’re doing, right? You would have to have literally been living under a rock to not know the name Moderna today.

[00:04:26] Sean Ammirati: But they may not be as familiar with the data science and AI part of Moderna. You’re starting to speak to that, but just a moment on even in those early days. Why do they have a data science team like yourself? And what does data science and AI mean at Moderna?

[00:04:42] Andrew Giessel: Yeah, broad question. I was the first data scientist we hired. We had a couple of other computational people at the time, but they were more traditional bioinformaticists or computational chemists. Data science has always been there, I guess, since I’ve been there; it’s kind of obvious, maybe.

[00:05:08] Andrew Giessel: But, now we probably have a community of 20 to 30 data scientists. The notion of mRNA as a platform has a lot of consequences for the company as a whole. If you have the same raw materials to make DNA, to make RNA, then you’re probably going to run into some of the same problems over and over again, no matter what the product is. It’s still like you’re still making DNA, you’re still making RNA. So, that motivated us to invest in digital technology and data science from an early stage because it pays off. I have code that I wrote in 2016 that was used in a small way to help design the COVID vaccine. It wasn’t written for the COVID vaccine; it was written for making DNA plasmids generally. But then it gets reused, and I get this claim to fame. My teams over the years have had stories like that over and over again where we do it initially for one problem, but it was prioritized or we were excited about it from a value perspective because we knew that it would be used over and over again.

[00:06:33] Sean Ammirati: It’s fascinating. I just would say that beyond that 20 or so in your community, there’s always been this vision of pushing these tools out to everybody in the organization. I want to click on the generative AI stuff that you’re doing specifically and mChat.

[00:06:56] Sean Ammirati: So, maybe to start with, what’s the idea behind mChat? How has it affected workflow use since it launched in May of last year?

[00:07:05] Andrew Giessel: I’ve not been a part of a technology that’s moved so quickly and had such a sweeping change, generally in the tech world and society, but also at Moderna. mChat was an internally deployed version of chat GPT, basically underneath the hood. It used OpenAI’s APIs to talk to their large language models, but we were able to expose it internally from a security point of view. This is really important because at the time, OpenAI didn’t have a good way for you to guarantee that your conversations wouldn’t be used in future training of the models. Everything over API was guaranteed to not be included in training. That was a really big deal for us as a business to make sure that our data was secure and safe. So, we launched it internally, and it just took off with a life of its own. In parallel to chat GPT, GPT-4 was really capable, and mChat came on the backs of that. 


[00:08:49] Andrew Giessel: We basically use GPT-4 by default now, and 75 percent of the company has used it. 50 percent of the company are monthly active users, and we only give devices out to 75 percent of the company. Like 25 percent of the people on the manufacturing floor don’t get phones or laptops. So essentially it’s done, complete penetrance, and people use it all the time.

[00:09:11] Andrew Giessel: You walk through the office, you see it up on screens, and it’s pretty remarkable. The team we had was super small, like four or five software engineers. Initially, it was just me. Well, I was happy that we got some engineers in before it really took off.

[00:09:34] Sean Ammirati: Everybody’s been trying to figure out what to do with large language models and generative AI. But a lot of orgs don’t have the same, you know, adoption rates that you just rattled off effectively 100 percent trying it out, 100 percent of possible audience trying it out and two-thirds using it because it’s 75 and 50 but 75 is 100. What do you attribute to this uniquely significant adoption of mChat inside Moderna?

[00:09:59] Andrew Giessel: Yeah, I think there’s an atmosphere of excitement and just trying it out. We hire people that would probably be likely to share it out, or at least we try to. I think the other is that I have a counterpart partner, Brice Challamel,, who’s head of AI transformation, and he’s just a dynamo and has done extensive outreach.

[00:10:33] Andrew Giessel: I’ve been with him at a lot of these town halls where we talk about it. We’ve thought about different personas and the uses that they have. And I think that’s been a huge part of it. He also established a group called GACT, the Generative AI Champions Team, which is GACT are also the letters of DNA. So it’s very clever. These are essentially power users across the company. They span all different business units. They tend to skew younger and more junior, but these people are users and constantly surprise us with all the different stuff that they do.

[00:11:19] Andrew Giessel: I think building that sort of community, the team’s site, is internal. There’s just a lot of outreach effort to make it click for people. And I think it takes some of that to make it sink in. But once it does, I find that people, it’s like a phase transition. They’re suddenly like, I couldn’t live without this tool in my part of the company. I’m pretty close to a lot of folks on that program.

[00:11:50] Andrew Giessel: I think it’s used for programming is amazing. We use both in chat, but then also we have an enterprise GitHub account and use GitHub’s CoPilot, which has a different flavor. It’s more like the best autocomplete you could hope for. Whereas in chat is more like, I have this bigger problem. Can you, or this code doesn’t work. Can you walk me through it? It’s more conversational. Those are two definitely two different styles. But they’re both generative AI and both things that we have rushed towards.

[00:12:38] Andrew Giessel: CoPilot has been out for a while, but it was actually a change in their terms of service, where they guaranteed that they wouldn’t use any information sent to them to improve their models that triggered us to be able to pick it up. And one of my data science leads and I did a back of the envelope calculation for our data scientists about how much time that tool saved, and we estimated roughly an hour a day per data scientist, which is crazy.

[00:13:11] Andrew Giessel: I think it really varies how much you’re in the trenches doing individual contributor work. But when it works well, it really feels like you’re flying. And I think programming is a really good sort of, I mean, it’s a really killer app for it. In part because, and we’ve talked about this before, this technology really shines when it is faster to verify a solution than it is to crank out the first draft of it.

[00:13:40] Andrew Giessel: And when you, and programming, when you code, you kind of have a little code simulator running around in your head all the time. That’s like part of the essence of programming, I think. And so people, developers are very much in that mode where they are looking at their code and wondering why it doesn’t work. They can look at someone else’s code and wonder if it works or not, but it’s pretty effective there and it feels like you’re flying when it works well.

[00:14:11] Sean Ammirati: Yeah, I think the other part of it is when it hallucinates, it internally can compile and see that it hallucinated, which is probably a helpful mental model for other creation activities. How do you have some type of closed-loop check on hallucination? Like if it just makes up a function and the code errors, you know, CoPilot from GitHub can actually see that happened. Whereas, if you’re writing an email and it makes up a date, that’s harder out of the box to test for hallucination.

[00:14:47] Andrew Giessel: Yeah. I think one of the remarkable things I’ve found and one of the intermediate tips that we have given people across the company is to iterate with these models. There’s a study looking at the kind of one-shot performance of code generation of GPT-4. If it didn’t work and you just pasted the error back in, you got like a 20 percent jump in overall accuracy. I think that’s really remarkable. It’s a good metaphor, and people can struggle to do analogous things in these non-quantitative outputs. That loop of going back and forth with the model is really a different way of using a computer.

[00:15:56] Andrew Giessel: I’ll give one example, which is I often have ChatGPT generate a unique competitive landscape. Here’s a startup, help me understand the competitors around it. The danger is it will make up startups that aren’t real with credible names and URLs. I’ve written Python code that takes that URL, loads it into a web scraper, and compares if the text that loads is similar to the text expected. GPT-3.5 had about 50 percent hallucinations, GPT-4 is like 8 out of 10 are good. It makes the two out of ten even harder because now you really got to be paying attention. It’s like a similar analogy to the compile, right? You have, you can also ask GPT to respond as JSON, so it’s pretty easy to take that and loop through it.

[00:17:56] Andrew Giessel: I had a similar experience trying to use it to research topics and ask for papers on a topic. I got incredibly relevant papers, titles that made sense, journals, all fake. These models aren’t trained to be right; they’re trained to be likely. It’s a very subtle difference. They are not a guarantee.

[00:18:47] Sean Ammirati: My brother-in-law’s on the faculty at Penn in their biomechanical engineering department. He got an email from a student looking to join his lab citing all the stuff he had done. Two of the funny five studies were completely made up, but exactly the kind of thing he would have done. It’s a weird world, and people are still figuring out the likely versus correct aspect.

[00:19:12] Andrew Giessel: Yeah. These models are so capable in many ways that it makes me reconsider certain aspects of the way I think or the way my job works. If a model can generate text for an email, memo, or PowerPoint, and it’s great, did I need to be writing that? I don’t know. Also, in terms of how much of my day-to-day life is. Contingent on having good mental models and acting on those mental models when I don’t have the evidence, kind of like in a Bayesian sense, having good priors, right?

[00:20:09] Sean Ammirati: 100%. That paper we were texting about this weekend with the AlphaGo thing, I’m still internalizing that. The way it changes what’s possible is really interesting. It’s the future of work.

[00:20:28] Andrew Giessel: Definitely. The paper analyzed 70+ years of Go games, found a shift in the quality and novelty of moves after AlphaGo. It’s like people are learning a different way of thinking about the game from these models.

[00:21:11] Andrew Giessel: I think having it as a study tool to see things you couldn’t see before or someone pointing out something to you is really interesting. The novelty of moves increased after the model came out.

[00:22:34] Andrew Giessel: mChat was meant to enable exploration and go where the future is heading. The landscape is changing, and it would be disconcerting to be an entrepreneur in this space. mChat focuses on high-value Moderna-specific applications and enabling data scientists and developers internally.

[00:23:54] Andrew Giessel: We are excited about ChatGPT Enterprise, which is similar to mChat but allows us to focus on high-value Moderna-specific applications. The future of AI transformation at Moderna doesn’t go away as models improve. The ability to double-check yourself and understand when things are too good to be true, especially in data science, becomes crucial as these capabilities are unlocked for more people.

[00:26:45] Andrew Giessel: A thin layer on top of these models is probably not defensible. The real moat is the models, and they cost a lot to train. A well-designed product is essential. It’s about how you use the models in a way that the rest of the field is clueless about.

[00:28:35] Andrew Giessel: Text as a substrate has been unlocked by these models, particularly in domains like clinical trials and regulatory filings. The ability to process large volumes of written text, like BLA filings, can significantly speed up processes while maintaining regulatory standards.

[00:29:51] Andrew Giessel: And if you can be as sure, but do it half as fast or twice as fast, then. Great. You get vaccines into people sooner. You save a ton of lives. And so that Moderna as a company is kind of entering this time where, you know, we have one commercial product and we’re in the late stage clinical trials for several more.

[00:30:15] Andrew Giessel: And a lot of what’s between our science and like affecting human health is clinical trials and getting that process through there. So I think, this is an example of a domain that data scientists at Moderna have a real opportunity to be able to use this new technology a very specific sort of way, very high value, sort of use cases, to make real, real impact and get drugs out to people sooner.

[00:30:47] Sean Ammirati: First of  all, as every time when I talk to you, like the mission part of Moderna rang through there and that was, that was awesome. And I think that is unique to the culture there, but leaving that part aside, I think generally for entrepreneurs, the journey that your team is going through with them, is probably the journey they’re going to need to go through as well.


[00:31:05] Sean Ammirati: A slightly better way to send text to the OpenAI API may not be a future selling point. However, more job-specific uses and data-specific applications could pave the way for companies to explore unique opportunities.

[00:31:37] Andrew Giessel: When people ask about the most important thing in an AI, email, or data science project, the answer is almost always data. Despite the promises of going from unstructured to structured data, it remains a challenging problem. Solving this problem can provide substantial value, especially if it involves structured unstructured data relevant to a specific business domain.

[00:33:44] Andrew Giessel: Leaving generative AI aside and focusing on data science and AI broadly, big trends to follow include the continued exploration of LLMs, increased focus on explainability and fairness in healthcare applications, addressing biases, and advancements in drug discovery. These technologies are expected to impact various aspects of the healthcare system in the coming years.

[00:36:08] Andrew Giessel: To stay updated on my work and interests, Twitter is my most active platform, and you can find me using my last name, Kiesel. I also share a lot of science and computer science information there. Additionally, you can connect with me on LinkedIn for more business-related updates.

[00:36:56] Sean Ammirati: Thank you, Andrew, for joining us today. I really appreciate it. For those following along with the entire season, thank you for your support. I encourage you to check out all the episodes available on your preferred podcast platform or YouTube. I’ve enjoyed these conversations, and I’m looking forward to sharing more insights in future seasons. Stay updated on my projects and activities by following me on Twitter or LinkedIn. We’ll be back with another season of Agile Giants soon.

Join the discussion

More from this show

Subscribe

Episode 2