Lessons from Corporate Innovators

Piotr Mirowski on Google DeepMind’s Dramatron: Redefining Creativity in the Age of AI

Play episode

Episode description:

Piotr Mirowski, Staff Research Scientist at Google DeepMind and co-creator of Dramatron shares his insights on the future of human-machine co-creativity especially with all the recent developments in Generative AI.  Piotr’s been exploring this space for years and his thought leadership continues to have a profound impact on my thinking.

In this episode, we explore the brilliance behind the language models driving Dramatron and the innovative use of AI in creative writing. Learn more about the system’s functionality, and delve into specific examples that shed light on the benefits and challenges of co-creativity with AI.

Don’t miss out on valuable insights and perspectives in the realm of creative writing and beyond.

Show links:

Dramatron article- https://arxiv.org/abs/2209.14958
Github page- https://github.com/google-deepmind/dramatron
Theater company- https://improbotics.org
X- https://twitter.com/MirowskiPiotr
LinkedIn- https://linkedin.com/in/piotrmirowski
Website- https://piotrmirowski.com/

Transcription:

[00:00:28] Sean Ammirati: This week’s episode is a real treat. I could not be more excited as we welcome Piotr Miroski, a staff research scientist at Google DeepMind, who brings a wealth of knowledge to the table.

[00:00:49] Sean Ammirati: His research interests are vast, looking at deep reinforcement learning, navigation and robotics, as well as natural language processing. And he wrote a paper that is frankly a huge inspiration for me. He was a co-writer on the paper, “Co-Writing Screenplays and Theater Scripts with Language Models”. It’s been incredibly impactful in my thinking, and I think by the end of this episode, you’ll see why. I hope you enjoyed this week’s conversation on Agile Giants.

[00:01:13] Sean Ammirati: So I talked about this a little bit in the introduction. The system Dramatron to me and the paper that you worked on is absolutely brilliant. But I think what I’d love to know is how did you come up with such an interesting use of large language models?

[00:01:30] Piotr Mirowski: Hello, Sean. And thank you so much for this. Maybe let me start with a bit of an origin story about how this whole project Dramatron and the paper, came to be. The idea started with Richard Evans, who is a research scientist at Google DeepMind, who was also a veteran of video games. He has been working on narrative video games for many years.

[00:01:54] Piotr Mirowski: And so had explored essentially this idea of having systems that generate interactively storieswith players playing the game and thinking about the elements of a story and how to make it work in the simplified context of video game. And then Kory Mathewson and myself, have been working since 2016 with language models on the stage of a theater in the context of improvised comedy.

[00:02:32] Piotr Mirowski: So specifically we looked at using at the very beginning recurrent neural networks and chatbots to have a stage partner that would generate mostly nonsense but we would react to it and improvise and perform and make a show about this. And then we brought it to a much much bigger scale and have performed several hundreds of times.

[00:03:00] Piotr Mirowski: Myself, actually, I’m just back from the Edinburgh Film Festival where we performed with the latest version of our show. And finally, the fourth person of the paper, Jaylen Pittman, is a researcher in linguistics, who has been investigating the various biases that exist in language models. And specifically, he was very much interested in how those language models perpetuate stereotypes.

[00:03:38] Piotr Mirowski: In the context of Dramatron, what was interesting is that it was not only a simple input response modality. We had a system that was generating stories in a hierarchical way, stage by stage. And so those biases would compound over time. And he was interested specifically into looking into that. So the four of us combined into building this, this tool and investigating it and specifically testing it with professional writers working with the film and theater industry.

[00:04:16] Sean Ammirati: And I do want to get to like the chaining aspect of it, but just at a high level, somebody who’s not technical, can you just kind of quickly explain like, okay, so you had, I believe it was 15 authors come in. What did you have them do? How long did they work with the system, that kind of thing for these?

[00:04:39] Piotr Mirowski: Yes, exactly. When we worked on the Dramatron, we actually asked ourselves two questions. The first question was, how can we generate more coherent stories? And the second question was, are automated story generation tools any useful for creative artists?

[00:04:59] Piotr Mirowski: And that’s why we recruited a pool of volunteer participants. Who are all coming from the industry, the film and theater industry, and that included directors, playwrights, screenwriters, actors, producers. Who had a very good experience of what a proper story, as before on the stage or on film, looks like.

[00:05:26] Piotr Mirowski: And we asked them to interact with the tool. And the tool itself relied on this idea that stories have some elements. So we went back to Aristotle. Poetics by Aristotle, so Aristotle defined, in poetics, multiple elements for a dramatic story, the most important one being the plot, a sequence of events.

[00:05:55] Piotr Mirowski: And so, of course, we needed to represent thousands of words generated by a story by something much more succinct, which would be the synopsis. Or even better, a decomposition of a plot into multiple steps or multiple beats. Then, of course, we needed to consider the characters as separate entities.

[00:06:19] Piotr Mirowski: And also consider the general through line of a story. And finally, the locations. So, our system would incorporate all those elements and be able to process them separately. So, the system hierarchical generation starting from the logline, which is this short narrative summary. And the writer would provide that logline.

[00:06:53] Piotr Mirowski: Examples of loglines are two sentences that you can find on the back cover of a book or on Rotten Tomatoes, when you try to find the general summary of a film. And so the writer would provide this logline, and then let the language model generate, iteratively, the title and then generate the list of characters and then generate the plot points and then generate location descriptions and then use all those elements with causal dependencies.

[00:07:29] Piotr Mirowski: So we imagine that the plot depends on the logline and on the characters and the locations depend on the plot elements, but also on the logline to have a general idea of what’s happening. And in the subsequent stage, the system would generate, given a plot, element, a plot element, and the logline and the list of characters would generate the dialogue for a specific scene.

[00:08:03] Piotr Mirowski: So this generation enabled us to allow for long-term sematic coherence. Because we would compress the entire story into just a sequence of plot elements that would fit within the context window of a language model. Right. 

[00:08:25] Sean Ammirati: So just to emphasize a couple of things I think that you said, and I will tell you, just for everybody, we will include afterwards a link to the paper on ARXIV, which even if you don’t read computer science papers, you should go read this one. It’ll give you a sense on kind of what, what we’re talking about today. And also a link to the GitHub page for the Dramatron repository, because there’s a very helpful animated GIF at the top of that GitHub repository that will kind of walk through the flow that we just heard about.

[00:08:55] Sean Ammirati: But just to reinforce two things I think that are important for people to think about here. So one, you’re not having the stories be autonomously created, right? These, these stories are co-written by humans. So it’s humans and your Dramatron system together that are creating this content after the logline that you get from the author.

[00:09:16] Sean Ammirati: And then you’re going as far as you could, it sounded like from the paper, you went as far as you could through that process. In a two hour user study with 15 different writers, is that correct? And how far did people get in two hours? 

[00:09:31] Piotr Mirowski: Yes, the whole point of Dramatron was from the beginning to be  a writing aid that would allow a writer to explore story building at a higher conceptual level.

[00:09:48] Piotr Mirowski:  I think it’s important to look at the context in which the study was done. It was back in 2021 and 2022, so before chatGPT, before people started playing with easily accessible language models provided with a very nice user interface. We wanted to have this writing aid that helps a writer generate not only the continuation of a bit of text and to explore alternatives, but actually to reflect on the consequences of small changes in a plot element upon the dialogue or small changes in the description of a character upon what sort of narratives are being generated.

[00:10:37] Piotr Mirowski: So the tool was designed as this sort of interactive back and forth between going back all the way up to the two sentence summary of a log line, generate things and then modify and rewrite, and then go all the way down to dialogue and then again back to essentially refine the story during the process.

[00:11:03] Piotr Mirowski: Yeah. So what could we done in two hours. Essentially the two hours were very packed. So we do this, essentially, a workshop that would conduct with those participants. First going through what the project is about and obtaining their consent.

[00:11:27] Piotr Mirowski: And then, trying to work out a log line that was actually interesting and relevant to them and working through that process back and forth of generation and rewriting and editing, and commenting on the process. We would typically manage to generate maybe one scene or two scenes at the most and the only time when Dramatron was actually used to produce something tangible was when we worked with one specific writer over 22 hours and that writer generated the scripts of four semi-improvised plays that were later staged and performed on the stage of the Robert Fier theater during the Edmonton Fringe Festival in summer 2022.

[00:12:24] Sean Ammirati: All right. So just to make sure people catch this one writer, 22 hours for screenplays that are kind of partially acted out and then partially improvised at the largest film festival in the US. Is that fair?

[00:12:44] Piotr Mirowski: So it was a theater place and so it was performed live on stage. It was at the Edmonton Fringe Festival, which is indeed the largest, fringe festival in North America. It was in August 2022. And what was really beautiful here was that these were improvised improvisational actors, professionally trained and amazing, hilarious actors who were able to take on anything.

[00:13:15] Piotr Mirowski: So if you had given them a phone book, they would still be able to read it in the right way and the right delivery and to tell a story with this. And so here we were given a script written, edited, corrected, rewritten, by a professional writer who was part of this project.

[00:13:40] Piotr Mirowski: So the conditions of success were all gathered from the beginning and the audience also was on the game. So they knew that the AI was part of a generation process and the premise of a show. It’s a show that has been happening every year, is that the actors are given the script on the night of a performance.

[00:14:05] Piotr Mirowski: So they literally discover the script from inside a craft envelope and they get a second craft envelope with costumes. And here again, you can see the joy on the face of the person who’s going to play the robot and we discover some Lycra costume, something that looks like a lampshade that is going to be on the helmet.

[00:14:29] Piotr Mirowski: Okay, great. Fantastic. We have five minutes to get changed into the costume and to have a very quick browse through the script. And then they are able to read the script on stage in real time, in live, and improvise a bit, but no, I wouldn’t say improvise, but improvise the physical reaction to what is in the script.

[00:14:54] Piotr Mirowski: But the beauty of this whole experiment is that what was generated by Dramatron and rewritten by the writer. The interactive process was good enough to provide a platform for the writers, for the actors. They were able to incorporate elements. From the first half of the performance, which was co-written using that tool, into the second half of the performance.

[00:15:29] Piotr Mirowski: So effectively, like in a good storytelling, there is this kind of story diamond where at the beginning, you add interesting elements and in the second half, you close the diamond by reincorporating everything. And so here the actors were able to, to make sense of everything into a driving to us. It was a beautiful conclusion. 

[00:15:51] Sean Ammirati: That’s amazing. And again, I don’t want to sort of dwell on this user study too much in that, that one experience too much, but just contextually, how long would it have taken that writer to write the same amount of content without a system like Dramatron, roughly? 

[00:16:07] Piotr Mirowski: So based on the discussions we had with the writer, it typically is of the order of days.

[00:16:13] Piotr Mirowski: So these are plays that are written in a very specific context. I don’t think they are published after performances. We are just kept as a repository of past material. But I think it’s a writing process that’s closer to devising or sketch writing where you just throw ideas around and see what sticks.

[00:16:41] Piotr Mirowski: So in that specific case, for that specific use case, Dramatron was a great tool because obviously it was perfectly okay if it was very rough around the edges because then the writer and the actress would make it work. And the expectation would be that not that this would be a play expressing a condensing a lifetime of experience and say something absolutely meaningful and true to the lived experience of a writer, rather it would, be a product that would be turned into a performance by the improvisational actors.

[00:17:29] Piotr Mirowski: And so the actors provided this interface between the output of partially generated by the machine and the audience, injecting their own humanity and interpretation on top of that. 

[00:17:41] Sean Ammirati: Yeah. This is, this is amazing. So. You mentioned earlier, part of the way you did this is with prompt chaining, right?

[00:17:48] Sean Ammirati: And that’s, that’s something that a lot of people, thanks to tools like LangChain now, are very excited about in general. Can you talk a little bit about how that concept made what you were doing possible? 

[00:18:02] Piotr Mirowski: Yes. So, um, So, again, I wanted to emphasize the fact that this study was done in 2021 and 2022. So, before LangChain.

[00:18:12] Piotr Mirowski: Before ChatGPT. And also before instruction based tuning. I think instruction based tuning papers were just being published at the time. So, prompt engineering, it was implemented in the following way, by having examples of, let’s say, logline and title and then feeding the logline written by the writer and using the language model with those few examples to produce the prediction of a language model for that specific title.

[00:18:49] Piotr Mirowski: And when we worked on a list of characters, we had over prompts with examples of logline and then lists of characters. In this case, we relied on two bodies of literature. The first one was Medea, a public domain, translation of Medea from the 19th century. Medea itself being a play by Euripides from the 4th century BC.

[00:19:21] Piotr Mirowski: A very relevant play today in today’s world, as many Greek tragedies are. And the second type of examples was taken from Plan 9 from Outer Space, which is a a screenpla that fell into public domain, and that is obviously much more modern. 

[00:19:56] Piotr Mirowski: Where you start with some given circumstances, a platform, and then there is a series of complications going up to a climax and then a falling action, going down to a resolution, a conclusion, a denouement. And we also worked with Plan 9 from Outer Space with a hero journey type of narrative, to generate, more heroic science fiction.

[00:20:34] Sean Ammirati: Right, so you’ve got these two examples that you’re feeding into the system, and then you’re chaining these different prompts together, again, being done long before tools like LangChain are out there that have made this a popular technique, I think, is the big thing to take away here. Is that fair? 

[00:20:45] Piotr Mirowski: Yes, indeed.

[00:20:49] Sean Ammirati: Yes. Awesome. So in some ways, the fact that you kind of saw around the corner there around this, hey, you’ve got to chain these prompts together to accomplish a lot of these, what I would call kind of co creativity tasks, you know, human and language model together. Maybe if we could just abstract this for a little bit again, most of the audience here are our business users.

[00:21:14] Sean Ammirati: What do you think the work that you did kind of helps people think about generally when they think about benefits and challenges in this kind of co creative activity. 

[00:21:25] Piotr Mirowski: It’s a very good question and I was trying to to synthesize many ideas because I recently gave several talks about AI as a partner or as a stage partner, and this whole concept of co-creation.

[00:21:44] Piotr Mirowski: And I would see the following benefits from bringing a tool, because it’s effectively, we’re only talking about tools, nothing else, and endowing it with some more potential than it actually has. There are many ways of seeing co-creativity with machine. The first one is if you rely on the glitch aesthetic and the limitations of a machine, you can decide to embrace those imperfections.

[00:22:20] Piotr Mirowski: So you consider that the machine is going to generate answers that are imperfect, but you still make it work. So it is a creative challenge for you as an artist. And the second way of working with the machine is to be surprised and to use it, as essentially a generator of ideas, but you wouldn’t necessarily have in the first place and see where this, where this takes you.

[00:22:59] Piotr Mirowski: [00:23:00] So in that case, it goes a little bit beyond this idea of glitch aesthetic and just exploiting the imperfections of the machine. It is rather using the tool as a sort of search engine in conceptual space. And so here we have the writers in the Dramatron study would keep generating again and again, to get new suggestions of titles.

[00:23:27] Piotr Mirowski: And sometimes they would come with their own logline and then use just a title generator of a list of of a plot generator to see what are the possible interpretations that could have been made based on that, that logline. And a third possible collaboration, that derives directly from this is acknowledging the fact that, again, the tool has a sort of limitations.

[00:24:02] Piotr Mirowski: Which is that it tends to revert to tropes. It tends to revert to cliches, which effectively come. The fact that it has been, the tool has been trained on massive amounts of data, but it effectively only generates derivative work from, from a data set. And so the artist can essentially get an idea of what will be the average answer, or what would be the set of average answers.

[00:24:33] Piotr Mirowski: For that specific log line or for that specific list of characters and then, use that as a starting point to, to go one step further and to inject their own personality, their own interpretation, their own ideas. 

[00:24:51] Sean Ammirati: That’s absolutely amazing, and I think, honestly, a lot of executives should be thinking about how they can apply the same things to the creative tasks going on inside their organizations, these three different models.

[00:25:04] Sean Ammirati: I want to step back, just for these last two questions, a little more general than just the Dramatron system itself, and just systems like this. So, I’m going to end on a positive note, but I do have a question I want to start with, which is maybe sort of a concern around this. Which is, how do you think about problems like plagiarism in models like this?

[00:25:26] Piotr Mirowski: Yes, it’s a very good question. Before we even started our study, we identified, thanks to work done by our colleagues at DeepMind, three directly relevant risks and ethical implications for the tool. So the first one, obviously, was bias and possibility of having offensive examples. The second one was The problem of cannibalizing creative economies, and the third one was directly reusing copyrighted data from a training data set, knowingly or unknowingly.

[00:26:01] Piotr Mirowski: And of course, the last two problems are very well related because effectively by reusing work done by artists, we are effectively cannibalizing the resource of income and revenue and destroying the ecosystem of the creative, the artistic ecosystem. Well, in general, the creative ecosystem. So we thought about two mitigation strategies.

[00:26:29] Piotr Mirowski: The first one would be to be extremely transparent from the beginning with the human artists and audiences about the whole co-authorship process so that everyone knows where this information comes from, what generated it in response to which context. 

[00:26:52] Piotr Mirowski: So our mitigation strategy was essentially twofold. First, we would invite the creative artist into the loop throughout the co-authorship process, maintaining clarity and transparency on the origin of that generated text. So that everyone is aware that it was co generated using a machine. And secondly, um, the old tools, plagiarism detection tools that are available that the writer could use, including a search engine.

[00:27:24] Piotr Mirowski: And we would recommend that the outputs are always compared and matched to what already exists in the literature, um, to see that at least the final output that is being presented. It’s not merely reproducing existing data. So these were the mitigation strategies we had when we developed the tool.

[00:27:49] Piotr Mirowski: However, moving forward, it is vital for the industry to be responsible in how it approaches generative AI. And that means that we need to make sure that from now on, the artists  provide consent to be able to participate, with a work in the data, that are used to train those models.

[00:28:16] Piotr Mirowski: Um, so it means that those artists can be opt out, empowered with the right to opt out or and to withdraw or refuse to participate in the generation. We also need to make sure that there is a mechanism to credit the contribution, if they decide to be part of this process. And, that they are fairly compensated. 

[00:28:43] Sean Ammirati: I think that’s very well said. So let me just finish with this question that’s a little bit more on the optimistic side. So you, again, I think you guys really saw around the corner on this co creativity, kind of where we were heading with these tools.

[00:28:58] Sean Ammirati: And so I’m curious, kind of standing where we stand today, if you were to forecast out a couple of years now, how would you think about kind of human and machine co creativity? 

[00:29:11] Piotr Mirowski: I would imagine that the generative AI tools used by my artists for co-creation would effectively serve as a potential first draft, a first rough draft to help them ideate and to explore alternatives.

[00:29:34] Piotr Mirowski: So the writers that we interviewed, the 15 writers participate in our study identified several possible applications for Dramatron, which was essentially to help them do world building, populate, ideas, populate some missing bits, and then generate a very first rough draft.

[00:29:58] Piotr Mirowski: That would later be discarded, but that would help them in the ideation process. So I imagined that was so I see a future for generative AI tools where they’re not used as a final product, but rather as an aid to go very quickly through ideas and to explore possible alternatives.

[00:30:20] Sean Ammirati: I think this concept of rough draft is a great way for all creatives to think about this, whether it’s code or screenplays or business plans. I think that’s very well said. So this has been an amazing conversation. Again, I hope that people who hear this aren’t thinking just about the implications for writing screenplays and and regular plays for that matter. But think about this in general for human machine co creativity going forward.

Join the discussion

More from this show

Subscribe

Episode 61