A Story Told by a Machine
The Circuitous Path to AI Writing
By Dirk Knemeyer and Jonathan Follett
[Illustration: Saint Jerome Writing by Hans Springinklee German, 1522 woodcut, National Gallery of Art, Open Access]
You may not be aware of it, but AI is already writing stories that you’ve probably read. For years, stock blurbs have been written by machines. More recently, minor league sports game summaries have been as well. Are feature-length stories and novels next? At the University of California Santa Cruz, AI researcher Snigdha Chaturvedi and her team are devising the Natural Language Processing building blocks that will someday enable machines to tell long form narrative stories. While at present the technology is impressive, it remains a long way away from that goal. Through their research we can begin to better understand how natural language generation technology will become part of our lives.
[Photo: “Base hit” by Chris Chow on Unsplash]
Language is a Mirror
If we were to attempt to mark the progress in AI writing so far, we’d find ourselves at the uncanny valley stage — at least when it comes to stories more complex than stock blurbs or baseball stats. Like the nearly human-looking robots whose weird aesthetics repulse people, AI writing can similarly present as not-quite-human, rough, and sometimes odd or off topic. There are a number of reasons why language is such a challenging area for AI research.
“Languages are mirrors of the societies that we live in,” says Chaturvedi. “So, the problem is that when computer scientists design algorithms that attempt to understand language, they don’t often considered the social aspect of language understanding. Most of the things humans say or do are defined by two criteria: their personalities and their relationships with others.”
“The kind of relationships between people plays a very important role in explaining how they behave or what they say,” Chaturvedi explains. “If you understand relationships, you [are better able to] explain human behavior. And since language is simply a reflection of the human world, this would help in understanding language.” All of these factors relate to context — something that artificial intelligence struggles with, at least in its current instantiation. AI can be ruthlessly effective in specific areas of expertise — such as breaking down games like chess, Go, and poker. In these scenarios, questions of context are almost irrelevant. AIs that squash these competitive strategy games typically don’t even consider the opponent. It’s just a matter of looking at the snapshot of the moment and acting in the mathematically optimal way. This is hard to do from a computational and strategic standpoint, operating at a level that the human brain essentially cannot achieve. However, humans excel at understanding context. And it is in this area that AI must advance if it is to realize the potential its champions imagine for it.
“There has been some work on modeling or understanding human relationships, but that has been limited,” says Chaturvedi. “There were two main challenges in this work. The first was going beyond the predominant notion of binary relationships used in most prior scientific work in natural language processing. In this paradigm of binary relationships, you assume that relationships can only be of two types. For example, people who work on social media data such as Facebook assume that users are either friends or not. So there are only two possibilities. Hence, the binary nature of these relationships. They don’t assume a more nuanced notion of human relationships. And this is because developing this nuanced notion is more challenging from a computational perspective, from a natural language processing perspective. But if you think about real world relationships, they are not binary. Real social relationships can have multiple facets. For example, two people can be friends or family members or they could be in a romantic relationship. They could be formal or informal relationships. People could be in a supervisor-supervisee relationship, and so on,” says Chaturvedi.
Binary is seminal to how engineers developed computing technologies. Thinking in ones-and-zeroes is where everything began, in a certain way. Binary patterns continue to infiltrate the way software behaves, such as in the coarse framing of relationships Chaturvedi identifies in Facebook. The real relationships, where some sense of meaning and understanding lie, are much more nuanced, which gets back to the difficulty of the problem: Not only are our various relationships nuanced and non-binary, the relationships of each person, with their unique galaxy of relationships, can differ wildly from the next person. A one-size-fits-all approach is unlikely to satisfy anyone. True understanding requires, at a minimum, context and nuance of the larger world of relationships and, separately, of a person’s smaller and unique world. And there are other wrinkles, as well: “Another major challenge of this work was incorporating the evolving or changing nature of human relationships. If you think about it, human relationships don’t remain constant, they change with time. For example, people who you were friends with when you were seven year old might be very different from people who you are friends with now. And this is even more important in analyzing stories, because the change in relationships between various characters is what makes the stories interesting,” says Chaturvedi.
A Challenge of Context: How Does This Story End?
So, how do you begin to address context in storytelling for AI? One research area Chaturvedi and her team took on was the challenge of helping a machine identify a good ending for a story. “The technical problem that we were trying to address is that you’re given a story and two ending options, but only one of the two options is correct or more logical while the other one is simply a bad ending. For a human, this would be a super easy problem to solve,” says Chaturvedi. “And humans are really good at identifying which of the two endings is more sensible because the other option, in most cases, is simply nonsensical. In this project, we wanted to see how easy it would be for a computer system or an AI algorithm to identify the sensible option. If the computer could solve this problem, it would mean that it has some understanding of text coherence and common sense knowledge and what makes tech sensible, and so on. So, it turned out that it was not an easy problem to design such an AI system.”
This is an obvious building block toward, first, machines developing comprehension to understand stories in some meaningful way, and then potentially developing their own stories down the line. Needless to say this has massive implications for writing professionals. “So this was a challenging research problem because identifying the good or the sensible ending requires understanding what makes a piece of text like a story consistent or sensible,” says Chaturvedi. “And if you think carefully about it, this is a very open domain problem. For example, it would be difficult to hire someone who would be able to write all the rules that tell us what makes a piece of text coherent. This is possibly because we, as humans, don’t even know what factors we consider when we judge a piece of text is coherent or not.”
To evaluate coherence, Chaturvedi and her research team considered three factors. First, the ending of the story needed to follow a logical sequence of events. Second, the completed story had to have a sensible sentiment trajectory — what happened to the protagonist had to make sense. And finally, the ending needed to be consistent with the theme of the story. “We presented a system that could stitch together all these three factors … and judge if an ending was consistent with the story,” says Chaturvedi. “I was very excited about this project, because it gave me a chance to think about text coherence and stories in general. And it motivated me to work on a more challenging problem of designing AI systems to generate entire stories.”
Chaturvedi describes this next step in the team’s research: “We are trying to make a machine collaborate with humans to generate a short story. So, essentially, the story is being generated by the machine, but the human is collaborating with it or guiding it to generate a particular type of story. Well, automatic storytelling in general is a really challenging task from a research perspective. It requires the AI system to not only speak good and grammatically correct English sentences, but also they have to tie together to form a coherent story. The generated story needs to be thematically consistent. It should have an interesting plot, characters with goals, rules, relationships, and so on. The story should have an emotional flow. It should have specific moral values, certain types of endings, and so on. In other words, story generation requires a lot of planning and thinking. And these challenges have attracted the attention of AI experts for a very, very long time.”
“Recently, there has been a renewed interest in storytelling, especially within the field of natural language processing with the advent of a certain type of learning algorithms which are called deep learning algorithms. The problem with these newer, fancier algorithms is that they are less controllable than traditional natural language processing generation systems. In our work, we want to have more control in the store regeneration process by bringing a human user in the loop. More specifically, the user would tell the system that they want to see something specific happening next to the story, and the system would have to generate the story one sentence at a time. This is similar to the interactive storytelling idea that was very successful in Bandersnatch, which was a Black Mirror episode that aired on Netflix.”
AI and the Great Computational Novel
In 2016, Hitoshi Matsubara of Future University Hakodate led a team that developed a novella in collaboration with an AI that passed the first round of judging for the Hinoshi Shinichi Literary Award. Judge Satoshi Hase felt the novel was well-structured but other aspects were lacking, such as character descriptions. The novella, called “The Day a Computer Writes a Novel”, was written using the following process according to reporting by Danny Lewis for Smithsonian Magazine: first, the human designers wrote their own novella and distilled it into its basic components: words, sentences, and basic structures. Based on these parameters, the computer used an algorithm to remix a new novella out of the original work. It overlaps with technologies that, for example, suggest autocomplete in our text messaging. Clearly, given this rudimentary approach, we are a long way away from machines competing for major literary prizes. We are more in the tool development stage, a phase where the applications are not just a far cry from output that would dazzle us, but not even functioning in a way that could be particularly useful to an actual trained writer.
[Photo: “Library at Orchard, Singapore” by Fahrul Azmi on Unsplash]
In general, AI systems aren’t good at generating long pieces of text. At their best, they can create a small amount of content that is coherent with existing content that precedes it in a story. However, the more text the AI generates sequentially — the farther away their original text gets away from the set narrative that preceded its own generated content — the less coherent it becomes. It seems more like a smart sentence generator than a writer.
“We are using deep learning-based systems, which are notorious for being very complicated to understand. At a very high level, deep learning models [are a] complex collection of mathematical representations like matrices and vectors that interact with each other to give you the desired output. While this complexity makes them extremely powerful, the mathematical representations make it difficult for even human experts or scientists to decipher how exactly the system is working,” says Chaturvedi. “Now, since we don’t understand how these models work, it becomes more and more difficult to command them to do a specific thing, like instructing them to generate a specific story plot. In our work, we are guiding the model to incorporate human supervision by having a specific and important component dedicated to this information. So, in essence, the main technical challenge is how do you teach these systems to pay attention to what the user is saying, what the user wants to happen in the story? And how do they incorporate this information while they are generating a story?”
That’s another more general limitation — and even risk — presented by deep learning models. We humans can’t understand them or reverse-engineer them. Their power and potential are unlocked by being turned loose on huge amounts of data to create their own models. But it turns the engine into something of a black box that we have limited agency to understand, much less, in certain ways, to adjust. Unlike physical or mechanical systems of the past, or even software more traditionally written by humans, we don’t fully understand what is going on.
Applications of AI Writing
The AI systems currently being used to write stories about finance or sports are limited in nature, at least for now. “There are several companies which are interested in automatically recognizing key events from news stories, press release documents, et cetera, and converting them to a story so that it can be used by a human or an algorithmic reader,” says Chaturvedi. “There are essentially two steps in this process. In the first step, an AI model is trained to automatically extract relevant events from press releases and such documents. Unfortunately, the output [of] these AI systems … are not necessarily in a nice and natural format. It is usually presented as a table that can be easily stored in a database, but that doesn’t necessarily convey the entire information in an easy-to-read manner. So, the second step is to convert these table-like event data into natural-looking English sentences or stories. This is the step [which] current commercial AI systems are not really good at, and a lot of manual effort is required in this process.”
These stories are generated using templates. “You could think of templates as human or semi-automatically generated sentences or short stories with slots. The job of the machine is to take a template, but it has to pick a template which is relevant for the context that it is working in. And then once it has picked the template, its task is to just fill these slots with the relevant material-like events.” These rigid templates often cannot produce smooth, human-like prose: For the time being at least, they cannot, for example, paraphrase. Because of this, such templated AI stories often resemble copy pasted text. “In practice, when these systems output the template-based generated text, a human goes into the system and smoothes out the content,” says Chaturvedi.
AI writing systems have the potential for numerous market applications. “Professional writers could use them to get ideas about their next create work,” says Chaturvedi. “You could imagine more films like [Netflix’s] Bandersnatch, but they would be produced … with help from AI and not entirely written by a human writer. They could also be used to improve existing computer games. The interactive nature of such systems can make them very successful in the domain of training and education. For example, you could use such a system for teaching children about language coherence and storytelling. AI-generated stories could also be used to assist journalists in producing news stories.”
So, while today the state-of-the-art in AI writing is still rough, Chaturvedi sees a path forward where generated text becomes much improved. Does that mean competent novels won’t be generated by an AI anytime soon? “That’s going to take some time. Currently, the systems usually are trying to generate short pieces of text and these texts are limited to a certain type of domain. For generating longer stories, I think we’ll have to wait maybe some decades to have stories that are generated entirely by AI systems and to have novels that are generated by AI systems,” says Chaturvedi. “The problem is that when humans write long documents, like a novel, they have a narrative plan or a story plot in mind for the document. When they write a sentence, it is in context of what they wrote immediately before, as well as what they have in mind for the overall document,” says Chaturvedi. It all comes back to the challenge of context. The machine is not even able to keep in mind the context of the entire story that came before, or even the entire parts of the story that it itself has written.
[Photo: “Weathered books” by Syd Wachs on Unsplash]
However, there is a practical future that Chaturvedi envisions, where this AI research will evolve to provide useful tools for creative professionals, even if novel writing is a long way off. “AI writing will certainly make the job of professional writers much easier. It will not just provide suggestions when something is incorrect, ambiguous, or difficult to read, but it will also provide writers with creative ideas and help them in clearing writer’s block.” One day, maybe sooner than we think, AI-based generation systems will enable amateur writers to express themselves in a professional manner, making it possible for essentially everyone to be a writer. And, in domains like journalism, such AI systems would be able to significantly reduce the workload of journalists, enabling them to focus their attention on gathering facts and newsworthy material. “It would also certainly help them in expediting the process of writing the stories to provide users with more real time news,” says Chaturvedi.
Creative Next is a podcast exploring the impact of AI-driven automation on the lives of creative workers, people like writers, researchers, artists, designers, engineers, and entrepreneurs. This article accompanies Season 2, Episode 4 — AI and Augmented Storytelling.
