PODCAST
Building apps with a new generation of language models
Amber Teng on her GPT-3-powered cover letter generator
Editor’s note: The TDS Podcast is hosted by Jeremie Harris, who is the co-founder of Gladstone, an AI safety startup. Every week, Jeremie chats with researchers and business leaders at the forefront of the field to unpack the most pressing questions around data science, machine learning, and AI.
It’s no secret that a new generation of powerful and highly scaled language models is taking the world by storm. Companies like OpenAI, AI21Labs, and Cohere have built models so versatile that they’re powering hundreds of new applications, and unlocking entire new markets for AI-generated text.
In light of that, I thought it would be worth exploring the applied side of language modelling — to dive deep into one specific language model-powered tool, to understand what it means to build apps on top of scaled AI systems. How easily can these models be used in the wild? What bottlenecks and challenges do people run into when they try to build apps powered by large language models? That’s what I wanted to find out.
My guest today is Amber Teng, and she’s a data scientist who recently published a blog that got quite a bit of attention, about a resume cover letter generator that she created using GPT-3, OpenAI’s powerful and now-famous language model. I thought her project would be make for a great episode, because it exposes so many of the challenges and opportunities that come with the new era of powerful language models that we’ve just entered.
So today we’ll be exploring exactly that: looking at the applied side of language modelling and prompt engineering, understanding how large language models have made new apps not only possible but also much easier to build, and the likely future of AI-powered products.
Here were some of my favourite take-homes from the conversation:
- It took Amber just 5 hours to build and deploy her cover letter generator end-to-end. This includes time reading OpenAI’s documentation on GPT-3, and time spent engineering prompts for the model. It’s incredible how fast new apps can be built with today’s language models, and that ease of development is a big reason why so many new startups are being created, whose core product offering is basically just a thin wrapper around GPT-3.
- Amber found that a prompt engineering strategy that worked well for her was to start with a very general and nonspecific prompt (e.g. “Write a cover letter for a machine learning job at Amazon.”) and see how GPT-3 handles it, before tailoring the prompt with more specific content to tune the model’s response. That way, she could increase the complexity of her prompts in a controlled way, ensuring that she understood roughly what role each component of the prompt was there to play.
- For especially complex and specific prompts, Amber found that GPT-3 would often repeat prompt content in its completions. For example, when given a prompt like, “Write a cover letter for Amber Teng, a data scientist with a CS degree from Brown University, for a machine learning engineer role at Google. Indicate that Amber is interested in the role because it involves building integrations for fintech startups, and a lot of finance-related visualization work that she’s had experience with from her time working in financial services companies,” GPT-3 might directly copy/paste relatively large portions of the prompt in the text it generates, offering essentially no benefit. Amber found that increasing the model’s temperature parameter would often help with this, but that it remained a problem for longer prompts.
- The potential malicious applications of AI are getting harder and harder to ignore, and Amber took some time to think about what they might look like for her cover letter generator. Surprisingly, she came up with at least one significant one: companies might choose to flood competitors’ recruitment pipelines with thousands of AI-generated cover letters and resumes, making it impossible for them to identify and follow up with legitimate candidates. It’s not obvious how OpenAI could detect this kind of malicious strategy, either, since cover letters seem like fairly innocuous content, and even requests for a large number of cover letter generations wouldn’t be inconsistent with what you might expect from a totally legitimate cover letter generating startup with significant traction.
Chapters:
- 0:00 Intro
- 2:30 Amber’s background
- 5:30 Using GPT-3
- 14:45 Building prompts up
- 18:15 Prompting best practices
- 21:45 GPT-3 mistakes
- 25:30 Context windows
- 30:00 End-to-end time
- 34:45 The cost of one cover letter
- 37:00 The analytics
- 41:45 Dynamics around company-building
- 46:00 Commoditization of language modelling
- 51:00 Wrap-up
