ARTIFICIAL INTELLIGENCE

A month ago I published this 35-minute-long overview of GPT-3. But I value your time as a reader, so I decided to write a super-condensed 5-minute article. I’ve summarized the main ideas from the longer article: What GPT-3 is, what it can do, and its present and Future impact on the world. Enjoy!
What GPT-3 is
GPT-3 is the third version of OpenAI’s family of Generative Pre-Trained models. GPT-1 and GPT-2 laid the foundations for GPT-3, proving the success of two key hypotheses: Transformers+unsupervised pre-training works fine (GPT-1) and language models can multitask (GPT-2).
GPT-3 is a language model based in the transformer architecture, pre-trained in a generative, unsupervised manner that shows decent performance in zero/one/few-shot multitask settings. It works by predicting the next token given a sequence of tokens and can do so for NLP tasks it hasn’t been trained on. After seeing just a few examples, it reached state-of-the-art levels in some benchmarks such as machine translation, Q&A, and cloze tasks.
GPT-3 was trained with huge Internet text datasets – 570GB in total. When it was released, it was the largest neural network with 175 billion parameters (100x GPT-2). Even today, it’s the largest dense neural net, only surpassed by sparse models like the switch transformer or Wu Dao 2.0.
The most impressive feature of GPT-3 is that it’s a meta-learner; it has learned to learn. You can ask it in natural language to perform a new task and it "understands" what it has to do, in an analogous way (keeping the distance) to how a human would.
What GPT-3 can do
In the paper, OpenAI researchers compare GPT-3 with previous models using standard benchmarks. It showed better performance than previous similar systems, but for some tasks, supervised systems – trained for a specific task – were far better. GPT-3’s main contribution is, in contrast to supervised systems, popularizing a new promising path towards AGI.
Apart from dull benchmarks, OpenAI provided another way to test GPT-3’s language skills. They released a beta API encouraging developers to find new exciting use cases. The API works by inputting text – what’s called a prompt— to the baseline GPT-3, conditioning it to specialize in a particular task. If you input: The woman is walking her dog → La mujer está paseando a su perro. The kid is playing in the park → ____, GPT-3 would know you’re asking for an English-Spanish translation and will be able to do it.
This specialized version of GPT-3 would be different than the GPT-3 of any other user. That’s the power of prompting+meta-learning; without changing the original model, users can make GPT-3 a specialist for different tasks. Baseline GPT-3 doesn’t know how to perform any task, it knows how to learn to do it, which makes it more powerful and versatile.
Here’s a list of what GPT-3 can do, with links (special mention to Gwern Branwen, who did a great compilation of examples):
- Nonfiction: Dialogue, impersonation, essays, news articles, plot summaries, tweets, teaching.
- Professional: Ads, emails, copywriting, CV generation, team management, content marketing, note-taking.
- Code: Python, SQL, JSX, React app, Figma, javascript, CSS, HTML, LaTeX
- Creativity: Fiction, poetry, songs, humor, online games, board games, memes, cooking recipes, guitar tabs, write in your unique style.
- Rational skills: Logic, uncertainty, common sense, analogies, concept blending, counting, anagrams, forecasting.
- Philosophy: Meaning of life, number 42, responses to philosophers.
GPT-3’s present and future impact on the world
Crazy hype
GPT-3 changed the AI landscape radically in both the Industry and Academia, and people went crazy about it. Some ascribed human characteristics to the system, others built products and companies on top of it, it made headlines everywhere, and groups of researchers started building similar AIs. Here are some examples of the hype:
- Attributions: People said GPT-3 was "self-aware," a "general intelligence" (or at least a "mindless path" to it), and capable of "understanding [and] reasoning."
- Magazines: It made headlines on The New York Times, Forbes, MIT Technology Review, Wired, The Guardian, Fortune, The Verge, Tech Crunch, Digital Trends.
- Startups: Viable, Fable Studio, Algolia, Copysmith, Latitude, OthersideAI, Debuild, Rezi, Totallib, Broca, Think Confluent.
- AI successors: Switch Transformer, DALL·E, CLIP, UC², LaMDA, MUM, Wu Dao 2.0.
Potential dangers
GPT-3 is an amazing AI but, like any other powerful technology, can be used maliciously. These are some of the many ways it can cause harm:
- Biases: OpenAI found GPT-3 was biased by race, gender, and religion, probably reflecting biases in the training data, for which the system was strongly criticized. OpenAI has recently published PALMS, a method to reduce GPT-3 bias using small curated datasets.
- Fake news: Because GPT-3 writes so well, it can write false articles that can pass as human-made, as blogger Liam Porr and The Guardian proved. OpenAI highlighted in the paper that human judges could only identify 52% of GPT-3 articles, slightly above mere chance.
- Environmental costs: Training GPT-3 generated roughly the same amount of carbon footprint as "driving a car to the Moon and back." The "bigger is better" trend should only continue if the environment isn’t at stake.
- Unusable information: GPT-3 isn’t accountable for the words it spits out. This low-quality data is accumulated on the Internet, making it, in the words of philosopher Shannon Vallor, "increasingly unusable and even harmful for people to access."
- Job losses: Systems like GPT-3 threat computer-based, non-routine cognitive jobs even more than robotics threats blue-collar jobs. Current reports estimate that around 40–50% of all jobs could be replaced in 15–20 years.
Critiques and debates
- Critiques: After the wild hype, people started looking for GPT-3’s weaknesses. The system was found lacking logic, common sense, and text understanding. Gary Marcus wrote a well-argued critique of the system: "[GPT-3] comprehension of the world is often seriously off, which means you can never really trust what it says."
- Counter-critiques: Gwern made good arguments against most critics following a single premise: "Sampling can prove the presence of knowledge but not the absence." He found that most of GPT-3’s failures were due to people’s defective prompts.
- Practical and philosophical debates: The practical debate concerns GPT-3 limitations: We can’t know with certainty when it’ll answer correctly and when it won’t, which makes it unreliable. The philosophical debate concerns the path towards AGI: GPT-3 – or any other computer-based AI – may not be able to ever reach AGI, unless they "grow up, belong to a culture, and act in the world."
If you want to know more about GPT-3, I recommend reading the complete overview:
Other recommended reading
4 Things GPT-4 Will Improve From GPT-3
GPT-3 Scared You? Meet Wu Dao 2.0: A Monster of 1.75 Trillion Parameters
_Travel to the future with me for more content on AI, philosophy, and the cognitive sciences! If you have any questions, feel free to ask in the comments or reach out on LinkedIn or Twitter! 🙂_