Building a Chatbot with Rasa

Getting Started

Image by Author

Chatbots are programs that simulate human conversation. These can range from simple rule-based chatbots, where the user is limited to clicking on buttons or suggested replies that the bot provides, all the way to fully-fledged bots that can handle context, chitchat, and other complex things, which are otherwise very common in human conversation.

Quantitatively, there are 5 levels of conversational AI, starting from simple rule-based systems to really adaptive chatbots, who can handle complex scenarios, like when even the user isn’t sure what they want and can reply to users based on the level of detail they want.

Table of Contents

- Rasa
- Intents, Entities, Slots and Responses
- Setting up Rasa
- Creating training data
- Building our bot -- training and testing

Rasa

Rasa is an open-source framework to build text and voice-based chatbots. It's working at Level 3 of conversational AI, where the bot can understand the context. A level 3 conversational agent can handle things like the user changing their mind, handling context and even unexpected queries.

Rasa is not the only tool available to you if you’re looking to build a chatbot, but it's one of the best. There are several others, like DialogFlow, though we won’t discuss them in this post.

Intents, Entities, Slots and Responses

These are the main components of any chatbot conversation. We’ll follow a sample conversation to better understand these terms. Let’s say we’re building a bot to collect user contact info.

Intents

What the user is implying is called an intent. For example, if a user says one of the following to start the chat:

• Good Morning!
• Hi!
• Hey!
• Hello.

the user is essentially saying a greeting. So we can group these into a single intent called greet . Now, whenever a bot gets a user message that’s similar to other phrases in greet , the bot will classify it as belonging to the greet intent.

Examples of intents — image by author

user: Hi. (bot classifies this message as "greet")

Entities

Entities are pieces of data that can be extracted from a user message.

Continuing our example, after the user greets the bot, the bot asks for their contact information. Something like this:

user: Hi.    (bot classifies this message as "greet")
bot: Hello! Could you please provide your contact information?

The user would then enter their details, like their name and their email id.

user: Sure. It's John Doe. My email is johndoe@email.com.

The message above has two pieces of information — the name and the email. These can be extracted as entities. Your bot will extract them depending on the quality of your training data.

Let’s say you have defined two entities for name and email. This is what the bot will extract.

name: John Doe
email: johndoe@email.com

An important question here is: how does the bot know what to extract? A user could potentially enter any combination of information. All these are valid user inputs:

• My name is John Doe. email: johndoe@gmail.com
• name: John Doe email: johndoe@gmail.com
• Yeah sure. I’m John Doe. My email is johndoe@gmail.com.
• John Doe, johndoe@gmail.com

Humans can easily extract the names and email ids. But how would a chatbot be able to do that?

The answer to this likes in how good our training data is, and we’ll talk about this in a bit.

Slots

Slots are the bot’s memory. Any information that needs to persist throughout the conversation, like a user’s name or their destination if you were building a flight booking bot, should be stored as slots.

Since we already have two entities (name and email), we can create slots with the same names, so when names or email-ids are extracted, they are automatically stored in their respective slots. This is because of two properties that are True by default.

• auto_fill
• store_entities_as_slots

How we’ll set these slots up, we’ll discuss in a little white.

Responses

Responses are what the bot says to the user. Naturally, for a bot to give an appropriate response, it has to figure out what the user is trying to say.

In our example, when the user greeted the bot, it understand the message as such and responded appropriately by saying hello, and asking for their contact details.

Continuing with our sample conversation, after the bot provides their information, let’s say we program the bot to say “thanks” after the user provides their details.

user: Hi.    (bot classifies this message as "greet")
bot: Hello! Could you please provide your contact information?
user: Sure. It's John. My email is john@email.com.
bot: Thanks John for the info!

Some Prerequisites

1. Install Rasa Opensource using pip install rasa==2.6.2

You can install the latest version, but this post is based on v2.6.2, so any v2.x should work perfectly with what’s covered here.

2. Run rasa init to set up the demo project.

Creating Training Data

Returning to our question of how to create bots that can extract useful information in multiple forms.

It's easy for us since we get the semantics of natural languages, but a chatbot can’t do that. A bot doesn’t know what “email” is. Nor does it know that John is probably a name and that emails contain the “@” symbol.

Our goal is to provide varied data so the bot can understand some associations between words, like what follows the word “email” is probably going to be an email id, or that words of the form <some_text>@<some_text>.com would be an email.

It's, of course, impossible to cover every scenario but we can teach a chatbot the most common ones, making sure we add generic words and phrases that may represent a good portion of messages the bot is likely to see.

As you can probably guess, this is more of an iterative process, where we evaluate our bot’s performance in the real world and use that to improve its performance.

Code

We’ll go file by file, to make it easier to follow. For the purpose of this post, there are three files that we would be discussing:

• nlu.yml
• domain.yml
• stories.yml

NLU Training data

Let’s handle the following cases for the supply_contact_info intent.

• My name is John Doe. email: johndoe@gmail.com
• name: John Doe email: johndoe@gmail.com
• Yeah sure. I’m John Doe. My email is johndoe@gmail.com.
• John Doe, johndoe@gmail.com
• Sure. It's John Doe. My email is johndoe@email.com.

This training data is called NLU data, and it houses the phrases and dialogues (intents) we expect from a user. Note that this doesn’t include the bot’s responses or how our conversation flows. That’ll exist in a separate part of our project.

YAML files are used for this purpose.

Adding intents — nlu.yml

Next, we’ll give this intent a name. Let’s call it supply_contact_info .

Replace the contents of nlu.ymlwith this. Varying the names and email-ids themselves is a good idea so the bot can generalize better.

version: "2.0"
nlu:
- intent: supply_contact_info
  examples: |
    - My name is John. email's john@gmail.com
    - name: David email: david@email.com
    - Yeah sure. I’m Barbara. My email is barbara@email.com.
    - Susan, susan@email.com
    - Sure. It's Fred. My email is fred@email.com.

The version key refers to the format of training data that rasa supports. All 2.x versions of Rasa support 2.0 .

Tagging entities — nlu.yml

The NLU data above would give the bot an idea of what kind of things a user could say. But we still haven’t tagged any entities , which as a quick reminder, are key pieces of information that the bot should collect.

First, let’s define some entities. Let’s creatively call the entity that would represent customer names as name . Similarly, for email, we’ll use an entity called email .

The syntax for tagging an entity is [entity_value](entity_name) . So for the line:

My name is John. email's john@email.com

we write it as:

My name is [John](name). email's [john@email.com](email)

Tagging entities — image by author

This way, we end up with our final NLU data for the supply_contact_info intent.

version: "2.0"
nlu:
- intent: supply_contact_info
  examples: |
    - My name is [John](name). email's [john@email.com](email)
    - name: [David](name) email: [david@email.com](email)
    - Yeah sure. I'm [Barbara](name). My email is [barbara@email.com](email)
    - [Susan](name), [susan@email.com](email)
    - Sure. It's [Fred](name). My email is [fred@email.com](email).

We also have another intent called the greet intent, which is when the user starts the conversation and says things like “hi” or “hello”. Finally, nlu.yml should look like this.

nlu:
- intent: greet
  examples: |
    - hi
    - hello
- intent: supply_contact_info
  examples: |
    - My name is [John](name). email's [john@email.com](email)
    - name: [David](name) email: [david@email.com](email)
    - Yeah sure. I'm [Barbara](name). My email is [barbara@email.com](email)
    - [Susan](name), [susan@email.com](email)
    - Sure. It's [Fred](name). My email is [fred@email.com](email).

Adding Slots — domain.yml

Now that we have intents and entities, we can add our slots.

Head to the file domain.yml . Replace the contents of slots key with this.

slots:
  name:
    type: text
  email:
    type: text

Since both name and email are strings, we’ll set the type as text .

Adding Responses — domain.yml

Just as we have intents to abstract out what the user is trying to say, we have responses to represent what the bot would say.

Simple responses are text-based, though Rasa lets you add more complex features like buttons, alternate responses, responses specific to channels, and even custom actions, which we’ll get to later.

In our example, after the user greets the bot (intent: greet ), the bot asks the user for their contact info. This can be represented as a bot response or utterance. By convention, we add an “utter” prefix to each bot utterance — something like this:

utter_ask_for_contact_info

Replace the contents of the responses key in domain.yml with our response.

responses:
utter_ask_for_contact_info:
- text: Hello! Could you please provide your contact information?

Similarly, we can add a response after the user has provided their information. Let’s call that: utter_acknowledge_provided_info (in bold, below).

responses:
utter_ask_for_contact_info:
- text: Hello! Could you please provide your contact information?
utter_acknowledge_provided_info:
- text: Thanks for provided your info!

We could make the user experience slightly better here by mentioning the user’s name along with the acknowledgement — something like “Thanks John for the info!”

To do this, we’ll modify the utter_acknowledge_provided_info above by adding a placeholder for the name slot like this:

utter_acknowledge_provided_info:
- text: Thanks {name} for provided your info!

A sample conversation — image by author

Stories

Stories give our bot an idea of how the conversation should flow. Returning to our example, where the bot asks the user for their contact details, the conversation went something like this:

user: Hi.    (bot classifies this message as "greet")
bot: Hello! Could you please provide your contact information?
user: Sure. It's John. My email is john@email.com.
bot: Thanks John for the info!

This can be broken down into intents and responses like so:

intent: greet
action: utter_ask_for_contact_info
intent: supply_contact_info
action: utter_acknowledge_provided_info

Adding Stories — stories.yml

Let’s turn this into a syntax rasa can understand. Replace the contents of the file stories.yml with what’ll discuss here. Let’s call the story “user supplies customer info”.

Story names do not affect how the bot works.

version: "2.0"
stories:
- story: user supplies customer info
  steps:
  - intent: greet
  - action: utter_ask_for_contact_info
  - intent: supply_contact_info
  - action: utter_acknowledge_provided_info

But there’s one thing left.

We also have to indicate what entities a user intent will likely provide, so it's easier for the bot to figure out how to respond.

Under an intent, just list the entities that are likely to appear in that intent.

version: "2.0"
stories:
- story: user supplies customer info
  steps:
  - intent: greet
  - action: utter_ask_for_contact_info
  - intent: supply_contact_info
    entities:
    - name
    - email
  - action: utter_acknowledge_provided_info

Building our Bot

Now that we have our data and stories ready, we’ll have to follow some steps to get our bot running.

Code Setup

We made a few changes to the files in our demo bot. Let’s recap.

nlu.yml — houses the NLU training data for our model

In this file, we keep the tagged data for our two intents:

• supply_contact_info
• greet

stories.yml— gives the bot an idea of how the conversation should flow

Our single story is located here.

domain.yml — the complete info of all intents, responses and entities

This file is a little more complex than the other two. If you look at the domain.yml file provided in the demo bot (post running rasa init ), you’ll notice keys like intents , actions , responses , entities ,etc.

We’ve already added slotsand responses to our domain.yml . All we need to do now is mention our intents and entities. Finally, the file will look like this:

version: '2.0'
intents:
- greet
- supply_contact_info
entities:
- name
- email
slots:
  name:
    type: text
  email:
    type: text
responses:
  utter_ask_for_contact_info:
  - text: Hello! Could you please provide your contact information?
  utter_acknowledge_provided_info:
  - text: Thanks {name}, for the info!

Training the bot

Validating the data

Before training the bot, a good practice is to check for any inconsistencies in the stories and rules, though in a project this simple, it's unlikely to occur.

$ rasa data validate

The truncated output looks like this. No conflicts were found, so we are good to train our model.

The configuration for policies and pipeline was chosen automatically. It was written into the config file at 'config.yml'.
2021-09-12 18:36:07 INFO     rasa.validator  - Validating intents...
2021-09-12 18:36:07 INFO     rasa.validator  - Validating uniqueness of intents and stories...

  ...

2021-09-12 18:36:08 INFO     rasa.validator  - No story structure conflicts found.

Note: You may get the following warning:

UserWarning: model_confidence is set to `softmax`. It is recommended to try using `model_confidence=linear_norm` to make it easier to tune fallback thresholds.

This is just a recommendation, and can be ignored.

Training

To train the bot, we simply use the rasa train command. We’ll provide a name to the model for better organization, but it's not necessary.

$ rasa train --fixed-model-name contact_bot

The output for this is a lot bigger, so I won’t be displaying it here.

Note: You may get the following warning:

UserWarning: Found a rule-based policy in your pipeline but no rule-based training data. Please add rule-based stories to your training data or remove the rule-based policy (`RulePolicy`) from your your pipeline.

We won’t be discussing rules in this post, but they are essentially what they sound like. They require something called a RulePolicy, which is by default, added to your bot pipeline. Can be ignored for now. This will be discussed in a future post.

Chatting with our bot

Chatting with our new bot is simple. Open a new terminal window and start a rasa shell.

$ rasa shell

This will let you chat with your bot in your terminal. But if you want to cleaner UI and a little more info like what intents were identified and what entities were extracted, you can use Rasa X.

Just a quick point about Rasa X: It lets you test your bot, fix incorrect responses from your bot, and more importantly, you can share your bot with other people to get a better idea of how it’ll perform in the real world.

To use it, install it in local mode.

$ pip3 install rasa-x --extra-index-url https://pypi.rasa.com/simple

Then, run it.

$ rasa x

Head to “Talk to Your bot” in the menu on the left, and start conversing with your bot.

Below, you can see how our bot performs.

A conversation with our bot in Rasa X — image by author

Note

In the conversation above, apart from the dialogues, you’ll notice some greyed out information. Below each user message, you can see what intent the user message fell into and with what confidence, along with what entities were extracted.

Above each bot message, you can see what action the bot decided to take with what confidence, along with any slots that were set.

action_listen is an inbuilt action that means that the chatbot expects user input.

Sample Code and Resources

All code used in this article is available here:

BlogCode/RasaChatbot · Polaris000/BlogCode

Links

• Rasa
• 5 levels of conversational AI

In Conclusion

Rasa is a great tool for conversational AI. Its flexibility and depth of customization make it a good choice of tool. In this post, the bot we built was a very simple one.

There’s a lot more to it than what I could fit in a single post, like actions, forms, rules, regex, synonyms, interactive learning, config files, pipelines, and so much more. But what has been covered in this post should be enough to get you started.

We’ll cover the rest in future posts. I’ll add links to all future posts here. Hope it helped!

Next parts in the series

Part II: Are Slots and Entities the same?

Part III: Handling Chatbot Failure

Part IV: How do Chatbots understand?

Updates

20.3.2022

Add links to the rest of the series