Implementing a sales & support agent with LangChain
Learn how to develop a chatbot that can answer questions based on the information provided in your company’s documentation
Recently, I have been fascinated by the power of ChatGPT and its ability to construct various types of chatbots. I have tried and written about multiple approaches to implementing a chatbot that can access external information to improve its answers. I joined a few Discord channels during my chatbot coding sessions, hoping to get some help as the libraries are relatively new, and not much documentation is available yet. To my amazement, I found custom bots that could answer most of the questions for the given library.
The idea is to provide the chatbot the ability to dig through various resources like company documentation, code, or other content in order to allow it to answer company support questions. Since I already have some experience with chatbots, I decided to test how hard it is to implement a custom bot with access to the company’s resources.
In this blog post, I will walk you through how I used OpenAI’s models to implement a sales & support agent with in the LangChain library that can be used to answer information about applications with a graph database Neo4j. The agent can also help you debug or produce any Cypher statement you are struggling with. Such an agent could then be deployed to serve users on Discord or other platforms.
We will be using the LangChain library to implement the support bot. The library is easy to use and provides an excellent integration of LLM prompts and Python code, allowing us to develop chatbots in only a few minutes. In addition, the library supports a range of LLMs, text embedding models, and vector databases, along with utility functions that help us load and embed frequent types of files we might come across, like text, PowerPoint, images, HTML, PDF, and more.
The code for this blog post is available on GitHub.
LangChain document loaders
First, we must preprocess the company’s resources and store them in a vector database. Luckily, LangChain can help us load external data, calculate text embeddings, and store the documents in a vector database of our choice.
First, we have to load the text into documents. LangChain offers a variety of helper functions that can take various formats and types of data and produce a document output. The helper functions are called Document loaders.
Neo4j has a lot of its documentation available in GitHub repositories. Conveniently, LangChain provides a document loader that takes a repository URL as input and produces a document for each file in the repository. Additionally, we can use the filter function to ignore files during the loading process if needed.
We will begin by loading the AsciiDoc files from the Neo4j’s knowledge base repository.
# Knowledge base
kb_loader = GitLoader(
clone_url="https://github.com/neo4j-documentation/knowledge-base",
repo_path="./repos/kb/",
branch="master",
file_filter=lambda file_path: file_path.endswith(".adoc")
and "articles" in file_path,
)
kb_data = kb_loader.load()
print(len(kb_data)) # 309Wasn’t that easy as a pie? The GitLoader function clones the repository and load relevant files as documents. In this example, we specified that the file must end with .adoc suffix and be a part of the articles folder. In total, 309 articles were loaded. We also have to be mindful of the size of the documents. For example, GPT-3.5-turbo has a token limit of 4000, while GPT-4 allows 8000 tokens in a single request. While number of words is not exactly identical to the number of tokens, it is still a good estimator.
Next, we will load the documentation of the Graph Data Science repository. Here, we will use a text splitter to make sure none of the documents exceed 2000 words. Again, I know that number of words is not equal to the number of tokens, but it is a good approximation. Defining the threshold number of tokens can significantly affect how the database is found and retrieved. I found a great article by Pinecone that can help you understand the basics of various chunking strategies.
# Define text chunk strategy
splitter = CharacterTextSplitter(
chunk_size=2000,
chunk_overlap=50,
separator=" "
)
# GDS guides
gds_loader = GitLoader(
clone_url="https://github.com/neo4j/graph-data-science",
repo_path="./repos/gds/",
branch="master",
file_filter=lambda file_path: file_path.endswith(".adoc")
and "pages" in file_path,
)
gds_data = gds_loader.load()
# Split documents into chunks
gds_data_split = splitter.split_documents(gds_data)
print(len(gds_data_split)) #771We could load other Neo4j repositories that contain documentation. However, the idea is to show various data loading methods and not explore all of Neo4j’s repositories containing documentation. Therefore, we will move on and look at how we can load documents from a Pandas Dataframe.
For example, say that we want to load a YouTube video as a document source for our chatbot. Neo4j has its own YouTube channel and, even I appear in a video or two. Two years ago I presented how to implement an information extraction pipeline.
With LangChain, we can use the captions of the video and load it as documents with only three lines of code.
yt_loader = YoutubeLoader("1sRgsEKlUr0")
yt_data = yt_loader.load()
yt_data_split = splitter.split_documents(yt_data)
print(len(yt_data_split)) #10It couldn’t get any easier than this. Next, we will look at loading documents from a Pandas dataframe. A month ago, I retrieved information from Neo4j medium publication for a separate blog post. Since we want to bring external information about Neo4j to the bot, we can also use the content of the medium articles.
article_url = "https://raw.githubusercontent.com/tomasonjo/blog-datasets/main/medium/neo4j_articles.csv"
medium = pd.read_csv(article_url, sep=";")
medium["source"] = medium["url"]
medium_loader = DataFrameLoader(
medium[["text", "source"]],
page_content_column="text")
medium_data = medium_loader.load()
medium_data_split = splitter.split_documents(medium_data)
print(len(medium_data_split)) #4254Here, we used Pandas to load a CSV file from GitHub, renamed one column, and used the DataFrameLoaderfunction to load the articles as documents. Since medium posts could exceed 4000 tokens, we used the text splitter to split the articles into multiple chunks.
The last source we will use is the Stack Overflow API. Stack Overflow is a web platform where users help others solve coding problems. Their API does not require any authorization. Therefore, we can use the API to retrieve questions with accepted answers that are tagged with the Neo4j tag.
so_data = []
for i in range(1, 20):
# Define the Stack Overflow API endpoint and parameters
api_url = "https://api.stackexchange.com/2.3/questions"
params = {
"order": "desc",
"sort": "creation",
"filter": "!-MBrU_IzpJ5H-AG6Bbzy.X-BYQe(2v-.J",
"tagged": "neo4j",
"site": "stackoverflow",
"pagesize": 100,
"page": i,
}
# Send GET request to Stack Overflow API
response = requests.get(api_url, params=params)
data = response.json()
# Retrieve the resolved questions
resolved_questions = [
question
for question in data["items"]
if question["is_answered"] and question.get("accepted_answer_id")
]
# Print the resolved questions
for question in resolved_questions:
text = (
"Title:",
question["title"] + "\n" + "Question:",
BeautifulSoup(question["body"]).get_text()
+ "\n"
+ BeautifulSoup(
[x["body"] for x in question["answers"] if x["is_accepted"]][0]
).get_text(),
)
source = question["link"]
so_data.append(Document(page_content=str(text), metadata={"source": source}))
print(len(so_data)) #777Each approved answer and the original question are used to construct a single document. Since most Stack overflow questions and answers do not exceed 4000 tokens, we skipped the text-splitting step.
Now that we have loaded the documentation resources as documents, we can move on to the next step.
Storing documents in a vector database
A chatbot finds relevant information by comparing the vector embedding of questions with document embeddings. A text embedding is a machine-readable representation of text in the form of a vector or, more plainly, a list of floats. In this example, we will use the ada-002 model provided by OpenAI to embed documents.
The whole idea behind vector databases is the ability to store vectors and provide fast similarity searches. The vectors are usually compared using cosine similarity. LangChain includes integration with a variety of vector databases. To keep things simple, we will use the Chroma vector database, which can be used as a local in-memory. For a more serious chatbot application, we want to use a persistent database that doesn’t lose data once the script or notebook is closed.
We will create two collections of documents. The first will be more sales and marketing oriented, containing documents from Medium and YouTube. The second collection focuses more on support use cases and consists of documentation and Stack Overflow documents.
# Define embedding model
OPENAI_API_KEY = "OPENAI_API_KEY"
embeddings = OpenAIEmbeddings(openai_api_key=OPENAI_API_KEY)
sales_data = medium_data_split + yt_data_split
sales_store = Chroma.from_documents(
sales_data, embeddings, collection_name="sales"
)
support_data = kb_data + gds_data_split + so_data
support_store = Chroma.from_documents(
support_data, embeddings, collection_name="support"
)This script runs each document through OpenAI’s text embedding API and inserts the resulting embedding along with text in the Chroma database. The process of text embedding costs 0.80$, which is a reasonable price.
Question answering using external context
The last thing to do is to implement two separate question-answering flow. The first will handle the sales & marketing requests, while the other will handle support. The LangChain library uses LLMs for reasoning and providing answers to the user. Therefore, we start by defining the LLM. Here, we will be using the GPT-3.5-turbo model from OpenAI.
llm = ChatOpenAI(
model_name="gpt-3.5-turbo",
temperature=0,
openai_api_key=OPENAI_API_KEY,
max_tokens=512,
)Implementing a question-answering flow is about as easy as it gets with LangChain. We only need to provide the LLM to be used along with the retriever that is used to fetch relevant documents. Additionally, we have the option to customize the LLM prompt used to answer questions.
sales_template = """As a Neo4j marketing bot, your goal is to provide accurate
and helpful information about Neo4j, a powerful graph database used for
building various applications. You should answer user inquiries based on the
context provided and avoid making up answers. If you don't know the answer,
simply state that you don't know. Remember to provide relevant information
about Neo4j's features, benefits, and use cases to assist the user in
understanding its value for application development.
{context}
Question: {question}"""
SALES_PROMPT = PromptTemplate(
template=sales_template, input_variables=["context", "question"]
)
sales_qa = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff",
retriever=sales_store.as_retriever(),
chain_type_kwargs={"prompt": SALES_PROMPT},
)The most important part of the sales prompt is to prohibit the LLM from basing its responses without relying on official company resources. Remember, LLMs can act very assertively while providing invalid information. However, we would like to avoid that scenario and avoid getting into problems where the bot promised or sold non-existing features. We can test the sales question answering flow by asking the following question:
The response to the question seems relevant and accurate. Remember, the information to construct this response came from Medium articles.
Next, we will implement the support question-answering flow. Here, we will allow the LLM model to use its knowledge of Cypher and Neo4j to help solve the user’s problem if the context doesn’t provide enough information.
support_template = """
As a Neo4j Customer Support bot, you are here to assist with any issues
a user might be facing with their graph database implementation and Cypher statements.
Please provide as much detail as possible about the problem, how to solve it, and steps a user should take to fix it.
If the provided context doesn't provide enough information, you are allowed to use your knowledge and experience to offer you the best possible assistance.
{context}
Question: {question}"""
SUPPORT_PROMPT = PromptTemplate(
template=support_template, input_variables=["context", "question"]
)
support_qa = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff",
retriever=support_store.as_retriever(),
chain_type_kwargs={"prompt": SUPPORT_PROMPT},
)And again, we can test the support question-answering abilities. I took a random question from Neo4j’s discord server.
The response is quite to the point. Remember, we retrieved the Graph Data Science documentation and are using it as context to form the chatbot questions.
Agent implementation
We now have two separate instructions and stores for sales and support responses. If we had to put a human in the loop to distinguish between the two, the whole point of the chatbot would be lost. Luckily, we can use a LangChain agent to decide which tool to use based on the user input. First, we need to define the available tools of an agent along with instructions on when and how to use them.
tools = [
Tool(
name="sales",
func=sales_qa.run,
description="""useful for when a user is interested in various Neo4j information,
use-cases, or applications. A user is not asking for any debugging, but is only
interested in general advice for integrating and using Neo4j.
Input should be a fully formed question.""",
),
Tool(
name="support",
func=support_qa.run,
description="""useful for when when a user asks to optimize or debug a Cypher statement or needs
specific instructions how to accomplish a specified task.
Input should be a fully formed question.""",
),
]The description of a tool is used by an agent to identify when and how to use a tool. For example, the support tool should be used to optimize or debug a Cypher statement and the input to the tool should be a fully formed question.
The last thing we need to do is to initialize the agent.
agent = initialize_agent(
tools,
llm,
agent="zero-shot-react-description",
verbose=True
)Now we can go ahead and test the agent on a couple of questions.
Remember, the main difference between the two QAs beside the context sources is that we allow the support QA to form answers that can’t be found in the provided context. On the other hand, we prohibit the sales QA from doing that to avoid any overpromising statements.
Summary
In the era of LLMs, you can develop a chatbot that uses company’s resources to answer questions in a single day thanks to LangChain library as it offers various document loaders as well as integration with popular LLM models. Therefore, the only thing you need to do is to collect company’s resources, import them into a vector database, and you are good to go. Just note that the implementation is not deterministic, which means you can get slightly different results on identical prompts. GPT-4 model is much better for more accurate and consistent responses.
Let me know if you have any ideas or feedback regarding the chatbot implementation. As always, the code is available on GitHub.
