When I was learning data science and machine learning at university, the curriculum was geared heavily towards algorithms and machine learning techniques. I still remember those days cracking the math, not exactly fun, but nonetheless a rewarding process that had given me a solid foundation.
Once I graduated and started working as a data scientist, I soon realized the challenge: In real life, problems rarely present themselves as nicely formulated and readily addressable by machine learning techniques. It is the data scientist’s job to first define, scope, and convert the real-life problem into a machine-learning problem, before even talking about the algorithms. This is a crucial step as completely different approaches may be adopted depending on how the problem is formulated, what is the desired outcome, what data is available, the timeline, the budget, the computing infrastructure, and many other factors. In a word, it is not a simple math problem anymore.
This gap in my data science training made me feel disoriented and pressured in the beginning. Luckily, I had my mentor and project colleagues, who helped me a lot in picking up the essentials and learning to ask the right questions. Step by step, I became more confident in managing data science projects.
Reflecting on my own experience, I really wish I could have the chance to learn those soft skills in data science to better prepare for my professional life. Now I have gone through the struggles, but is there anything I could do for the newly graduated data scientists?
A famous book for preparing interviews in management consulting is "Case in Point". This book provides numerous practice case studies that cover a wide range of topics and industries. By observing and understanding how those case studies are solved, the candidates can learn quite a lot in practical problem-solving processes and be ready for real-life challenges.
Inspired by this case-study format, a thought occurred to me: Can we leverage the recent large language models (LLM) to generate relevant, diverse data science case studies on-demand and simulate the problem-solving process of a data scientist? This way, we could create a "Case in Point (Data Science Edition)" and help prepare the aspiring data scientist for real-life challenges.
So, in this blog post, we will try to bring this idea to life. Specifically, we will walk through the process of developing a role-playing dual-chatbot system that can simulate problem-solving for data science problems.
Does this idea resonate with you? let’s get started!
[NOTE]: All the prompts shown in this blog are generated and optimized by ChatGPT (GPT-4). This is necessary as it ensures the quality of the prompts and beneficial as it saves one from tedious manual prompt engineering.
This is the 3rd blog on my series of LLM projects. The 1st one is Building an AI-Powered Language Learning App, and the 2nd one is Developing an Autonomous Dual-Chatbot System for Research Paper Digesting. Feel free to check them out!
Table of Content
· 1. Solution Strategy ∘ 1.1 System overview ∘ 1.2 Abstract LLM class · 2. Scenario Generation ∘ 2.1 User options ∘ 2.2 Generation strategy ∘ 2.3 Code implementation ∘ 2.4 Testing: Scenario generation · 3. Client-Data Scientist Simulation ∘ 3.1 Client bot design ∘ 3.2 Data scientist bot design ∘ 3.3 Simulating conversation · 4. Conversation Assessment ∘ 4.1 Strategy overview ∘ 4.2 Summarizer bot design ∘ 4.3 Assessor bot design ∘ 4.4 Testing the workflow · 5. Retrospective
1. Solution Strategy
1.1 System overview
The foundation of our solution evolves around the concept of a role-playing dual-chatbot system. In essence, this system involves two Chatbots (powered by large language models) taking on different roles and engaging in an autonomous conversation.
Given that our ultimate purpose is to simulate the real-life problem-solving process of a data scientist, a natural option for setting the roles of the chatbots can be "client" and "data scientist", i.e., one bot plays the role of a client, who is seeking the solution for a problem its company is currently facing, and the other bot plays the role of a data scientist. Through their conversation, the data scientist bot will try to understand the problem in-depth and the client bot will clarify and confirm various aspects of the problem. Together, two bots collaborate to properly define and scope the problem, and agree on suitable machine learning solutions. In section 3, we will dive into the design of this dual-chatbot system.
To facilitate the client-data scientist conversation, we first need to generate a high-quality scenario description, and the produced description should be tailored to the user’s interests. More specifically, in my current design, the user can choose, e.g., their interested problem type, target industry, and business size. Then, a relevant, concrete Data Science project scenario will be generated accordingly, which will be subsequently fed to both bots to serve as the foundation for their dialogue. In section 2, we will discuss the design details of the scenario generation procedure.
To further enhance the user’s learning experience, it is beneficial to reflect on the conversation and extract the key learning points for the user to review. Those key learning points could include the specific strategy **** adopted by the data scientist bot in terms of scoping the problem, the various aspects covered/not covered in the conversation, as well as potential follow-up questions or topics for discussion. In section 4, we will take a close look at the design details of this assessment procedure.
Overall, our entire system consists of scenario generation, conversation Simulation, as well as conversation assessment modules, which can be depicted as follows:
1.2 Abstract LLM class
Throughout this blog, we will create different LLM bots for different purposes. To streamline the code, we can define an abstract base class LLMBot to serve as the template. We will use the versatile LangChain library to manage our interaction with the language model.
from abc import ABC, abstractmethod
from langchain.chat_models import ChatOpenAI
from langchain.llms import OpenAI
class LLMBot(ABC):
"""Class definition for a single LLM bot"""
def __init__(self, endpoint_type, temperature):
"""Initialize the large language model.
Args:
--------------
endpoint_type: "chat" or "completion".
temperature: temperature of the LLM.
"""
# Instantiate llm
# Reminder: need to set up openAI API key
# (e.g., via environment variable OPENAI_API_KEY)
if endpoint_type == 'chat':
self.llm = ChatOpenAI(model_name="gpt-3.5-turbo",
temperature=temperature)
elif endpoint_type == 'completion':
self.llm = OpenAI(model_name="text-davinci-003",
temperature=temperature)
else:
raise KeyError("Currently unsupported endpoint type!")
@abstractmethod
def instruct(self):
"""Determine the context of LLM bot behavior.
"""
pass
@abstractmethod
def step(self):
"""Response produced by the LLM bot.
"""
passFor the LLM bot, we distinguish between a chat endpoint and a completion endpoint. The chat endpoint is designed for multi-round conversations (i.e., a chatbot), whereas the completion endpoint is designed for single-turn tasks. Different LLMs will be invoked based on which endpoint is used.
In addition, we defined two common methods: instruct() is used to determine the LLM bot’s behavior, and step() is used to send input to the LLM bot and receive the bot’s response.
With the template in place, we are prepared to create specific instances of LLM bots later. In the next sections, we will discuss the design of each sub-module of the system and its code implementation.
2. Scenario Generation
Let’s start with the scenario generation. The main objective of this module is to generate a concrete and detailed data science case study that is tailored to the user’s interests. In this section, we discuss two design considerations relevant to the scenario generation: the user options for defining the case study, and the strategy for generating the case study with LLM.
2.1 User options
To define a case study that aligns with the user’s interests, we need to first get input from the user. Here, the strategy I adopted is to give the user a set of categories of options that are limited in number so that they are not overwhelmed, but also comprehensive enough to allow LLM to generate meaningful data science scenarios.
After some experiments, I found the following three categories of options could serve as the seed for shaping the generated case study:
1️⃣ Problem type, defines the specific type of machine learning problems that the user is interested in learning. This category includes options such as classification, regression, clustering, anomaly detection, recommendation, time series analysis, natural language processing, and computer vision.
2️⃣ Target industry, indicates which industry the user would like to see the application of the machine learning techniques. This category includes options such as healthcare, finance, retail, technology, manufacturing, transportation, energy, real estate, education, government, and non-profit.
3️⃣ Business size, **** may suggest the complexity and constraints of the data science problem. This category includes options such as small (less than 100 employees), medium (100–500 employees), and large (more than 500 employees).
After the user selects the option for each category, we need to generate a corresponding data science case study. How exactly can we do that?
2.2 Generation strategy
As we have mentioned in the "strategy overview" section, we can employ an LLM (e.g., GPT-3.5) to achieve our goal.
Technically, I adopted a staged strategy with two passes of the LLM to create the desired description of the problem scenario. More specifically, in the first stage, the LLM was prompted to generate a broad description of the scenario; in the second stage, the LLM was prompted to fill in the details for the previously generated scenario.
Compared to getting everything from a single pass of the LLM, this staged approach mimics the process of draft-and-refine and therefore has a clear goal at each stage. As a result, it is easier for the LLM to follow our instructions and produce a final problem description that is more specific and detailed, which could better serve as the foundation for the subsequent client-data scientist conversation.
2.3 Code implementation
Given that the two stages are tightly connected, a chat endpoint (instead of a completion endpoint) would be a natural fit as it automatically keeps the context in memory. This way, we can simply ask the chat LLM two questions in turn (each question represents one stage) and use the chat model’s second answer as the final result.
The following is the definition of the "scenario generator" bot:
from langchain.memory import ConversationBufferMemory
from langchain.prompts import (
ChatPromptTemplate,
MessagesPlaceholder,
SystemMessagePromptTemplate,
HumanMessagePromptTemplate
)
class ScenarioGenerator(LLMBot):
"""Class definition for the scenario generator bot."""
def __init__(self, temperature=1.0):
"""Setup scenario generator bot.
Args:
--------------
temperature: temperature of the LLM.
"""
# Instantiate llm
super().__init__('chat', temperature)
# Instantiate memory
self.memory = ConversationBufferMemory(return_messages=True)
def instruct(self, industry, business_size, problem_type, details):
"""Determine the context of scenario generator.
Args:
--------------
industry: interested industry, e.g., healthcare, finance, etc.
business_size: large, medium, small
problem_type: type of machine learning problem, e.g., classification, regression, etc.
details: specific details added to the description.
"""
self.industry = industry
self.business_size = business_size
self.problem_type = problem_type
self.details = ScenarioGenerator.industry_specifics[industry]
prompt = ChatPromptTemplate.from_messages([
MessagesPlaceholder(variable_name="history"),
HumanMessagePromptTemplate.from_template("""{input}""")
])
# Construct a conversation chain
self.scen_generator = ConversationChain(memory=self.memory, prompt=prompt,
llm=self.llm)
def step(self):
"""Interact with the LLM bot.
Outputs:
--------------
scenario: the generated scenario description.
"""
# 1st stage (draft)
print("Generating scenario description: drafting stage...")
prompt_1st = self._get_1st_stage_prompt()
self.interm_scenario = self.scen_generator.predict(input=prompt_1st)
# 2nd stage (review and enrich)
print("Generating scenario description: refining stage...")
prompt_2nd = self._get_2nd_stage_prompt()
self.scenario = self.scen_generator.predict(input=prompt_2nd)
print("Scenario description generated!")
return self.scenarioNote that we set a high default temperature value for the backbone LLM as we want diversity in the generated case study scenarios. In the self.instruct() method, we introduced a new attribute self.details. This attribute specifies the extra details added to the problem description and will be useful for the 2nd stage generation. Additionally, we have instantiated a ConversationChain to set up the scenario generator bot. Finally, in the self.step() method, we implemented the two-stage approach, and the bot’s response in the second stage is used as the final produced scenario description. Note that there is no need to re-input the LLM’s response in the 1st stage to the LLM in the 2nd stage. Since the scenario generator is a chatbot, its memory automatically carries the previous response to the next round of conversation.
Now, let’s take a closer look at the prompt used in the two stages. The purpose of the first stage is to draft a description of a typical data science scenario given the user input (i.e., industry, business size, and problem type). The prompt used there is shown below:
def _get_1st_stage_prompt(self):
# Note that the prompt is generated and fine-tuned by ChatGPT (GPT-4)
prompt = f"""For a {self.industry} company of {self.business_size} size
focusing on {self.problem_type} problems, generate a concrete data science
project scenario that a data scientist might encounter in real life.
Please provide concrete and specific details relevant to the selected
industry and problem type.
For the generated scenario, please provide:
1. A specific and realistic description of a problem faced by the company.
2. The desired outcome that the company is hoping to achieve by solving
the problem.
3. A list of the top 3 most relevant data sources that might be available
for solving the problem.
Output format:
Problem description: [content of problem description]
Desired outcome: [content of desired outcome]
Available data: [content of available data]
"""
return promptIn the prompt above, the scenario generator bot is explicitly asked to output the problem description, desired outcome, and available data for describing the data science scenario. Those three pieces of information constitute the foundation for the subsequent client-data scientist conversation.
For the second stage, the purpose is to enrich the scenario description obtained from the 1st stage with more tangible details. This is where the self.details come into play. The ScenarioGenerator class contains a class variable called industry_specifics, which is a dictionary with the keys being the industry names and values being the tangible details associated with a specific industry. A snippet of the dictionary is shown below as an example. The complete dictionary can be found in the companion notebook 💻 .
class ScenarioGenerator(LLMBot):
# Note that the descriptions of the industry specifics
# are generated and optimized by ChatGPT (GPT-4)
industry_specifics = {
'healthcare': """types of patients treated (e.g., age, medical conditions),
common treatments and procedures, challenges faced in patient care,
medical equipment used.""",
'finance': """types of financial products and services offered,
...""",
...
}The following is the prompt used in the 2nd stage:
def _get_2nd_stage_prompt(self):
# Note that the prompt is generated and fine-tuned by ChatGPT (GPT-4)
prompt = f"""Based on the previously generated scenario, please enrich
the problem description by providing more specific details
(such as {self.details}) about the problem.
Output format:
Enriched problem description: [content of enriched problem description]
Desired outcome: [content of desired outcome]
Available data: [content of available data]
"""
return prompt2.4 Testing: Scenario generation
To assess if the adopted two-stage approach can generate a concrete data science case study given the user input, we put the defined ScenarioGenerator class to a test. As an example, I selected the "manufacturing" industry, the "medium" business size, and the "anomaly detection" problem type, and passed them to the scenario generator bot to create a possible scenario.
# User selections
industry = "manufacturing"
business_size = "medium"
problem_type = "anomaly detection"
details = "Types of products manufactured, machines used in the production process,
common issues faced by the company, tools and technologies used for quality control."
# Scenario generation
generator = ScenarioGenerator()
generator.instruct(industry, business_size, problem_type, details)
scenario = generator.step()The produced scenario is shown in the figure below:
We can see that the generated scenario contained very specific details and proposed a typical data science problem faced by the manufacturing industry. This shows that the scenario generator bot has followed our instructions closely and fulfilled our goal in a satisfactory manner. Later, the client-data scientist bot interaction will be based on this produced scenario.
3. Client-Data Scientist Simulation
The key component of our entire system is the dual-chatbot interaction module. For our current problem, the two chatbots take the role of client and data scientist, respectively. The client bot is in charge of clarifying the situation and giving feedback on possible solutions, while the data scientist bot is in charge of understanding the problem in depth and proposing possible solutions. Collaboratively, they define and scope the data science project.
In this section, we discuss the technical details of the dual-chatbot interaction. We will start by outlining the design of the two chatbots, followed by simulating their conversation based on the scenario generated from the previous section.
3.1 Client bot design
Let’s start with the client bot design:
class ClientBot(LLMBot):
"""Class definition for the client bot."""
def __init__(self, temperature=0.8):
"""Setup scenario generator bot.
Args:
--------------
temperature: temperature of the LLM.
"""
# Instantiate llm
super().__init__('chat', temperature)
# Instantiate memory
self.memory = ConversationBufferMemory(return_messages=True)
def instruct(self, industry, business_size, scenario):
"""Determine the context of client chatbot.
"""
self.industry = industry
self.business_size = business_size
self.scenario = scenario
# Define prompt template
prompt = ChatPromptTemplate.from_messages([
SystemMessagePromptTemplate.from_template(self._specify_system_message()),
MessagesPlaceholder(variable_name="history"),
HumanMessagePromptTemplate.from_template("""{input}""")
])
# Create conversation chain
self.conversation = ConversationChain(memory=self.memory, prompt=prompt,
llm=self.llm, verbose=False)
def step(self, prompt):
"""Client chatbot speaks.
Args:
--------
prompt: data scientist's response.
"""
response = self.conversation.predict(input=prompt)
return responseThe definition of the ClientBot is quite similar to the ScenarioGenerator. Noticeable differences include:
- The
self.instruct()method now takes the "scenario" generated by theScenarioGeneratorearlier as the input. This information will be used to guide the client bot. - The instruction of the client bot is not fed as a direct prompt (like we did with the
ScenarioGenerator) but rather a so-calledSystemMessage. This is a common way to instruct the chatbot on how to behave during the entire conversation. - The
self.step()method is simplified as the client will only need to respond to what the data scientist bot says (contained in thepromptvariable).
Finally, let’s examine the prompt design for the client bot. The overall goal of the prompt design should be to emphasize collaboration, clear communication, and problem definition. Meanwhile, it should also keep the chatbot’s responses aligned with its role (i.e., a client). To achieve this goal, we can structure it in a way that first sets the stage for the client bot, then informs it about the proposed problem, followed by setting the goals and guidelines to constrain its behavior:
def _specify_system_message(self):
"""Specify the behavior of the client chatbot.
"""
# Prompt
# The prompt is generated and fine-tuned by ChatGPT (GPT-4)
prompt = f"""You are role-playing a representative from a {self.industry}
company of {self.business_size} size and you are meeting with a
data scientist (which is played by another bot), to discuss how to
leverage machine learning to address a problem your company is facing.
The problem description, desired outcome, and available data are:
{self.scenario}.
Your ultimate goal is to work with the data scientist to define a clear
problem and agree on a suitable data science solution or approach.
Guidelines to keep in mind:
- **Get Straight to the Point**: Start the conversation by directly
addressing the problem at hand. There is no need for pleasantries or
introductions.
- **Engage in Conversation**: Respond to the data scientist's questions
and prompts. Do not provide all the information at once or provide
the entire conversation yourself.
- **Clarify and Confirm**: Always make sure to clarify and confirm the
problem, desired outcome, and any proposed solutions with the data
scientist.
- **Stay in Role**: Your role as a client is to represent your company's
needs and work with the data scientist to define a clear problem and
agree on a suitable data science solution or approach.
Do not try to propose solutions.
- **Provide Information as Needed**: Provide information about the problem,
available data, constraints, and requirements as it becomes relevant in
the conversation. If the data scientist asks a question and the
information was not provided in the problem description, it is okay to
improvise and create details that seem reasonable.
- **Collaborate**: Collaborate with the data scientist to clearly define
the problem and to consider any proposed solutions or approaches.
"""
return promptOne thing worth mentioning is that in the prompt, we gave the permission to let the bot improvise if the data scientist asks questions beyond what’s described in the self.scenario. The motivation for that is to further promote diversity.
That’s it for the client bot part. Let’s now switch gears to the data scientist bot.
3.2 Data scientist bot design
The data scientist bot has almost the same class design as the client bot, except the the information input to the self.instruct() method, and the prompt design.
For the data scientist bot, we do not input the detailed problem description. It is the data scientist bot’s job to figure that out via discussing with the client bot. Instead, we simply inform it what type of machine learning problem the client is interested in.
def instruct(self, industry, business_size, problem_type):
"""Determine the context of data scientist chatbot.
"""
self.industry = industry
self.business_size = business_size
self.problem_type = problem_type
# Define prompt template
...
# Create conversation chain
...As for the prompt design, it basically follows the same design philosophy as the client bot, with different focuses on understanding the problem and suggesting solutions:
def _specify_system_message(self):
"""Specify the behavior of the data scientist chatbot.
"""
# Prompt
# The prompt is generated and fine-tuned by ChatGPT (GPT-4)
prompt = f"""You are role-playing a data scientist meeting with a
representative (which is played by another chatbot)
from a {self.industry} company of {self.business_size} size.
They are currently concerned with a {self.problem_type} problem.
Your ultimate goal is to understand the problem in depth and agree on a
suitable data science solution or approach by engaging in a conversation
with the client representative.
Guidelines to keep in mind:
- **Engage in Conversation**: You are only the data scientist.
Do not provide the entire conversation yourself.
- **Understand the Problem**: Make sure to ask questions to get a clear
and detailed understanding of the problem, the desired outcome,
available data, constraints, and requirements.
- **Propose Solutions**: Based on the information provided by the client,
suggest possible data science approaches or solutions to address the
problem.
- **Consider Constraints**: Be mindful of any constraints that the client
may have, such as budget, timeline, or data limitations, and tailor your
proposed solutions accordingly.
- **Collaborate**: Collaborate with the client to refine the problem
definition, proposed solutions, and ultimately agree on a suitable data
science approach.
"""
return prompt3.3 Simulating conversation
Now we have set up both bots, let’s see what kind of conversation they will have!
# Create two chatbots
client = ClientBot()
data_scientist = DataScientistBot()
# Specify instructions
client.instruct(industry, business_size, scenario)
data_scientist.instruct(industry, business_size, problem_type)
# Book-keeping
question_list = []
answer_list = []
# Start conversation
for i in range(6):
if i == 0:
question = client.step('Start the conversation')
else:
question = client.step(answer)
question_list.append(question)
print("👨 💼 Client: " + question)
answer = data_scientist.step(question)
answer_list.append(answer)
print("👩 💻 Data Scientist: " + answer)
print("nn")The generated conversation script is shown below:
We can see that the two bots have a very interesting and productive conversation😃 . The data scientist bot asked probing questions to better understand the problem at hand, the client bot responded with necessary clarifications, and then the data scientist bot proposed solutions and explained how the solutions might work for the client’s case. Overall the conversation went as we would expect, and could potentially serve as valuable material for aspiring data scientists to learn practical problem-solving.
4. Conversation Assessment
To further enhance the learning experience, it could be useful to re-examine the generated conversation script and extract the key learning points. In this section, we discuss the final module of our system, the conversation assesser.
4.1 Strategy overview
To create an assessment of the conversation, we could instantiate another LLM bot to achieve the goal. Here, we need to answer two questions: first, what information should be made available to the assessor bot? Second, how should we design the instruction/prompt for the assessor bot? Let’s investigate them one by one.
For the first question, it is important that the assessor bot has knowledge of the basic problem settings. Meanwhile, it should also have access to the generated conversation script. The basic problem settings are already encapsulated in the user-selected "problem type", "industry", and "problem size", thus providing them to the assessor bot is straightforward.
However, for the generated conversation script, we need to be a bit careful. Usually, the conversation script can be quite lengthy. As a consequence, directly feeding in the entire conversation script may easily exceed the context window limit of the underlying LLM. On the other hand, for the assessor bot to make an assessment, it is actually not necessary for it to know all the details of the script. What really matters is the logical flow of the conversation. Therefore, we could have a "summarizer" bot to first summarize the conversation script, and later send the condensed script to the assessor bot to make the assessment. This staged strategy is illustrated in the figure below.
In the next two subsections, we discuss how to create the summarizer bot and how to design the prompt for the assessor bot.
4.2 Summarizer bot design
Let’s start by constructing a class to define the summarizer bot:
class SummarizerBot(LLMBot):
def __init__(self, temperature=0.8):
"""Setup summarizer bot.
Args:
--------------
temperature: temperature of the LLM.
"""
# Instantiate llm
super().__init__('completion', temperature)Note that we are now using the completion endpoint, as the summarization is intended to be a one-time task and there is no need for the summarizer bot to keep the memory.
Next, we set up the instructions for the summarizer bot:
def instruct(self):
"""Determine the context of summarizer.
"""
# Note: The prompt is generated and optimized by ChatGPT (GPT-4)
template = """Please concisely summarize the following segment of a
conversation between a client and a data scientist discussing a
potential data science project:
{conversation}
"""
self.prompt = PromptTemplate(
template=template,
input_variables=["conversation"],
)Here, our strategy is to loop over individual rounds of the conversation and extract their associated key points. Therefore, our self.step() method looks like this:
def step(self, q_list, a_list):
"""Summarize the conversation script.
Args:
---------
q_list: list of responses from the client bot
a_list: list of responses from the data scientist bot
"""
# Loop over individual rounds
conversation_summary = []
for i, (q, a) in enumerate(zip(q_list, a_list)):
print(f"Processing {i+1}/{len(q_list)}th conversation round.")
# Compile one round of conversation
conversation_round = ''
conversation_round += 'Client: ' + q + 'nn'
conversation_round += 'Data scientist: ' + a
response = self.llm.predict(self.prompt.format(conversation=conversation_round))
conversation_summary.append(response)
return conversation_summary4.3 Assessor bot design
With the conversation summary ready, we can now focus on developing the assessor bot. Same as the summarizer bot, we invoke the completion endpoint for the assessor bot:
class AssessorBot(LLMBot):
def __init__(self, temperature=0.8):
"""Setup assessor bot.
Args:
--------------
temperature: temperature of the LLM.
"""
# Instantiate llm
super().__init__('completion', temperature)Here we define the instruction:
def instruct(self, industry, business_size, problem_type):
"""Determine the context of assessor.
"""
self.industry = industry
self.business_size = business_size
self.problem_type = problem_type
# Note: The prompt is generated and optimized by ChatGPT (GPT-4)
template = """You are a senior data scientist who has been asked to
review a conversation between a data scientist and a client from a
{industry} company of {business_size} size, focusing on a {problem_type}
problem. The client and data scientist are discussing how to define and
scope a data science project to address the problem.
Please provide an assessment of the conversation, focusing on the strategy
adopted by the data scientist to define and scope the problem,
any potential room for improvement, and any other key points you think
are important. Please organize your response with nicely formatted
bulletpoints.
Here is the conversation:
{conversation}
"""
self.prompt = PromptTemplate(
template=template,
input_variables=["industry", "business_size",
"problem_type", "conversation"],
)Note that in the above prompt, we ask the assessor bot to act as a senior data scientist to provide feedback on the strategies adopted by the data scientist bot, as well as to indicate room for improvement. Both aspects are valuable for understanding the data science problem-solving process.
Finally, to invoke the assessor bot:
def step(self, conversation_summary):
"""Assess the conversation script.
Args:
---------
conversation_summary: condensed version of the conversation script.
"""
analysis = self.llm.predict(self.prompt.format(industry=industry,
business_size=business_size,
problem_type=problem_type,
conversation=' '.join(conversation_summary)))
return analysis4.4 Testing the workflow
To test the workflow of the summarizer-assessor bot, we can feed in the previously generated conversation script and analyze how it goes. The produced assessment is shown below:
We can see that the assessor bot produced a useful analysis of how the data scientist bot performed in understanding the problem and proposing corresponding solutions. It also hinted at which aspects that were not covered. All those summaries can serve as valuable learning points.
5. Retrospective
In this blog, we investigated the problem of simulating real-world data science problem-solving processes, which could provide valuable insights for aspiring data scientists to prepare for practical challenges. Toward that end, we developed a system that constitutes a scenario generator bot, which generates a realistic, detailed description of data science scenarios, a client-data scientist dual-chatbot, which collaborative defines and scopes the problem, and an assessor bot, that analyzes the conversation and extracted the key learning points. Together, the system allows for generating interesting data science case studies on demand and possesses the potential as a learning tool that complements traditional, algorithm-focused machine learning education.
Reflecting on what we have built, there are certainly many things to be improved:
- Although the initial results seemed promising, more tests are needed to assess if the same quality can be obtained for other industries, problem types, and business sizes.
- It is important to investigate the sensitivity of different design decisions on the quality of the generated conversation. For example, one important design decision we made is that only the problem description, desired outcome, and data availability are presented in the generated scenario, and the client bot will improvise if any other information is requested by the data scientist bot. Is this a good choice? Would the conversation quality be higher if the generated scenario contained more information? It would be interesting to find out.
- Currently, there are no explicit criteria to naturally stop the conversation, except a hard-coded number of exchanges we specified. I had a similar problem in my previous project, and I proposed two different strategies to handle this issue. Further investigation is necessary to see if those two strategies (or other strategies) could work.
- In the current design, the only place where the user can influence the conversation is when specifying the inputs (e.g., problem type, etc.) for scenario generation. However, we can further enhance the user experience by allowing the user to participate in the conversation, either acting as the data scientist to ask clarifying questions/propose novel solutions, or acting as the client to provide missing details. A similar idea was also pursued in my previous blog. Feel free to check it out if you are interested in its implementation details.
- Our developed system can also be extended to other learning settings. In the current blog, we only focused on the problem-solving aspect. However, we could also use the same system (with the right prompts of course) to simulate scenarios for project management, conflict management, communication, and many other important soft skills required by the data scientist role.
If you find my content useful, you could buy me a coffee here 🤗 Thank you very much for your support! As always, you can find the companion notebook with full code here 💻 Looking forward to sharing with you more exciting LLM projects. Stay tuned!
