Researching a Multilingual FEMA Disaster Bot Using LangChain and GPT-4
Some pros and cons of Retrieval-Augmented Generation (RAG) for high-risk chat applications
Published in
54 min readAug 17, 2023
TL;DR
In this article, we explore how to build a multilingual US Federal Emergency Management Agency (FEMA) disaster chatbot to help people prepare for and survive disasters such as floods, tornados, wildfires, earthquakes, and winter storms. A chat interface was created from 34 FEMA PDF documents using LangChain and GPT-4. As amazing as this popular pattern is, care is needed with high-risk applications such as disaster response bots. Though Large Language Model (LLM) hallucinations are minimized, common issues with semantic search of documents can lead to critical information being excluded in chat responses. We tested a few simple techniques to improve performance for this specific analysis, such as incorporating document metadata into embeddings, enriching user questions with LLM zero-shot context classification, and automatic language detection and translation using Google Translate to better support languages like Swahili. The techniques applied were pretty basic, but the ability of the prototype bot to surface information efficiently from FEMA PDF documents shows promise. Obviously, testing and validation would be needed if using this technique for high-risk situations, ideally using automated LLM techniques to create question-and-answer validation data.
A few weeks ago we had a major flooding event in Vermont. The small brook that gurgles cheerfully past our house turned into a raging monster hellbent on destruction. Luckily we weren’t damaged badly, but sadly, many lost property and livelihoods. At one point during the flood, it looked like we might have to evacuate so I started checking the US Federal Emergency Management Agency (FEMA) website for advice. I had already prepared somewhat, but when trouble arrived I wanted to reconfirm a few things. FEMA has really excellent, concise resources in PDF documents and web pages which I started searching, but I wondered ...
- In an emergency, is there a faster way to get helpful information that doesn’t require searching and reading multiple documents?
One obvious solution might be to ask a chatbot. I think chatbots can get a bit overused at times, but this does seem like a solid use case as a conversational interface can be more efficient when time is of the essence.
It is not a new idea, organisations such as the American Red Cross have developed bots like Clara for disaster response. However, a promising new pattern has recently emerged to use Generative AI Large Language Models (LLMs) such as OpenAI’s GPT-4, Meta’s LLAMA 2, and a growing plethora of models on HuggingFace. These models can be used to index a specified set of documents and interact with them conversationally. Called Retrieval-Augmented Generation (RAG), there are hundreds of tutorials on the web demonstrating it with the amazing LangChain Python package and I fully expect this technique to appear soon in software we use every day. What’s interesting is that it restricts responses to the provided content and so is less prone to hallucinations which can prevent using LLMs in critical situations such as disaster response.
But how safe is it in cases where the information retrieved might save lives?
In this article, I briefly explore creating a LangChain GPT-4 chat interface for asking questions about disaster safety based on a set of documents from the US Federal Emergency Management Agency (FEMA). We will encounter some of the limitations that need to be considered if using this technique in high-risk situations.
FEMA Disaster Preparation and Safety PDF Documents
For this study, I downloaded 34 PDFs from FEMA (listed here) which cover a wide range of disaster-related topics for preparing and reacting to emergencies such as wildfires, tornadoes, floods, earthquakes, and winter storms. This isn’t the full set of amazing resources FEMA offer, but should suffice to test our chat interface.
Indexing Documents For Information Retrieval
Once downloaded, we can use LangChain to read the PDF documents. Documents are split into chunks of text, and each is given a fingerprint (embeddings) using an embedding model. In this analysis, we will use OpenAI’s embeddings, but LangChain supports many others.
import os
from langchain.document_loaders import PyPDFDirectoryLoader
from langchain.document_loaders import PyPDFLoader
files = os.listdir(pdf_folder_path)
files.sort()
all_docs_list =[]
for file in files:
if file.endswith('.pdf'):
print(file)
all_docs_list.append(file)
loader = PyPDFDirectoryLoader(pdf_folder_path)
all_docs = loader.load()
print(pdf_folder_path)cfpb_adult-fin-edyour-disaster-checklist.pdf
fema_protect-your-home_flooding.pdf
fema_protect-your-property-storm-surge.pdf
fema_protect-your-property_coastal-erosion.pdf
fema_protect-your-property_earthquakes.pdf
fema_protect-your-property_severe-wind.pdf
fema_protect-your-property_wildfire.pdf
fema_safeguard-critical-documents-and-valuables.pdf
fema_scenario_1-active_shooter-01102020.pdf
fema_scenario_10_power_outage_01102020.pdf
fema_scenario_10_power_outage_answer_key_01102020.pdf
fema_scenario_11_winter_storm_01102020.pdf
fema_scenario_11_winter_storm_answer_key_01102020.pdf
fema_scenario_12_small_business_01102020.pdf
fema_scenario_12_small_business_answer_key_01102020.pdf
fema_scenario_1_active_shooter_TTX_answer_key-01102020.pdf
fema_scenario_2-tornado_TTX_answer_key-01102020.pdf
fema_scenario_2_tornado-01102020.pdf
fema_scenario_3-wildfire_TTX_answer_key-01102020.pdf
fema_scenario_3_wildfire-01102020.pdf
fema_scenario_4-hurricane-01102020.pdf
fema_scenario_4_hurricane_flood_TTX_answer_key-01102020.pdf
fema_scenario_5_extreme_heat-01102020.pdf
fema_scenario_5_extreme_heat_TTX_answer_key_01102020.pdf
fema_scenario_6-pet_preparedness_01102020.pdf
fema_scenario_6-pet_preparedness_TTX_answer_key_01102020.pdf
fema_scenario_7-shelter_in_place_TTX_answer_key_01102020.pdf
fema_scenario_7_shelter_in_place_01102020.pdf
fema_scenario_8_earthquake_01102020.pdf
fema_scenario_8_earthquake_answer_key_01102020.pdf
fema_scenario_9_pandemic_Influenza_01102020.pdf
fema_scenario_9_pandemic_answer_key_01102020.pdf
ready_12-ways-to-prepare_postcard.pdf
ready_document-and-insure-your-property.pdfLooking at data extracted for one document https://www.fema.gov/sites/default/files/2020-11/fema_protect-your-home_flooding.pdf …
import json
for d in all_docs:
if 'fema_protect-your-home_flooding.pdf' in d.metadata['source']:
print('\n')
print(json.dumps(vars(d), indent=4)){
"page_content": " \n \n \nPROTECT YOUR \nPROPERTY FROM FLOODING\n \n",
"metadata": {
"source": "docs_data/fema_protect-your-home_flooding.pdf",
"page": 0
}
}
{
"page_content": " \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n -\n\u2014\n- - -\nOwning a property is one of the most important investments most people make \nin their lives. We work hard to provide a home and a future for ourselves and our loved ones. Why risk losing it when bad weather hits close to home? \nFlooding is the most common and costly natural disaster in the United States \nand can happen anywhere. Just one inch of water can cause $25,000 in damages to your home. \nWhile you can\u2019t prevent a natural disaster from happening, there are ways to secure \nyour property to minimize damage and keep your home and your future safe. \nFirst, determine the Base Flood Elevation (BFE) for your home. The \nBFE is how high the water is expected to rise during fooding in high risk areas. You need to know your BFE because it is used in foodplain management regulations in your community that could affect your home for example, how high above the BFE a home or other building should be built. Your local foodplain manager can help you fnd this information. If you need help fnding your foodplain manager, contact FEMA\u2019s Flood Mapping and Insurance eXchange\n a\nt FEMAMapSpecialist@ \nr\niskmapcds.com or (877) FEMA MAP (1 877 336 2627). \nThe following are some additional steps you can take to protect yourself and your property against foods. ",
"metadata": {
"source": "docs_data/fema_protect-your-home_flooding.pdf",
"page": 1
}
}
{
"page_content": " \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \nINSIDE THE HOME \nPREPARE OR \nUPDA\nTE A LIST OF \nBELONGINGS Documenting all of your belongings will help with the \ninsurance claims process. Consider taking photos of \nhigh-value items or doing a video walkthrough of your \nho\nme to document its contents. \nGET FLOOD \nINSURANCE Most homeowners insurance \npolicies don\u2019t cover food damage. \nProtect your investment by purchasing food insurance for your home and contents, even if you do not live in \na hi\ngh-risk food zone. \nSTORE \nVALUABLES Store valuables and important documents in waterproof or wa\nter-resistant containers above the BFE \n(preferably on an upper foor). Make copies and store them online or offsite. For more \ninformation on purchasing food insurance through the National Flood \nInsurance Program, visit \nFlo\nodSmart.gov or contact \nyour agent to get coverage. EL\nEVATE UTILITIES \nABOVE THE BFE Elevate or foodproof mechanical units, furnaces, water \nheaters, electrical systems, and other utilities on masonry, concrete, or pressure-treated lumber at least 12 inches above the BFE. \nREPLACE CARPETING \nWITH TILES Tiles are more food-resistant than carpet. Using tile or other food-resistant materials in areas below the BFE can help reduce water damage. \nFLOODPROOF \nBASEMENTS If you have a basement, minimize damage by foodproofng your basement and sealing walls with waterproofng compounds. Consider installing a sump pump. \nINSTALL FLOOD \nVENTS Install food vents in foundation walls, garages, and other enclosed areas to allow water \nto fow through, drain out, an\nd lower the risk of \nstructural damage. \nUSE FLOOD-RESIST\nANT \nINSULATION & DRYWALL Fl\nood-resistant insulation and drywall will help minimize \ndamage and can be easily cleaned and sanitized. \nPREVENT SEW\nAGE \nBACK-UP I\nn some areas, fooding can cause sewage to back \nup through drain pipes in your home. Consult with a plumber and, if applicable, invest in a sewer backfow valve to prevent this potential health hazard. DID YOU KNOW? \nWhen following National Flood \nInsurance Program regulations, \nv\nents can also help lower \ninsu\nrance rates. \n",
"metadata": {
"source": "docs_data/fema_protect-your-home_flooding.pdf",
"page": 2
}
}
{
"page_content": " \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \nOUTSIDE THE HOME \nELEVATE YOUR \nHOME While it is an investment, elevating your home prepares your property against foods and lowers food insurance premiums. When a home is properly elevated, the lowest foor should be above the BFE. Areas below the BFE can be used for parking, storage, or access to the house. DID YOU KNOW? \nElevating your home \nmay reduce your food insurance premium. \nSECURE Y\nARD \nITEMS U\nnsecure items can be swept away or damaged by foodwaters. \nThey can also be swept into your home, causing damage. \nSecure items in your yard by anchoring them or attaching them to more substantial structures. \nSEAL CRA\nCKS \nAND GAPS C\nheck caulking around windows and doors to make it is not \ncracked, broken, or missing. Fill any holes or gaps around pipes and wires that enter your building. \nSET YOUR HOME OR \nBUILDINGS BACK, \nAWAY FROM WATER Build your home, garage, shed, or other building away from river channels and shore lines. If possible, build on higher ground. ",
"metadata": {
"source": "docs_data/fema_protect-your-home_flooding.pdf",
"page": 3
}
}
{
"page_content": " \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \nDIRECT WATER AWAY \nFROM STRUCTURES If you have a single-family home, make sure your yard \nslopes away from buildings on your property and that water has a place to drain. Clear your gutters, assess drainage issues, or collect water in rain barrels. \nANCHOR FUEL \nTANKS Anchor any fuel tanks to the pad to prevent them from tipping over or foating in a food. Spilled fuel could become a fre hazard. Make sure vents and fll-line openings are above the BFE. Note: This may require permission from your fuel provider. \nFLOODPROOF WALLS Add water-resistant exterior sheathing on walls and seal them to prevent shallow fooding from damaging your home. Cover openings below the BFE and seal all exterior openings around pumping and equipment. \nSECURE \nMANUFACTURED \nHOMES If you have a manufactured home and you want food insurance from the National Flood Insurance Program, your home must be affxed to a permanent foundation so that the wheels and axles do not support its weight and resist fotation, collapse, or side-to-side movement. Your local foodplain manager can help you understand the requirements, and a professional engineer or architect can make sure the anchoring system is designed and installed correctly. REMEMBER: \nSome of these tips may work better together than \nothers. Mitigation measures need to be tailored to your property. Also, not all of these options work together, so talk with an expert who can help you identify which options work best for you. \nAlways consult professionals such as your insurance \nagent, architects, engineers, contractors, or other experts in design and construction before making changes to your home. Your local planning and zoning offce or building department is a good place to start for advice. \nFinally, be kind to your neighbors! Talk to adjacent \nproperty owners before you make changes, since some actions on your property may affect theirs. ",
"metadata": {
"source": "docs_data/fema_protect-your-home_flooding.pdf",
"page": 4
}
}
{
"page_content": " \n \n \n \n \n \n \n \n \n \n \n \n ADDITIONAL RESOURCES \nFEMA, PROTECT YOUR PROPERTY \nLearn how to protect your home or business from natural disasters. fema.gov/protect-your-property \nHOMEOWNER\u2019S GUIDE TO RETROFITTING \nfema.gov/media-library/assets/documents/480 \nREDUCING FLOOD RISK TO RESIDENTIAL BUILDINGS \nTHAT CANNOT BE ELEVATED \nfema.gov/media-library/assets/documents/109669 \nPROTECTING YOUR HOME AND PROPERTY FROM \nFLOOD DAMAGE fema.gov/media-library/assets/documents/21471 \nPROTECTING BUILDING UTILITY SYSTEMS FROM \nFLOOD DAMAGE \nfema.gov/media-library/assets/documents/3729 \nPROTECT YOUR PROPERTY FROM FLOODING \nfema.gov/media-library/assets/documents/13261 ",
"metadata": {
"source": "docs_data/fema_protect-your-home_flooding.pdf",
"page": 5
}
}
{
"page_content": "",
"metadata": {
"source": "docs_data/fema_protect-your-home_flooding.pdf",
"page": 6
}
}We can see that it has split the document by page. As it happens this is not an unreasonable approach for the FEMA documents we are processing, which are very concise guides where each page is a discrete topic, but for other applications, it is usually better to split text at a more granular level using LangChain’s text_splitter.
We can now use our text excerpts to create a database of embeddings …
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Chroma
embedding_model = OpenAIEmbeddings()
embeddings = OpenAIEmbeddings()
vectordb = Chroma.from_documents(all_docs, embedding=embedding_model,persist_directory=vecs_dir)
vectordb.persist()We have chosen a simple option to persist the embeddings to the file system, but it’s worth noting that Chroma supports more options should you have a large number of documents, where performance may be an issue.
Setting up our conversational interface
We will use just one PDF to start with so it’s easier to verify results …
import shutil
from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ConversationBufferMemory
from langchain.llms import OpenAI
from langchain.chat_models import ChatOpenAI
from langchain import PromptTemplate, LLMChain
# Load OPENAI_API_KEY from a .env file
from dotenv import load_dotenv
def setup_model(
vecs_dir: str,
docs_sublist: list,
all_docs: list
) -> ConversationalRetrievalChain:
# Subset for docs we are interested in
docs = []
for d in all_docs:
d_dict = vars(d)
# Create vector DB directory
if os.path.exists(vecs_dir):
shutil.rmtree(vecs_dir)
os.makedirs(vecs_dir)
# Choose our models
embedding_model = OpenAIEmbeddings()
chat_model = ChatOpenAI(temperature=temperature,model_name="gpt-4")
# Calculate embeddings
embeddings = OpenAIEmbeddings()
vectordb = Chroma.from_documents(docs, embedding=embedding_model,persist_directory=vecs_dir)
vectordb.persist()
# Set up chat
memory = ConversationBufferMemory(memory_key="chat_history", input_key='question', output_key='answer', return_messages=True)
pdf_qa = ConversationalRetrievalChain.from_llm(chat_model, vectordb.as_retriever(), memory=memory, \
return_source_documents=True)
return pdf_qa
vecs_dir = './vector_dbs/one_flood_doc'
docs = all_docs
# Subset to one document
docs_sublist = ['fema_protect-your-home_flooding.pdf']
pdf_qa = setup_model(vecs_dir, docs_sublist, docs)In the above, we have chosen GPT-4 for the chat model where the API key was defined in a .env file with variable OPENAI_API_KEY. LangChain offers support for many other models also.
With just a few lines of code, we have set up a conversational interface onto a set of documents, which includes all the power of LLMs. I have developed chatbots over the years and can say that this concise pattern reduces a LOT of complexity.
Kudos to the amazing LangChain package!
Asking our first question
Since we would also like to see the documents referenced in order to validate chat responses, we need to make a slight adjustment to a LangChain method in order to account for an issue when retrieving matched documents when also using chat memory (The solution was suggested here) ...
# A little mod to enable using memory *and* getting docs. See: https://github.com/langchain-ai/langchain/issues/2256#issuecomment-1665188576
import langchain
from typing import Dict, Any, Tuple
from langchain.memory.utils import get_prompt_input_key
def _get_input_output(
self, inputs: Dict[str, Any], outputs: Dict[str, str]
) -> Tuple[str, str]:
if self.input_key is None:
prompt_input_key = get_prompt_input_key(inputs, self.memory_variables)
else:
prompt_input_key = self.input_key
if self.output_key is None:
output_key = list(outputs.keys())[0]
else:
output_key = self.output_key
return inputs[prompt_input_key], outputs[output_key]
langchain.memory.chat_memory.BaseChatMemory._get_input_output = _get_input_output
OK, now we are ready to ask a question about our one PDF document!
def ask_question(
query: str,
qa: object,
output_docs: bool = True
) -> dict:
print(f"\nQuestion: \n{query}")
result = qa({"question": query})
print(f"\nAnswer:\n{result['answer']}")
if output_docs:
for doc in result['source_documents']:
print('\n')
print(json.dumps(vars(doc), indent=4))
return result
ask_question("How do I prepare my home for floods?", pdf_qa)
Question:
How do I prepare my home for floods?
Answer:
There are several steps you can take to prepare your home for floods:
1. Determine the Base Flood Elevation (BFE) for your home. This is how high the water is expected to rise during flooding in high risk areas. Your local floodplain manager can help you find this information.
2. Direct water away from structures. Make sure your yard slopes away from buildings on your property and that water has a place to drain. Clear your gutters, assess drainage issues, or collect water in rain barrels.
3. Anchor fuel tanks to prevent them from tipping over or floating in a flood.
4. Floodproof walls by adding water-resistant exterior sheathing and sealing them to prevent shallow flooding from damaging your home.
5. Secure manufactured homes to a permanent foundation to resist flotation, collapse, or side-to-side movement.
6. Inside the home, prepare or update a list of belongings for insurance claims.
7. Get flood insurance as most homeowners insurance policies don’t cover flood damage.
8. Store valuables and important documents in waterproof or water-resistant containers above the BFE.
9. Elevate utilities above the BFE.
10. Replace carpeting with tiles as they are more flood-resistant.
11. Floodproof basements by sealing walls with waterproofing compounds and consider installing a sump pump.
12. Install flood vents in foundation walls, garages, and other enclosed areas to allow water to flow through and drain out.
13. Use flood-resistant insulation and drywall.
14. Prevent sewage back-up by consulting with a plumber and investing in a sewer backflow valve if applicable.
Remember to consult professionals such as your insurance agent, architects, engineers, contractors, or other experts in design and construction before making changes to your home.That seems very reasonable, let’s see what content was used to generate this summary …
{
"page_content": " \n \n \nPROTECT YOUR \nPROPERTY FROM FLOODING\n \n",
"metadata": {
"page": 0,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}
{
"page_content": " \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n -\n\u2014\n- - -\nOwning a property is one of the most important investments most people make \nin their lives. We work hard to provide a home and a future for ourselves and our loved ones. Why risk losing it when bad weather hits close to home? \nFlooding is the most common and costly natural disaster in the United States \nand can happen anywhere. Just one inch of water can cause $25,000 in damages to your home. \nWhile you can\u2019t prevent a natural disaster from happening, there are ways to secure \nyour property to minimize damage and keep your home and your future safe. \nFirst, determine the Base Flood Elevation (BFE) for your home. The \nBFE is how high the water is expected to rise during fooding in high risk areas. You need to know your BFE because it is used in foodplain management regulations in your community that could affect your home for example, how high above the BFE a home or other building should be built. Your local foodplain manager can help you fnd this information. If you need help fnding your foodplain manager, contact FEMA\u2019s Flood Mapping and Insurance eXchange\n a\nt FEMAMapSpecialist@ \nr\niskmapcds.com or (877) FEMA MAP (1 877 336 2627). \nThe following are some additional steps you can take to protect yourself and your property against foods. ",
"metadata": {
"page": 1,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}
{
"page_content": " \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \nDIRECT WATER AWAY \nFROM STRUCTURES If you have a single-family home, make sure your yard \nslopes away from buildings on your property and that water has a place to drain. Clear your gutters, assess drainage issues, or collect water in rain barrels. \nANCHOR FUEL \nTANKS Anchor any fuel tanks to the pad to prevent them from tipping over or foating in a food. Spilled fuel could become a fre hazard. Make sure vents and fll-line openings are above the BFE. Note: This may require permission from your fuel provider. \nFLOODPROOF WALLS Add water-resistant exterior sheathing on walls and seal them to prevent shallow fooding from damaging your home. Cover openings below the BFE and seal all exterior openings around pumping and equipment. \nSECURE \nMANUFACTURED \nHOMES If you have a manufactured home and you want food insurance from the National Flood Insurance Program, your home must be affxed to a permanent foundation so that the wheels and axles do not support its weight and resist fotation, collapse, or side-to-side movement. Your local foodplain manager can help you understand the requirements, and a professional engineer or architect can make sure the anchoring system is designed and installed correctly. REMEMBER: \nSome of these tips may work better together than \nothers. Mitigation measures need to be tailored to your property. Also, not all of these options work together, so talk with an expert who can help you identify which options work best for you. \nAlways consult professionals such as your insurance \nagent, architects, engineers, contractors, or other experts in design and construction before making changes to your home. Your local planning and zoning offce or building department is a good place to start for advice. \nFinally, be kind to your neighbors! Talk to adjacent \nproperty owners before you make changes, since some actions on your property may affect theirs. ",
"metadata": {
"page": 4,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}
{
"page_content": " \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \nINSIDE THE HOME \nPREPARE OR \nUPDA\nTE A LIST OF \nBELONGINGS Documenting all of your belongings will help with the \ninsurance claims process. Consider taking photos of \nhigh-value items or doing a video walkthrough of your \nho\nme to document its contents. \nGET FLOOD \nINSURANCE Most homeowners insurance \npolicies don\u2019t cover food damage. \nProtect your investment by purchasing food insurance for your home and contents, even if you do not live in \na hi\ngh-risk food zone. \nSTORE \nVALUABLES Store valuables and important documents in waterproof or wa\nter-resistant containers above the BFE \n(preferably on an upper foor). Make copies and store them online or offsite. For more \ninformation on purchasing food insurance through the National Flood \nInsurance Program, visit \nFlo\nodSmart.gov or contact \nyour agent to get coverage. EL\nEVATE UTILITIES \nABOVE THE BFE Elevate or foodproof mechanical units, furnaces, water \nheaters, electrical systems, and other utilities on masonry, concrete, or pressure-treated lumber at least 12 inches above the BFE. \nREPLACE CARPETING \nWITH TILES Tiles are more food-resistant than carpet. Using tile or other food-resistant materials in areas below the BFE can help reduce water damage. \nFLOODPROOF \nBASEMENTS If you have a basement, minimize damage by foodproofng your basement and sealing walls with waterproofng compounds. Consider installing a sump pump. \nINSTALL FLOOD \nVENTS Install food vents in foundation walls, garages, and other enclosed areas to allow water \nto fow through, drain out, an\nd lower the risk of \nstructural damage. \nUSE FLOOD-RESIST\nANT \nINSULATION & DRYWALL Fl\nood-resistant insulation and drywall will help minimize \ndamage and can be easily cleaned and sanitized. \nPREVENT SEW\nAGE \nBACK-UP I\nn some areas, fooding can cause sewage to back \nup through drain pipes in your home. Consult with a plumber and, if applicable, invest in a sewer backfow valve to prevent this potential health hazard. DID YOU KNOW? \nWhen following National Flood \nInsurance Program regulations, \nv\nents can also help lower \ninsu\nrance rates. \n",
"metadata": {
"page": 2,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}The answer looks great and has summarized brilliantly the excerpts that were matched. However, it seems to have missed a page in the source document. This PDF is a short document where *everything* in it is relevant to flood preparation, so losing pages is actually significant.
It goes to show that blindly accepting LLM patterns on the web may give amazing-looking results, but work is needed to make them truly useful.
Document Context (Metadata) Can Be Important
For this scenario, we have a document that relates to one topic, flooding, even if individual text sections might not mention it explicitly. We might get better results if we provide some context with each text excerpt, ie some document metadata.
For our simple test, let’s try prefixing all text excerpts with the filename (fema-protect-your-home-flooding.pdf) with punctuation and suffix removed, plus a text saying “This snippet relates to”. The final prefix will be “This snippet relates to fema protect your home flooding:”, which provides the LLM a bit more context …
def setup_model(
vecs_dir: str,
docs_sublist: list,
all_docs: list,
prefix_file_name_to_chunks: bool = False,
temperature: float = 0.0,
extra_prefix: str = '',
) -> ConversationalRetrievalChain:
# Subset for docs we are interested in
docs = []
for d in all_docs:
d_dict = vars(d)
if d_dict['metadata']['source'].replace('docs_data/','') in docs_sublist:
if len(d.page_content) > 20:
# Add file name to content for more context
if prefix_file_name_to_chunks:
file_clean = re.sub(r'docs_data\/|\.pdf', '', d_dict['metadata']['source'])
file_clean = re.sub(r'\-|\_', ' ', file_clean)
d.page_content = f"{extra_prefix} {file_clean}: {d.page_content}"
docs.append(d)
# Create vector DB directory
if os.path.exists(vecs_dir):
shutil.rmtree(vecs_dir)
os.makedirs(vecs_dir)
# Choose our models
embedding_model = OpenAIEmbeddings()
chat_model = ChatOpenAI(temperature=temperature,model_name="gpt-4")
# Calculate embeddings
embeddings = OpenAIEmbeddings()
vectordb = Chroma.from_documents(docs, embedding=embedding_model,persist_directory=vecs_dir)
vectordb.persist()
# Set up chat
memory = ConversationBufferMemory(memory_key="chat_history", input_key='question', output_key='answer', return_messages=True)
pdf_qa = ConversationalRetrievalChain.from_llm(chat_model, vectordb.as_retriever(), memory=memory, \
return_source_documents=True)
return pdf_qa
docs = all_docs
# Note the argument prefix_file_name_to_chunks=True
pdf_qa = setup_model(vecs_dir, docs_sublist, docs, prefix_file_name_to_chunks=True, extra_prefix="This snippet relates to ")
ask_question("How do I prepare my home for floods?", pdf_qa)Question:
How do I prepare my home for floods?
Answer:
There are several steps you can take to prepare your home for floods:
1. Direct Water Away from Structures: Make sure your yard slopes away from buildings on your property and that water has a place to drain. Clear your gutters, assess drainage issues, or collect water in rain barrels.
2. Anchor Fuel Tanks: Anchor any fuel tanks to the pad to prevent them from tipping over or floating in a flood. Spilled fuel could become a fire hazard.
3. Floodproof Walls: Add water-resistant exterior sheathing on walls and seal them to prevent shallow flooding from damaging your home. Cover openings below the Base Flood Elevation (BFE) and seal all exterior openings around pumping and equipment.
4. Secure Manufactured Homes: If you have a manufactured home, it must be affixed to a permanent foundation so that the wheels and axles do not support its weight and resist flotation, collapse, or side-to-side movement.
5. Elevate Your Home: Elevating your home prepares your property against floods and lowers flood insurance premiums. The lowest floor should be above the BFE.
6. Secure Yard Items: Unsecure items can be swept away or damaged by floodwaters. Secure items in your yard by anchoring them or attaching them to more substantial structures.
7. Seal Cracks and Gaps: Check caulking around windows and doors to make sure it is not cracked, broken, or missing. Fill any holes or gaps around pipes and wires that enter your building.
8. Prepare or Update a List of Belongings: Documenting all of your belongings will help with the insurance claims process.
9. Get Flood Insurance: Most homeowners insurance policies don’t cover flood damage. Protect your investment by purchasing flood insurance for your home and contents.
10. Store Valuables: Store valuables and important documents in waterproof or water-resistant containers above the BFE.
11. Elevate Utilities Above the BFE: Elevate or floodproof mechanical units, furnaces, water heaters, electrical systems, and other utilities.
12. Replace Carpeting with Tiles: Tiles are more flood-resistant than carpet. Using tile or other flood-resistant materials in areas below the BFE can help reduce water damage.
13. Floodproof Basements: Minimize damage by floodproofing your basement and sealing walls with waterproofing compounds. Consider installing a sump pump.
14. Install Flood Vents: Install flood vents in foundation walls, garages, and other enclosed areas to allow water to flow through, drain out, and lower the risk of structural damage.
15. Use Flood-Resistant Insulation & Drywall: Flood-resistant insulation and drywall will help minimize damage and can be easily cleaned and sanitized.
16. Prevent Sewage Back-Up: In some areas, flooding can cause sewage to back up through drain pipes in your home. Consult with a plumber and, if applicable, invest in a sewer backflow valve to prevent this potential health hazard.
Remember to consult professionals such as your insurance agent, architects, engineers, contractors, or other experts in design and construction before making changes to your home.
{
"page_content": "This snippet relates to fema protect your home flooding: \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \nDIRECT WATER AWAY \nFROM STRUCTURES If you have a single-family home, make sure your yard \nslopes away from buildings on your property and that water has a place to drain. Clear your gutters, assess drainage issues, or collect water in rain barrels. \nANCHOR FUEL \nTANKS Anchor any fuel tanks to the pad to prevent them from tipping over or foating in a food. Spilled fuel could become a fre hazard. Make sure vents and fll-line openings are above the BFE. Note: This may require permission from your fuel provider. \nFLOODPROOF WALLS Add water-resistant exterior sheathing on walls and seal them to prevent shallow fooding from damaging your home. Cover openings below the BFE and seal all exterior openings around pumping and equipment. \nSECURE \nMANUFACTURED \nHOMES If you have a manufactured home and you want food insurance from the National Flood Insurance Program, your home must be affxed to a permanent foundation so that the wheels and axles do not support its weight and resist fotation, collapse, or side-to-side movement. Your local foodplain manager can help you understand the requirements, and a professional engineer or architect can make sure the anchoring system is designed and installed correctly. REMEMBER: \nSome of these tips may work better together than \nothers. Mitigation measures need to be tailored to your property. Also, not all of these options work together, so talk with an expert who can help you identify which options work best for you. \nAlways consult professionals such as your insurance \nagent, architects, engineers, contractors, or other experts in design and construction before making changes to your home. Your local planning and zoning offce or building department is a good place to start for advice. \nFinally, be kind to your neighbors! Talk to adjacent \nproperty owners before you make changes, since some actions on your property may affect theirs. ",
"metadata": {
"page": 4,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}
{
"page_content": "This snippet relates to fema protect your home flooding: \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \nOUTSIDE THE HOME \nELEVATE YOUR \nHOME While it is an investment, elevating your home prepares your property against foods and lowers food insurance premiums. When a home is properly elevated, the lowest foor should be above the BFE. Areas below the BFE can be used for parking, storage, or access to the house. DID YOU KNOW? \nElevating your home \nmay reduce your food insurance premium. \nSECURE Y\nARD \nITEMS U\nnsecure items can be swept away or damaged by foodwaters. \nThey can also be swept into your home, causing damage. \nSecure items in your yard by anchoring them or attaching them to more substantial structures. \nSEAL CRA\nCKS \nAND GAPS C\nheck caulking around windows and doors to make it is not \ncracked, broken, or missing. Fill any holes or gaps around pipes and wires that enter your building. \nSET YOUR HOME OR \nBUILDINGS BACK, \nAWAY FROM WATER Build your home, garage, shed, or other building away from river channels and shore lines. If possible, build on higher ground. ",
"metadata": {
"page": 3,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}
{
"page_content": "This snippet relates to fema protect your home flooding: \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \nINSIDE THE HOME \nPREPARE OR \nUPDA\nTE A LIST OF \nBELONGINGS Documenting all of your belongings will help with the \ninsurance claims process. Consider taking photos of \nhigh-value items or doing a video walkthrough of your \nho\nme to document its contents. \nGET FLOOD \nINSURANCE Most homeowners insurance \npolicies don\u2019t cover food damage. \nProtect your investment by purchasing food insurance for your home and contents, even if you do not live in \na hi\ngh-risk food zone. \nSTORE \nVALUABLES Store valuables and important documents in waterproof or wa\nter-resistant containers above the BFE \n(preferably on an upper foor). Make copies and store them online or offsite. For more \ninformation on purchasing food insurance through the National Flood \nInsurance Program, visit \nFlo\nodSmart.gov or contact \nyour agent to get coverage. EL\nEVATE UTILITIES \nABOVE THE BFE Elevate or foodproof mechanical units, furnaces, water \nheaters, electrical systems, and other utilities on masonry, concrete, or pressure-treated lumber at least 12 inches above the BFE. \nREPLACE CARPETING \nWITH TILES Tiles are more food-resistant than carpet. Using tile or other food-resistant materials in areas below the BFE can help reduce water damage. \nFLOODPROOF \nBASEMENTS If you have a basement, minimize damage by foodproofng your basement and sealing walls with waterproofng compounds. Consider installing a sump pump. \nINSTALL FLOOD \nVENTS Install food vents in foundation walls, garages, and other enclosed areas to allow water \nto fow through, drain out, an\nd lower the risk of \nstructural damage. \nUSE FLOOD-RESIST\nANT \nINSULATION & DRYWALL Fl\nood-resistant insulation and drywall will help minimize \ndamage and can be easily cleaned and sanitized. \nPREVENT SEW\nAGE \nBACK-UP I\nn some areas, fooding can cause sewage to back \nup through drain pipes in your home. Consult with a plumber and, if applicable, invest in a sewer backfow valve to prevent this potential health hazard. DID YOU KNOW? \nWhen following National Flood \nInsurance Program regulations, \nv\nents can also help lower \ninsu\nrance rates. \n",
"metadata": {
"page": 2,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}
{
"page_content": "This snippet relates to fema protect your home flooding: \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n -\n\u2014\n- - -\nOwning a property is one of the most important investments most people make \nin their lives. We work hard to provide a home and a future for ourselves and our loved ones. Why risk losing it when bad weather hits close to home? \nFlooding is the most common and costly natural disaster in the United States \nand can happen anywhere. Just one inch of water can cause $25,000 in damages to your home. \nWhile you can\u2019t prevent a natural disaster from happening, there are ways to secure \nyour property to minimize damage and keep your home and your future safe. \nFirst, determine the Base Flood Elevation (BFE) for your home. The \nBFE is how high the water is expected to rise during fooding in high risk areas. You need to know your BFE because it is used in foodplain management regulations in your community that could affect your home for example, how high above the BFE a home or other building should be built. Your local foodplain manager can help you fnd this information. If you need help fnding your foodplain manager, contact FEMA\u2019s Flood Mapping and Insurance eXchange\n a\nt FEMAMapSpecialist@ \nr\niskmapcds.com or (877) FEMA MAP (1 877 336 2627). \nThe following are some additional steps you can take to protect yourself and your property against foods. ",
"metadata": {
"page": 1,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}Bingo! That does now seem to have captured the important pages from the PDF and summarized them nicely. Obviously, a very crude approach, a more formal method using metadata instead of just the filename would be better. It might also be more elegant to use a template rather than just prefixing, but it does illustrate how a little context like this can help.
What about if we now use all documents in our set …
vecs_dir = './vector_dbs/all_docs'
docs = all_docs
# Note the argument prefix_file_name_to_chunks=True
pdf_qa = setup_model(vecs_dir, all_docs_list, docs, prefix_file_name_to_chunks=True, extra_prefix="This snippet relates to ")
result = ask_question("How do I prepare my home for floods?", pdf_qa)Question:
How do I prepare my home for floods?
Answer:
There are several steps you can take to prepare your home for floods:
1. Create an emergency plan for your family and practice it regularly. When a storm is approaching, evacuate and move your car to higher ground.
2. Purchase flood insurance for your home and its contents, even if you do not live in a high-risk flood zone.
3. Document your belongings. This will help with the insurance process if you need to file a claim.
4. Store valuables and important documents above the Base Flood Elevation (BFE) in waterproof or water-resistant containers.
5. Elevate appliances and utilities such as water heaters, washers, dryers, and electric panels on higher floors to prevent them from getting damaged by flood water.
6. Use flood-resistant materials for insulation, drywall, and floor coverings like tile to minimize damage.
7. Make sure your yard slopes away from buildings on your property and that water has a place to drain.
8. Anchor any fuel tanks to prevent them from tipping over or floating in a flood.
9. Add water-resistant exterior sheathing on walls and seal them to prevent shallow flooding from damaging your home.
10. If you have a manufactured home, make sure it is affixed to a permanent foundation.
11. Elevate your home above the BFE.
12. Secure items in your yard by anchoring them or attaching them to more substantial structures.
13. Check caulking around windows and doors to make sure it is not cracked, broken, or missing. Fill any holes or gaps around pipes and wires that enter your building.
14. Build your home, garage, shed, or other building away from river channels and shore lines. If possible, build on higher ground.
15. Replace carpeting with tiles as they are more flood-resistant.
16. Floodproof your basement and seal walls with waterproofing compounds. Consider installing a sump pump.
17. Install flood vents in foundation walls, garages, and other enclosed areas to allow water to flow through, drain out, and lower the risk of structural damage.
18. Use flood-resistant insulation and drywall.
19. Prevent sewage back-up by consulting with a plumber and investing in a sewer backflow valve if applicable.
Remember to consult professionals such as your insurance agent, architects, engineers, contractors, or other experts in design and construction before making changes to your home.
{
"page_content": "This snippet relates to fema protect your property storm surge: This snippet relates to fema protect your property storm surge: This snippet relates to fema protect your property storm surge: This snippet relates to fema protect your property storm surge: \n \n \n \n \n \n \n \n \n INSIDE YOUR HOME\nHAVE A \nPLANCreate an emergency plan for your family and practice it regularly. \nWhen a storm is approaching, evacuate and move your car to higher ground. According to the National Weather Service , just two feet \nof water can move a vehicle. Visit Ready.gov/plan to learn more.\nGET FLOOD \nINSURANCE Most homeowners insurance policies don\u2019t cover flood damage. Protect your investment by purchasing flood insurance \nfor your home and its contents. \nDo so even if you do not live in \na high-risk flood zone.For more information on purchasing \nflood insurance through the \n \nNational Flood Insurance Program, visit FloodSmart.gov or contact \n \nyour agent to \nget coverage.\n \n PREPARE OR \nUPDATE A LIST OF \n YOUR HOME\u2019S \nCONTENTSDocument your belongings. This will give you peace of mind \nand help with the insurance process if you need to file a claim. Consider documenting your home\u2019s contents visually. You can either take photos of high-value items or walk through your home and videotape each room\u2019s belongings.\nSTORE \nVALUABLESStore valuables and important documents above the BFE (preferably on an upper floor). Place them in waterproof or \nwater-resistant containers. Also, make copies and store them \nonline or offsite.\nELEVATE \nAPPLIANCES AND \nUTILITIES ABOVE \nTHE BFEKeep appliances and utilities such as water heaters, washers, dryers, and \nelectric panels on higher \nfloors. It can prevent them from getting damaged or ruined by flood water.\nTalk to your floodplain \nmanager about how high to elevate your utilities. Many coastal \ncommunities have codes that require utilities to be elevated 12 inches or more above the BFE, called freeboard. \nUSE FLOOD- \nRESISTANT \nMATERIALS\nKNOW YOUR \nPROPERTY AND \nNEIGHBORHOODFlood-resistant insulation, drywall, and floor coverings like tile \nwill help minimize damage and are easier to clean and sanitize.\nIf you are moving near a large lake or the ocean, talk with \nneighbors. Find out about any issues they\u2019ve had, or any \nmitigation measures they have taken. Take the time to look \nup flood information. You can find flood maps for coastal areas on the National Flood Hazard Layer .ADDITIONAL RESOURCES\nFEMA, Homebuilder\u2019s Guide to Coastal Construction\nFEMA, Homeowner\u2019s Guide to Retrofitting\nFEMA, Recommended Residential Construction for Coastal Areas\nNATIONAL HURRICANE CENTER, Storm Surge Overview\nNATIONAL HURRICANE CENTER, Storm Surge Inundation Map\nREMEMBER: \nSome of these tips may work better than others. Tailor \nany mitigation measures to meet your property\u2019s needs.\nAlways consult professionals, such as your insurance agent, architects, \nengineers, contractors, or other experts in design and construction, \nbefore making changes to your home. Your local planning and zoning \noffice or building department is a good place to start for advice.\n",
"metadata": {
"page": 3,
"source": "docs_data/fema_protect-your-property-storm-surge.pdf"
}
}
{
"page_content": "This snippet relates to fema protect your home flooding: This snippet relates to fema protect your home flooding: This snippet relates to fema protect your home flooding: This snippet relates to fema protect your home flooding: \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \nDIRECT WATER AWAY \nFROM STRUCTURES If you have a single-family home, make sure your yard \nslopes away from buildings on your property and that water has a place to drain. Clear your gutters, assess drainage issues, or collect water in rain barrels. \nANCHOR FUEL \nTANKS Anchor any fuel tanks to the pad to prevent them from tipping over or foating in a food. Spilled fuel could become a fre hazard. Make sure vents and fll-line openings are above the BFE. Note: This may require permission from your fuel provider. \nFLOODPROOF WALLS Add water-resistant exterior sheathing on walls and seal them to prevent shallow fooding from damaging your home. Cover openings below the BFE and seal all exterior openings around pumping and equipment. \nSECURE \nMANUFACTURED \nHOMES If you have a manufactured home and you want food insurance from the National Flood Insurance Program, your home must be affxed to a permanent foundation so that the wheels and axles do not support its weight and resist fotation, collapse, or side-to-side movement. Your local foodplain manager can help you understand the requirements, and a professional engineer or architect can make sure the anchoring system is designed and installed correctly. REMEMBER: \nSome of these tips may work better together than \nothers. Mitigation measures need to be tailored to your property. Also, not all of these options work together, so talk with an expert who can help you identify which options work best for you. \nAlways consult professionals such as your insurance \nagent, architects, engineers, contractors, or other experts in design and construction before making changes to your home. Your local planning and zoning offce or building department is a good place to start for advice. \nFinally, be kind to your neighbors! Talk to adjacent \nproperty owners before you make changes, since some actions on your property may affect theirs. ",
"metadata": {
"page": 4,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}
{
"page_content": "This snippet relates to fema protect your home flooding: This snippet relates to fema protect your home flooding: This snippet relates to fema protect your home flooding: This snippet relates to fema protect your home flooding: \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \nOUTSIDE THE HOME \nELEVATE YOUR \nHOME While it is an investment, elevating your home prepares your property against foods and lowers food insurance premiums. When a home is properly elevated, the lowest foor should be above the BFE. Areas below the BFE can be used for parking, storage, or access to the house. DID YOU KNOW? \nElevating your home \nmay reduce your food insurance premium. \nSECURE Y\nARD \nITEMS U\nnsecure items can be swept away or damaged by foodwaters. \nThey can also be swept into your home, causing damage. \nSecure items in your yard by anchoring them or attaching them to more substantial structures. \nSEAL CRA\nCKS \nAND GAPS C\nheck caulking around windows and doors to make it is not \ncracked, broken, or missing. Fill any holes or gaps around pipes and wires that enter your building. \nSET YOUR HOME OR \nBUILDINGS BACK, \nAWAY FROM WATER Build your home, garage, shed, or other building away from river channels and shore lines. If possible, build on higher ground. ",
"metadata": {
"page": 3,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}
{
"page_content": "This snippet relates to fema protect your home flooding: This snippet relates to fema protect your home flooding: This snippet relates to fema protect your home flooding: This snippet relates to fema protect your home flooding: \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \nINSIDE THE HOME \nPREPARE OR \nUPDA\nTE A LIST OF \nBELONGINGS Documenting all of your belongings will help with the \ninsurance claims process. Consider taking photos of \nhigh-value items or doing a video walkthrough of your \nho\nme to document its contents. \nGET FLOOD \nINSURANCE Most homeowners insurance \npolicies don\u2019t cover food damage. \nProtect your investment by purchasing food insurance for your home and contents, even if you do not live in \na hi\ngh-risk food zone. \nSTORE \nVALUABLES Store valuables and important documents in waterproof or wa\nter-resistant containers above the BFE \n(preferably on an upper foor). Make copies and store them online or offsite. For more \ninformation on purchasing food insurance through the National Flood \nInsurance Program, visit \nFlo\nodSmart.gov or contact \nyour agent to get coverage. EL\nEVATE UTILITIES \nABOVE THE BFE Elevate or foodproof mechanical units, furnaces, water \nheaters, electrical systems, and other utilities on masonry, concrete, or pressure-treated lumber at least 12 inches above the BFE. \nREPLACE CARPETING \nWITH TILES Tiles are more food-resistant than carpet. Using tile or other food-resistant materials in areas below the BFE can help reduce water damage. \nFLOODPROOF \nBASEMENTS If you have a basement, minimize damage by foodproofng your basement and sealing walls with waterproofng compounds. Consider installing a sump pump. \nINSTALL FLOOD \nVENTS Install food vents in foundation walls, garages, and other enclosed areas to allow water \nto fow through, drain out, an\nd lower the risk of \nstructural damage. \nUSE FLOOD-RESIST\nANT \nINSULATION & DRYWALL Fl\nood-resistant insulation and drywall will help minimize \ndamage and can be easily cleaned and sanitized. \nPREVENT SEW\nAGE \nBACK-UP I\nn some areas, fooding can cause sewage to back \nup through drain pipes in your home. Consult with a plumber and, if applicable, invest in a sewer backfow valve to prevent this potential health hazard. DID YOU KNOW? \nWhen following National Flood \nInsurance Program regulations, \nv\nents can also help lower \ninsu\nrance rates. \n",
"metadata": {
"page": 2,
"source": "docs_data/fema_protect-your-home_flooding.pdf"
}
}That’s done a great job, it surfaced information from our key articles in https://www.fema.gov/sites/default/files/2020-11/fema_protect-your-home_flooding.pdf, plus an article from https://www.fema.gov/sites/default/files/documents/fema_protect-your-property-storm-surge.pdf, the only other document in the set which mentions flooding.
Retrieval Prompt Length
At this point, it’s worth noting that the final summarization prompt which turns document excerpts into a nice answer can be quite long depending on the size and number of excerpts. A continual battle with LLMs is to achieve our goals without breaching token limits. Though it hasn’t had a significant effect for our use case, other scenarios might need to employ contextual compression methods.
Question Context Can Be Important
We intentionally mixed documents related to (i) planning for disasters, and (ii) reacting immediately to dangerous events. This can result in confusing responses that mix both contexts …
result = ask_question("I live next to a canal and see the water rising right now, what should I do?", pdf_qa, output_docs=True)Question:
I live next to a canal and see the water rising right now, what should I do?
Answer:
If you notice water levels rising immediately, it's important to prioritize your safety. Here are some steps you can take:
1. Evacuate: If the water levels are rising rapidly, it's crucial to evacuate to higher ground as soon as possible.
2. Move your car: If you have time, move your car to higher ground. According to the National Weather Service, just two feet of water can move a vehicle.
3. Secure your home: If you have time, secure your home by moving valuables and important documents to an upper floor or at least above the Base Flood Elevation (BFE). Place them in waterproof or water-resistant containers.
4. Communicate: Let your family, friends, and neighbors know about the situation. If you have a pre-established meeting point, head there or inform others of your evacuation plans.
5. Contact authorities: Inform local authorities about the situation. They can provide guidance and also alert others who might be at risk.
6. Check flood maps: If you have access to the internet, check flood maps for your area to understand the potential risk.
Remember, these steps are general guidelines. Always follow the advice of local authorities and emergency services during a flood situation.The answer is now a bit of a mixed bag, with some points relating to immediate action “Move your car to higher ground” and some to preparation “Purchase flood insurance”. When people are stressed during an emergency, they probably aren’t thinking of prompt engineering and so we can expect slightly ambiguous inputs.
We could of course solve this problem by using more document metadata to split into sub-groups, but this requires work if that metadata is not available. Another option is to provide more context to the question to indicate if the user is interested in disaster preparation, or needs help immediately. We could build a classifier for this, but in these days of powerful LLMs let’s use zero-shot classification with GPT-4 …
def get_time_context(question):
template = """Does the following question relate to 'planning' or 'taking immediate action': {question}
Answer with one of the following: 'I am planning ahead:' or 'I need to take immediate action:' or 'ambiguous'"""
prompt = PromptTemplate(template=template, input_variables=["question"])
llm = OpenAI()
llm_chain = LLMChain(prompt=prompt, llm=llm)
answer = llm_chain.run(question)
return answer
questions = [
"I live next to a canal and the water is rising, what should I do?",
"Help my roof is blowing off!",
"How do I prevent my roof from blowing off in a hurricane",
"Dog",
"How do I prepare my home for floods?"
]
for q in questions:
print(f"Question: {q}")
answer = get_time_context(q)
print(answer.strip(), "\n")Question: I live next to a canal and the water is rising, what should I do?
I need to take immediate action:
Question: Help my roof is blowing off!
I need to take immediate action:
Question: How do I prevent my roof from blowing off in a hurricane
I am planning ahead.
Question: Dog
Ambiguous
Question: How do I prepare my home for floods?
I am planning ahead. Nice! With very minimal effort, we can easily determine whether a question relates to planning or taking immediate action.
We can now prefix the user question with this …
def get_time_context(question: str) -> str:
template = """Does the following question relate to 'planning' or 'taking immediate action': {question}
Answer with one of the following: 'I am planning ahead:' or 'I need to take immediate action:' or 'ambiguous'"""
prompt = PromptTemplate(template=template, input_variables=["question"])
llm = OpenAI()
llm_chain = LLMChain(prompt=prompt, llm=llm)
answer = llm_chain.run(question)
return answer
def ask_question(
query: str,
qa: object,
output_docs: bool = True,
preprocess_time_context: bool = False
) -> dict:
# First get the time context
if preprocess_time_context:
time_context = get_time_context(query)
if 'planning' in time_context.lower():
query = f"I am planning ahead: {query}"
elif 'take immediate action' in time_context.lower():
query = f"I need to take immediate action: {query}"
print(f"\nQuestion: \n{query}")
result = qa({"question": query})
print(f"\nAnswer:\n{result['answer']}")
if output_docs:
for doc in result['source_documents']:
print('\n')
print(json.dumps(vars(doc), indent=4))
return result
result = ask_question("I live next to a canal and see the water rising, what should I do?", pdf_qa, output_docs=False, preprocess_time_context=True)Which gives …
Question:
I need to take immediate action: I live next to a canal and see the water rising, what should I do?
Answer:
If you see the water rising next to your canal, you should immediately implement your emergency plan. This includes evacuating your home and moving your car to higher ground, as just two feet of water can move a vehicle. You should also ensure that your valuables and important documents are stored above the Base Flood Elevation (BFE), preferably on an upper floor, and placed in waterproof or water-resistant containers. If you have time, consider moving appliances and utilities such as water heaters, washers, dryers, and electric panels to higher floors to prevent them from getting damaged or ruined by flood water. Always prioritize your safety and the safety of your family over protecting property.Great, worked nicely and didn’t try to sell us any insurance, it gave advice that could be acted upon immediately.
Let’s test the converse …
result = ask_question("What can I do to make my house forest fire resistant?", pdf_qa, output_docs=False, preprocess_time_context=True)Question:
I am planning ahead: What can I do to make my house forest fire resistant?
Answer:
There are several steps you can take to make your house resistant to forest fires:
1. Install or replace your roof with a Class A-rated roof with noncombustible coverings. The roof is the most at-risk part of a house in a wildfire due to its size and orientation.
2. Install and replace exterior wall coverings with noncombustible or fire-resistant materials. A minimum fire-resistance rating of one hour for the wall assembly is recommended.
3. Create a 30 feet defensible space around your home by reducing or removing flammable vegetation and using noncombustible materials such as gravel, brick, or concrete.
4. Regularly clean and remove debris from the roof and gutters to reduce the likelihood of something catching on fire on top of your home.
5. Enclose your home's foundation to lower the chance of wind-blown embers getting underneath your home.
6. Plan for access to water by purchasing and installing external sprinkler systems with dedicated power sources or a water tank.
7. Protect large windows from radiant heat by installing multi-pane windows, tempered safety glass, or fireproof shutters.
8. Install highly visible street signs and property addresses to help firefighters and other emergency responders quickly find your property.
9. Seal gaps around openings in exterior walls and roofs with fire-resistant caulk, mortar, or fire-protective expanding foam.
10. Cover exterior attic vents and under-eave vents with metal wire mesh no larger than 1/8 inch to keep embers out.
11. Install a fire block in the gap between the top of framed walls and the foundation of the house to starve the fire of oxygen and prevent it from spreading.
Remember, these tips may work better together than others and need to be tailored to your property. Always consult professionals such as your insurance agent, architects, engineers, contractors, or other experts in design and construction before making changes to your home.Perfect, it provided forest fire preparation information.
Obviously, this would need a lot of testing for anything high-risk that could be used in a real emergency, but it illustrates one way we can enhance performance by enriching the user’s prompt.
Making Sure Answers Only Come from the Provided Documents
For an application where the information can affect safety, it’s important that the information presented only comes from the documents provided. Hallucinations containing incorrect information could have some really dire effects.
Let’s try asking something totally unrelated to disasters …
result = ask_question("How do I make a sponge cake?", pdf_qa, output_docs=False)Which gives …
Question:
How do I make a sponge cake?
Answer:
I'm sorry, but the provided context does not contain information on how to make a sponge cake.Though I am a bit sad not finding out how to make some yummy cake, LangChain took care of this scenario nicely and determines that the question does not relate to the information in the PDF guides provided.
That’s good, no major hallucinations! The last thing I’d want in a disaster is to be told to make a sponge cake and fly a spaceship or something. 😊
Conversation History
One great feature of LangChain is that it seamlessly takes care of conversation history. With one line of code, the model can pick up on previous questions …
result = ask_question("There's a hurricane coming, what should I do?", pdf_qa, output_docs=False)
Question:
I need to take immediate action: There's a hurricane forecast, what should I do?
Answer:
If there's a hurricane forecast, you should take the following immediate actions:
1. Check the flood risk for your area. You can do this by visiting https://msc.fema.gov/portal/home and entering your home’s address.
2. Understand the difference between a Hurricane Watch and a Hurricane Warning. A Hurricane Watch means that hurricane conditions are possible within a specified area and is issued 48 hours in advance of the anticipated onset of tropical-storm-force winds. A Hurricane Warning means that hurricane conditions are expected within a specified area and is issued 36 hours in advance of the anticipated onset of tropical-storm-force winds.
3. Take actions around your house to help reduce the impact of a flooding event. This could include placing sand bags in areas that are most at risk from flooding and elevating mechanical devices like air conditioners, generators, and circuit breakers to minimize the risk of impact by storm surge.
4. Establish a central meeting point so that if the power does go out, people know where to meet up with their loved ones.
5. Prepare a Go-Kit which should include water, a first-aid kit, a flashlight, batteries, a hand-crank/solar powered radio, non-perishable food, cash, a wrench, hand sanitizer, a mylar blanket, ear plugs, and a book and/or deck of cards.
6. Make sure to have items and services in place before a hurricane watch or warning is issued. This could include packing comfort items for loved ones and ensuring that any relatives who require regularly scheduled mental health therapy have their providers aware of the location of the shelter.
7. Store critical documents in Ziploc bags and keep them in a safe place. Alternatively, you can store them on a cloud drive, on a flash drive kept in an alternate location, or in a safety deposit box.
8. Create a Go-Kit for your pet with essential items, including food, water, medication and favorite toys. Make sure your pet is microchipped or at least has ID tags displayed.
9. Familiarize yourself with the predetermined coastal evacuation routes in your area.
10. Plan for communication methods in case phone lines and cellular towers get damaged. This could include text messaging or using social media platforms like Twitter, Facebook, Instagram, etc.Then ask it a question referring to ‘It’ (the hurricane) …
result = ask_question("It's about to arrive, help!", pdf_qa, output_docs=False, preprocess_time_context=True)
Question:
It's about to arrive, help!
Answer:
Here are some immediate actions you should take for safety:
1. If you are in a flood zone, consider evacuating to a safer location. If evacuation is not possible, move to higher ground within your home.
2. Secure your home by closing and boarding up windows. Remove any outdoor items that could be picked up by the wind.
3. Gather your emergency supplies, including your Go-Kit which should include water, a first-aid kit, a flashlight, batteries, a hand-crank/solar powered radio, non-perishable food, cash, a wrench, hand sanitizer, a mylar blanket, ear plugs, and a book and/or deck of cards.
4. If you have pets, ensure you have a Go-Kit for them as well, including food, water, medication and favorite toys. Make sure they are microchipped or have ID tags.
5. Make sure your car is filled with gas and ready to go in case you need to evacuate.
6. Unplug all electrical appliances to prevent power-surge damage.
7. Stay informed about the storm's path and intensity by listening to the radio or checking trusted websites.
8. Communicate with family members and make sure everyone knows the plan. If possible, establish a central meeting point in case you get separated.
9. Store valuable items and important documents in a safe, high place or in waterproof containers.
10. If you have time, place sandbags in areas that are most at risk from flooding.
Remember, your safety is the most important thing. Material items can be replaced, but lives cannot.Perfect, it maintains a history and knows what I’m referring to. Under pressure, one might expect a user to ask follow-up questions, so this ability — implemented with 2 lines of code! — is very important.
Translation
To make the chatbot more versatile, we can explore multilingual support. Many major LLMs such as GPT-4 offer native support for top languages, but performance can vary depending on the language.
Let’s start with Portuguese …
from googletrans import Translator
translator = Translator()
q = translator.translate("How do I prepare my home for floods?", dest='pt')
print("================ RAW =================")
pdf_qa = setup_model(vecs_dir, all_docs_list, docs, prefix_file_name_to_chunks=True, extra_prefix="This snippet relates to ")
result = ask_question(q.text, pdf_qa, output_docs=True, preprocess_time_context=True)
q = translator.translate(result['answer'], dest='en')
print("\n\n=============== TRANSLATED TO ENGLISH USING GOOGLE TRANSLATE =================\n")
print(q.text)================ RAW =================
Question:
I am planning ahead: Como preparar minha casa para enchentes?
Answer:
Existem várias maneiras de preparar sua casa para enchentes:
1. Tenha um plano de emergência: Crie um plano de emergência para sua família e pratique-o regularmente. Quando uma tempestade se aproxima, evacue e mova seu carro para um terreno mais alto.
2. Obtenha seguro contra inundações: A maioria das apólices de seguro residencial não cobre danos causados por inundações. Proteja seu investimento comprando seguro contra inundações para sua casa e seu conteúdo.
3. Prepare ou atualize uma lista do conteúdo da sua casa: Documente seus pertences. Isso lhe dará tranquilidade e ajudará no processo de seguro se você precisar fazer uma reclamação.
4. Armazene objetos de valor: Armazene objetos de valor e documentos importantes acima do BFE (preferencialmente em um andar superior). Coloque-os em recipientes à prova d'água ou resistentes à água.
5. Eleve aparelhos e utilidades acima do BFE: Mantenha aparelhos e utilidades como aquecedores de água, lavadoras, secadoras e painéis elétricos em andares mais altos. Isso pode evitar que sejam danificados ou arruinados pela água da enchente.
6. Use materiais resistentes a inundações: Isolamento resistente a inundações, drywall e revestimentos de piso como azulejos ajudarão a minimizar os danos e são mais fáceis de limpar e higienizar.
7. Conheça sua propriedade e vizinhança: Se você está se mudando para perto de um grande lago ou do oceano, converse com os vizinhos. Descubra quaisquer problemas que eles tiveram ou quais medidas de mitigação eles tomaram.
8. Eleve sua casa: Embora seja um investimento, elevar sua casa prepara sua propriedade contra inundações e reduz os prêmios de seguro contra inundações.
9. Proteja os itens do seu quintal: Itens inseguros podem ser levados ou danificados pelas águas da enchente. Eles também podem ser levados para dentro de sua casa, causando danos.
10. Selo rachaduras e lacunas: Verifique o calafetagem ao redor das janelas e portas para garantir que não esteja rachado, quebrado ou faltando. Preencha quaisquer buracos ou lacunas ao redor de tubos e fios que entram em seu prédio.
Lembre-se, sempre consulte profissionais, como seu agente de seguros, arquitetos, engenheiros, empreiteiros ou outros especialistas em design e construção, antes de fazer alterações em sua casa.
=============== TRANSLATED TO ENGLISH USING GOOGLE TRANSLATE =================
There are several ways to prepare your home for flooding:
1. Have an emergency plan: Create an emergency plan for your family and practice it regularly. When a storm approaches, evacuate and move your car to higher ground.
2. Get flood insurance: Most home insurance policies don't cover flood damage. Protect your investment by purchasing flood insurance for your home and contents.
3. Prepare or update a list of your household contents: Document your belongings. This will give you peace of mind and help with the insurance process if you ever need to file a claim.
4. Store valuables: Store valuables and important documents above the BFE (preferably on an upper floor). Place them in waterproof or water-resistant containers.
5. Elevate appliances and utilities above the BFE: Keep appliances and utilities like water heaters, washers, dryers, and electrical panels on higher floors. This can prevent them from being damaged or ruined by floodwater.
6. Use Flood Resistant Materials: Flood resistant insulation, drywall and floor coverings like tile will help minimize damage and are easier to clean and sanitize.
7. Know your property and neighborhood: If you're moving near a large lake or the ocean, talk to your neighbors. Find out about any issues they've had or what mitigation measures they've taken.
8. Elevate your home: Although it is an investment, elevating your home prepares your property against flooding and reduces flood insurance premiums.
9. Protect Your Yard Items: Unsafe items can be washed away or damaged by floodwaters. They can also be brought into your home, causing damage.
10. Seal Cracks and Gaps: Check the caulking around windows and doors to ensure it isn't cracked, broken, or missing. Fill in any holes or gaps around pipes and wires entering your building.
Remember, always consult with professionals, such as your insurance agent, architects, engineers, contractors, or other design and construction experts, before making changes to your home.Which is the right information. However, with Swahili …
q = translator.translate("How do I prepare my home for floods?", dest='sw')
print("================ RAW =================")
pdf_qa = setup_model(vecs_dir, all_docs_list, docs, prefix_file_name_to_chunks=True, extra_prefix="This snippet relates to ")
result = ask_question(q.text, pdf_qa, output_docs=True, preprocess_time_context=True)
q = translator.translate(result['answer'], dest='en')
print("\n\n=============== TRANSLATED TO ENGLISH USING GOOGLE TRANSLATE =================\n")
print(q.text)================ RAW =================
Question:
Je, ninatayarishaje nyumba yangu kwa mafuriko?
Answer:
Kuna hatua kadhaa unazoweza kuchukua ili kulinda nyumba yako dhidi ya mafuriko:
1. Elekeza Maji Mbali na Majengo: Hakikisha eneo lako linaelekea mbali na majengo kwenye mali yako na kwamba maji yana mahali pa kumwagika. Safisha mifereji yako, tathmini masuala ya mifereji ya maji, au kukusanya maji katika mapipa ya mvua.
2. Nanga Mizinga ya Mafuta: Nanga mizinga yoyote ya mafuta kwenye pedi ili kuzuia kuzunguka au kuogelea katika mafuriko. Mafuta yaliyomwagika yanaweza kuwa hatari ya moto. Hakikisha kuwa matundu ya uingizaji hewa na ufunguzi wa mstari wa kujaza yako juu ya BFE (Base Flood Elevation). Kumbuka: Hii inaweza kuhitaji ruhusa kutoka kwa mtoa huduma wako wa mafuta.
3. Kinga Kuta dhidi ya Mafuriko: Ongeza sheathing inayostahimili maji kwenye kuta na uzibane ili kuzuia mafuriko ya kina kifupi kuharibu nyumba yako. Funika ufunguzi chini ya BFE na ziba ufunguzi wote wa nje karibu na pampu na vifaa.
4. Thibitisha Nyumba Zilizotengenezwa: Ikiwa una nyumba iliyotengenezwa na unataka bima ya mafuriko kutoka kwa Programu ya Bima ya Mafuriko ya Kitaifa, nyumba yako lazima iwe imewekwa kwenye msingi wa kudumu ili magurudumu na axles zisitoe uzito wake na kuzuia kuelea, kuanguka, au harakati ya upande kwa upande.
5. Kumbuka: Baadhi ya vidokezo hivi vinaweza kufanya kazi vizuri pamoja kuliko vingine. Hatua za kupunguza madhara zinahitaji kubinafsishwa kwa mali yako. Pia, sio chaguo zote hufanya kazi pamoja, kwa hivyo ongea na mtaalamu anayeweza kukusaidia kutambua chaguo zipi zinafanya kazi vizuri kwako.
Daima shauriana na wataalamu kama wakala wako wa bima, wasanifu, wahandisi, wakandarasi, au wataalamu wengine katika kubuni na ujenzi kabla ya kufanya mabadiliko kwenye nyumba yako. Ofisi yako ya mipango na mipango ya eneo au idara ya ujenzi ni mahali pazuri pa kuanza kwa ushauri.
Mwishowe, kuwa mwema kwa majirani zako! Zungumza na wamiliki wa mali jirani kabla ya kufanya mabadiliko, kwani vitendo fulani kwenye mali yako vinaweza kuathiri yao.
=============== TRANSLATED TO ENGLISH USING GOOGLE TRANSLATE =================
There are several steps you can take to protect your home from flooding:
1. Direct Water Away from Buildings: Make sure your area faces away from buildings on your property and that water has a place to drain. Clean your gutters, assess drainage issues, or collect water in rain barrels.
2. Anchor Oil Tanks: Anchor any oil tanks to a pad to prevent them from rolling or swimming in a flood. Spilled oil can be a fire hazard. Make sure the vents and fill line opening are above the BFE (Base Flood Elevation). Note: This may require permission from your fuel supplier.
3. Protect Walls from Flooding: Add waterproof sheathing to walls and seal them to prevent shallow flooding from damaging your home. Cover the opening under the BFE and cover all external openings around the pump and equipment.
4. Verify Manufactured Homes: If you have a manufactured home and want flood insurance from the National Flood Insurance Program, your home must be placed on a permanent foundation so that the wheels and axles do not give off its weight and prevent floating, falling, or movement of side by side.
5. Note: Some of these tips may work better together than others. Mitigation measures need to be customized to your property. Also, not all options work together, so talk to a professional who can help you figure out which options work best for you.
Always consult with professionals such as your insurance agent, architects, engineers, contractors, or other professionals in design and construction before making changes to your home. Your local planning and development office or building department is a good place to start for advice.
Finally, be kind to your neighbors! Talk to neighboring property owners before making changes, as certain actions on your property may affect theirs.The results look very plausible but are missing important information. Text excerpts matched are missing some key flood sections and we also have snippets related to wildfires when we asked about floods. Basically, using embeddings created from English documents with Swahili questions doesn’t perform well. Not unreasonable, though it did work well for a Latin language Portuguese.
Perhaps a more robust approach would be to first detect language and translate it into English with Google Translate, then convert the response back into the prompt language.
Let’s add this to our chat interface …
def get_time_context(question: str) -> str:
template = """Does the following question relate to 'planning' or 'taking immediate action': {question}
Answer with one of the following: 'I am planning ahead:' or 'I need to take immediate action:' or 'ambiguous'"""
prompt = PromptTemplate(template=template, input_variables=["question"])
llm = OpenAI()
llm_chain = LLMChain(prompt=prompt, llm=llm)
answer = llm_chain.run(question)
return answer
def ask_question(
query: str,
qa: object,
output_docs: bool = True,
preprocess_time_context: bool = False,
auto_translate: bool = False,
) -> dict:
# Auto-translate
if auto_translate:
lang = translator.detect(query)
lang = lang.lang
print(f"Language detected: {lang}")
q = translator.translate(query, dest='en')
query = q.text
# First get the time context
if preprocess_time_context:
time_context = get_time_context(query)
if 'planning' in time_context.lower():
query = f"I am planning ahead: {query}"
elif 'take immediate action' in time_context.lower():
query = f"I need to take immediate action: {query}"
print(f"\nQuestion: \n{query}")
result = qa({"question": query})
# Translate back to source language
if auto_translate:
q = translator.translate(result['answer'], dest=lang)
result['answer'] = q.text
print(f"\nAnswer:\n{result['answer']}")
if output_docs:
for doc in result['source_documents']:
print('\n')
print(json.dumps(vars(doc), indent=4))
return result
# Translate to Swahili
q = translator.translate("How do I prepare my home for floods?", dest='sw')
print("================ RAW =================")
result = ask_question(q.text, pdf_qa, output_docs=False, preprocess_time_context=True, auto_translate=True)
q = translator.translate(result['answer'], dest='en')
print("\n\n=============== TRANSLATED TO ENGLISH USING GOOGLE TRANSLATE =================\n")
print(q.text)================ RAW =================
Language detected: sw
Question:
I am planning ahead: How do I prepare my home for flooding?
Answer:
Kuna hatua kadhaa unazoweza kuchukua ili kuandaa nyumba yako kwa mafuriko yanayoweza kutokea:
1. Tengeneza mpango wa dharura kwa ajili ya familia yako na uufanyie kazi mara kwa mara. Dhoruba inapokaribia, ondoka na usogeze gari lako hadi sehemu ya juu.
2. Nunua bima ya mafuriko kwa ajili ya nyumba yako na vilivyomo, hata kama huishi katika eneo lenye hatari kubwa ya mafuriko.
3. Andika vitu vyako. Hii itasaidia katika mchakato wa bima ikiwa unahitaji kuwasilisha dai.
4. Hifadhi vitu vya thamani na hati muhimu juu ya Mwinuko wa Mafuriko ya Msingi (BFE), ikiwezekana kwenye ghorofa ya juu, katika vyombo visivyo na maji au visivyo na maji.
5. Kuinua vifaa na huduma kama vile hita za maji, washers, vikaushio na paneli za umeme kwenye sakafu ya juu ili kuvizuia kuharibiwa na maji ya mafuriko.
6. Tumia nyenzo zinazostahimili mafuriko nyumbani kwako.
7. Hakikisha yadi yako ina mteremko mbali na majengo kwenye mali yako na kwamba maji yana mahali pa kutolea maji.
8. Tia nanga matangi yoyote ya mafuta ili kuyazuia yasitembee juu au kuelea katika mafuriko.
9. Ongeza sheati za nje zinazostahimili maji kwenye kuta na uzifunge ili kuzuia mafuriko ya kina yasiharibu nyumba yako.
10. Ikiwa una nyumba iliyotengenezwa, hakikisha kuwa imewekwa kwenye msingi wa kudumu.
11. Pandisha nyumba yako juu ya BFE.
12. Linda vitu katika yadi yako kwa kuvitia nanga au kuviambatanisha na miundo mikubwa zaidi.
13. Angalia kuzunguka kwa madirisha na milango ili kuhakikisha kuwa haijapasuka, haijavunjwa au kukosa. Jaza mashimo au mapengo yoyote karibu na mabomba na waya zinazoingia kwenye jengo lako.
14. Jenga nyumba yako, karakana, banda, au jengo lingine mbali na njia za mito na ufuo. Ikiwezekana, jenga juu ya ardhi ya juu.
Kumbuka kushauriana na wataalamu kama vile wakala wako wa bima, wasanifu majengo, wahandisi, wakandarasi, au wataalamu wengine wa usanifu na ujenzi kabla ya kufanya mabadiliko kwenye nyumba yako.
=============== TRANSLATED TO ENGLISH USING GOOGLE TRANSLATE =================
There are several steps you can take to prepare your home for potential flooding:
1. Make an emergency plan for your family and work on it regularly. When a storm approaches, take off and move your car to the top.
2. Buy flood insurance for your home and contents, even if you don't live in a high flood risk area.
3. Write your items. This will help with the insurance process if you need to file a claim.
4. Store valuables and important documents above the Base Flood Elevation (BFE), preferably on an upper floor, in waterproof or waterproof containers.
5. Elevate equipment and services such as water heaters, washers, dryers and electrical panels on the upper floor to prevent them from being damaged by flood water.
6. Use flood resistant materials in your home.
7. Make sure your yard is sloped away from the buildings on your property and that the water has a place to drain.
8. Anchor any fuel tanks to prevent them from rolling over or floating in flooding.
9. Add waterproof outer sheathing to the walls and seal them to prevent deep flooding from damaging your home.
10. If you have a manufactured home, make sure it is placed on a permanent foundation.
11. Raise your house above the BFE.
12. Secure things in your yard by anchoring them or attaching them to larger structures.
13. Check window and door surrounds to make sure they are not cracked, broken or missing. Fill any holes or gaps around pipes and wires entering your building.
14. Build your house, garage, shed, or other building away from river and beach paths. If possible, build on higher ground.
Remember to consult with professionals such as your insurance agent, architects, engineers, contractors, or other design and construction professionals before making changes to your home.Much better, using Google Translate to translate Swahili automatically to English and then translating the answer back to Swahili gives us all the required information from our set of PDFs. For a risk-critical use-case like disaster response, this would need a LOT of testing with native speakers to ensure safety of course, but it shows promise towards being multilingual.
Asking a Wider Set of Questions
OK, let’s take the final version for a spin and ask more disaster-related questions …
# Set up the model again or our Swahili conversation history will be used
vecs_dir = './vector_dbs/all_docs'
docs = all_docs
pdf_qa = setup_model(vecs_dir, all_docs_list, docs, prefix_file_name_to_chunks=True, extra_prefix="This snippet relates to ")
questions = [
"What should I do for pets if there is a forest fire?",
"My roof is blowing off!!! Help, what should I do?!!!",
"I live next to a brook, should I worry about floods?",
"If I live inland, will sea level affect me?",
"Is there a cheap way to get flood insurance?",
"We're in an Earthquake, what should we do!!???",
"What things should I have in an emergency kit?",
]
for q in questions:
print("\n ===================== ")
ask_question(q, pdf_qa, output_docs=False, preprocess_time_context=True, auto_translate=True)=====================
Language detected: en
Question:
I need to take immediate action: What should I do for pets if there is a forest fire?
Answer:
If there is a forest fire, you should have a Go-Kit for your pet(s) which should include food, water, medication, medical records kept in Ziploc bags, and favorite toys (if applicable). Include your veterinarian’s contact information and be sure that your pet is microchipped or at least has the proper ID tags displayed. If you need to evacuate, take your pet with you if possible. If you cannot take your pet with you, ensure they are in a safe place with access to food and water.
=====================
Language detected: en
Question:
I need to take immediate action: My roof is blowing off!!! Help, what should I do?!!!
Answer:
The FEMA brochures do not provide specific instructions for immediate action if your roof is blowing off. However, it is generally recommended to seek immediate shelter in a safe area of your home, away from windows and potential falling debris. After the severe wind event, you should contact a professional to assess and repair the damage. It's also important to contact your insurance company to report the damage.
=====================
Language detected: en
Question:
I live next to a brook, should I worry about floods?
Answer:
Yes, you should be concerned about floods if you live next to a brook. Flooding can happen anywhere and is not limited to high-risk areas. Heavy rainfall or poor drainage can cause flooding, especially in low-lying areas. It's important to take steps to protect your home, such as elevating your home, securing yard items, and ensuring proper drainage. You should also consider purchasing flood insurance, as federal disaster assistance may be limited or unavailable without a presidential disaster declaration.
=====================
Language detected: en
Question:
If I live inland, will sea level affect me?
Answer:
Sea level changes primarily affect coastal areas, causing increased flooding, erosion, and storm damage. However, if you live inland, you may still experience indirect effects. Changes in sea level can alter weather patterns, potentially leading to increased rainfall or drought in some areas. Additionally, if you rely on resources from coastal areas, such as seafood or tourism, changes in sea level could impact these industries. However, the snippet does not provide specific information on how sea level changes might affect those living inland.
=====================
Language detected: en
Question:
I am planning ahead: Is there a cheap way to get flood insurance?
Answer:
Yes, you can purchase flood insurance through the National Flood Insurance Program. Most homeowners insurance policies don’t cover flood damage, so it's important to protect your investment by purchasing flood insurance for your home and its contents. This is recommended even if you do not live in a high-risk flood zone. For more information on purchasing flood insurance, you can visit FloodSmart.gov or contact your insurance agent to get coverage. Planning ahead and purchasing flood insurance before a storm or flood event is a cost-effective way to protect your property.
=====================
Language detected: en
Question:
I need to take immediate action: We're in an Earthquake, what should we do!!???
Answer:
When an earthquake starts, you should "Drop, Cover, and Hold On!" This means you should drop to the ground, cover your head and neck with your arms, and if a safer place is nearby, crawl to it and hold on. If you are in bed, stay there and cover your head and neck with a pillow. If you are in a moving vehicle, stop as quickly and safely as possible and stay in the vehicle. Avoid stopping near or under buildings, trees, overpasses, and utility wires. Proceed cautiously once the earthquake has stopped, avoiding roads, bridges, or ramps that the earthquake may have damaged.
=====================
Language detected: en
Question:
I am planning ahead: What things should I have in an emergency kit?
Answer:
An emergency kit should include the following items:
- Water (one gallon per person, per day) and/or water purification tablets
- A first-aid kit
- A flashlight
- Batteries
- A hand-crank/solar-powered radio
- Nonperishable food
- Medications/prescription glasses
- Cash (in small denominations)
- A wrench (to turn off utilities)
- Hand sanitizer
- A Mylar blanket
- Ear plugs
- A book/deck of cards
If you have an infant, then include baby food, diapers, formula, etc. If you have pets, consider having a kit for them that includes food, water, medication, medical records, and a favorite toy.
In addition, consider including portable charging devices in your kit as this will allow you to charge from a motor vehicle. If a loved one is comforted by a certain item (a blanket, photograph, stuffed animal, etc.), be sure to pack the item when evacuating. If a relative requires regularly scheduled mental health therapy, make sure that his/her mental health provider is aware of the location of the shelter.I think those are pretty amazing, spot checking a few and they seem to capture key content in the PDF documents used.
[ Based on the above, my disaster kit now has Ear plugs! ]
Evaluating Performance
In this analysis, we have only spot-checked how well our chat interface is returning critical information for a small set of FEMA PDF documents. This is good to illustrate some concepts, but for a production application, we would need something better than spot-checking. Luckily, LangChain has provided a set of evaluators, and of particular interest is a Streamlit application that generates question-answer pairs automatically and then uses these to evaluate retrieval chains, allowing developers to experiment with some of the parameters involved. I haven’t yet tried this yet, but the idea of generating evaluation data automatically using LLMs seems a great way to build scaffolding for a more systematic approach for testing our FEMA disaster bot.
Conclusions
Using a set of 34 PDF documents on the FEMA website, we were easily able to build a multilingual conversational interface using LangChain and GPT-4 that was able to answer a wide range of questions related to disaster preparedness and safety protocols. However, using this common document retrieval LangChain pattern for a high-risk safety critical chatbot faces the same challenges as found with semantic search. Even with advanced LLM embeddings capturing more nuance in natural language, it’s still quite easy to lose important content.
For our FEMA disaster bot use case, this was due to:
- Mixed context documents — Without including document metadata, responses mixed information from different contexts. Simply prefixing document names improved performance for our scenario, and more sophisticated metadata strategies could be applied. Also, adding a zero-shot LLM classifier to enrich the user question was helpful.
- Underrepresented languages in LLMs —Relying on LLM translation can result in poor performance for less well-represented languages such as Swahili. Adding automatic Google translation improved performance for our use case.
We explored crude methods to address the above issues, but for a production chatbot, more advanced techniques and other mitigation found through testing and validation would be required.
Still though, pretty amazing what LLMs can do these days!
References
You can find a notebook with all code here.
