Photo Credit: Alex Blajan, Unsplash.

A Very Precise & Fast Way to Pull Google Trends Data Automatically

A one-stop-shop script to automatically pull google trends by exact keywords using Pytrends

COVID-19 has provided a boon for “Google Trends” usage in the U.S. (see chart below). After all, it’s a free publicly available tool that provides access to actual search requests across google search engine. More companies have leveraged Google Trends to gain insights on category trend, consumer top searched queries, competitors’ performance amid the “black swan” pandemic event.

Each Google Trends request consists of 5 unique parameters:

• KEYWORDS: Your keywords go here. You can use the prompt under the drop-down box to select the correct keywords.
• COUNTRY: Companies with an international footprint could use this function to get trends across countries.
• DATE INTERVAL: Choose the time range of the trend. Please note that google trends data is normalized based on location and time. Thus, different time ranges will yield various results.
• CATEGORY: Indicate what categories you prefer to search the keywords.
• SEARCH TYPE: Select the source of the search, i.e. Google Search, Image Search, News Search, Google Shopping, and YouTube Search.

CHALLENGES:

There are two main challenges to pull Google trends data in scale.

1. Individual keyword by keyword manually pulling is time-consuming.

Although Google Trends provides the “Compare” function to compare keywords, the downside is that it scales the results from 0 to 100 based on the most popular term entered. The less popular keyword will lose sensitivity quickly if you compare it with a popular one. For example, if you compare “Nike” brand with “Supreme” brand, you will basically get a flat line for “Supreme”. In this situation, you’ll get big errors when reporting “Supreme” search growth trend. Thus, it is suggested to pull “Nike” trend and “Supreme” trend individually and separately.

You can still afford the time until you have too many keywords to pull. E.g. when to compare 50 apparel and footwear brands, you need to download 50 excels and combine them together.

2. Current available automation python methods don’t query exact keywords, meaning they’re not accurate.

There are many available methods to pull google trend data using python. However, none of them builds automatic codes that could pull EXACT KEYWORDS. It’s not unusual that words often have multiple meanings. For example, “Patagonia” could either be a clothing company or a region in South American. Simply search “Patagonia” will give us ambiguous results as it contains both search terms. We need to make sure we search the right keywords.

GOAL:

The goal is to provide a solution to pull google trends data for multiple keywords exactly, individually, and automatically.

Specifically, we’ll pull google trends data for six apparel/footwear brands (Nike, Adidas, Under Armour, Zara, H&M, Louis Vuitton) in three countries (US, UK, Germany). Using Python, there are four steps to achieve this:

• Step 1: Install pytrends API
• Step 2: Get exact keywords
• Step 3: Pull Google trends data by exact keywords by country
• Step 4: Visualize Google trends

Step 1: Install pytrends API

First of all, we need to install that package called “pytrends”, which is designed to pull google trends using python. Simply execute the following code from your Terminal. You could find the comprehensive documentation of the pytrends API here.

pip install pytrends

We will then import the necessary packages.

import pandas as pd
import pytrends
from pytrends.request import TrendReq
pytrend = TrendReq()

Step 2: Get exact keywords

As discussed earlier, we need to be precise on the keywords to avoid ambiguity. Pytrends provides a function called pytrend.suggestions that could return several suggestions for a keyword. Usually, the first suggestion is the most popular one. “mid” column contains those exact keywords we’d like to search.

KEYWORDS=['Nike','Adidas','Under Armour','Zara','H&M','Louis Vuitton'] 
KEYWORDS_CODES=[pytrend.suggestions(keyword=i)[0] for i in KEYWORDS] 
df_CODES= pd.DataFrame(KEYWORDS_CODES)
df_CODES

Step 3: Get Google trends data by exact keywords

Next, we’ll set those 5 parameters.

EXACT_KEYWORDS=df_CODES['mid'].to_list()
DATE_INTERVAL='2020-01-01 2020-05-01'
COUNTRY=["US","GB","DE"] #Use this link for iso country code
CATEGORY=0 # Use this link to select categories
SEARCH_TYPE='' #default is 'web searches',others include 'images','news','youtube','froogle' (google shopping)

Then, we’ll write codes to pull google trends data by exact keywords by country using the above parameters.

Individual_EXACT_KEYWORD = list(zip(*[iter(EXACT_KEYWORDS)]*1))
Individual_EXACT_KEYWORD = [list(x) for x in Individual_EXACT_KEYWORD]
dicti = {}
i = 1
for Country in COUNTRY:
    for keyword in Individual_EXACT_KEYWORD:
        pytrend.build_payload(kw_list=keyword, 
                              timeframe = DATE_INTERVAL, 
                              geo = Country, 
                              cat=CATEGORY,
                              gprop=SEARCH_TYPE) 
        dicti[i] = pytrend.interest_over_time()
        i+=1
df_trends = pd.concat(dicti, axis=1)

And we will do some cleaning work and change those exact keywords back to readable brand names.

df_trends.columns = df_trends.columns.droplevel(0) #drop outside header
df_trends = df_trends.drop('isPartial', axis = 1) #drop "isPartial"
df_trends.reset_index(level=0,inplace=True) #reset_index
df_trends.columns=['date','Nike-US','Adidas-US','Under Armour-US','Zara-US','H&M-US','Louis Vuitton-US','Nike-UK','Adidas-UK','Under Armour-UK','Zara-UK','H&M-UK','Louis Vuitton-UK',
'Nike-Germany','Adidas-Germany','Under Armour-Germany','Zara-Germany','H&M-Germany','Louis Vuitton-Germany'] #change column names

Step 4: Visualize Google trends

In the blink of an eye, we get our google trends data. Finally, let’s visualize Louis Vuitton's google trends across countries. As we could see, Louis Vuitton clearly gets hard hit by COVID-19, just like many other brands and industries.

import seaborn as sns
sns.set(color_codes=True)
dx = df_trends.plot(figsize = (12,8),x="date", y=['Louis Vuitton-US','Louis Vuitton-UK','Louis Vuitton-Germany'], kind="line", title = "Louis Vuitton Google Trends")
dx.set_xlabel('Date')
dx.set_ylabel('Trends Index')
dx.tick_params(axis='both', which='both', labelsize=10)

SUMMARY

Google Trends tool usages see surging growth as companies closely monitor shifting consumer behavior during COVID -19. To help with this, we build a light application to pull Google Trends data precisely and automatically.

Finally, stay positive, thanks for reading!