Searching for tweets with Python

Would you like to know how to search for recent tweets with Python?

Apparently, there are a million ways and packages such us: python-twitter, search-tweets-python, snscrape or tweepy ** but finding a straight forward and easy tutorial on simply querying recent tweets on a topic of my choice without any dependencies wasn’t easy. That’s why I wrote this short tutorial. So please continue reading if this applies to you**!

Step 1: Set up your twitter developer account

To get started, you first need to set up your Twitter developer account (if you already have one, skip to Step 2). To create such an account, follow the instructions below:

Navigate to https://developer.twitter.com/en/apply-for-access and sign in with your twitter account if you already have one. If not, sign up for Twitter with a new account. Once you are signed in you should see the following webpage (see image below).

2. Click on "Apply for a developer account". This opens a dialogue where you will be asked how you want to use the Twitter API. In my case I chose the following settings:

3. Describe your intended use: Subsequently, you will be forwarded to a page where you have to state your intended use of your work with the Twitter API. In total, you need to write about 200–600 characters depending on what you intend to do. Simply be honest and describe your use case in the best way possible.

Twitter API access application: Description of intended use

Review your access application and read the terms. Done!
Set up a project and application: Once your access is granted, navigate to https://developer.twitter.com/en/portal/dashboard where you can set up your Twitter API project and an application (in other words a use case).

Copy your Bearer Token: When you are done with the setup, copy your bearer token of the application – you will be needing this token shortly. If you can’t remember your Bearer Token, navigate to "Keys and tokens" and click on regenerate.

Step 2: Python environment setup

In order to send API requests in Python and to be able to use the Twitter API we wont be relying on any Twitter wrapper modules but only on the very handy requests module which can be installed via pip:

pip install requests

Step 3: Prepare Search Twitter Function

It took me a while to find this but there is a dedicated Github repository with code examples for many languages on how to use the Twitter API: https://github.com/twitterdev/Twitter-API-v2-sample-code

My function is heavily inspired by this repositories code but was simplified to fit my needs better:

import requests
import json

#its bad practice to place your bearer token directly into the script (this is just done for illustration purposes)
BEARER_TOKEN = "YOUR BEARER TOKEN HERE"

#define search twitter function
def search_twitter(query, tweet_fields, bearer_token = BEARER_TOKEN):

    headers = {"Authorization": "Bearer {}".format(bearer_token)}

    url = "https://api.twitter.com/2/tweets/search/recent?query={}&amp;{}".format(
        query, tweet_fields
    )

    response = requests.request("GET", url, headers=headers)

    print(response.status_code)

    if response.status_code != 200:
        raise Exception(response.status_code, response.text)
    return response.json()

As you can see, the function is rather short and needs three parameters:

bearer_token [str]: the bearer token you copied in Step 1.
query [str]: This is the actual string that will be used for matching the desired Tweets. These query strings can be simple such as "skateboarding dog" or very complex and powerful. They allow you to filter tweets by hashtags, identify retweets, exclude certain words or phrases, or only include tweets of a specific language. A dedicated description of how to write these query strings can be found here.
tweet_fields [str]: Fields to return in the query, such as attachments, author_id, text, etc. If you for example want the author_id, text, and publishing date of the tweet, the tweet_fields string would look like this:

tweet.fields=text,author_id,created_at"

A very detailed API reference can be found here. Other optional parameters that you can include in the function are described in the reference. Here three very useful ones:

max_results: parameters that specify how many tweets should be returned: A number between 10 and the system limit (currently 100). By default, a request response will return 10 results.
start_time: The oldest UTC timestamp (from most recent seven days) from which the Tweets will be provided. (YYYY-MM-DDTHH:mm:ssZ (ISO 8601/RFC 3339).)
end_time: The newest, most recent UTC timestamp to which the Tweets will be provided. (YYYY-MM-DDTHH:mm:ssZ (ISO 8601/RFC 3339).

Step 4: Run search_twitter function

Finally, let’s run our function and search for all tweets about skateboarding dogs:

#search term
query = "skateboarding dog"

#twitter fields to be returned by api call
tweet_fields = "tweet.fields=text,author_id,created_at"

#twitter api call
json_response = search_twitter(query=query, tweet_fields=tweet_fields, bearer_token=BEARER_TOKEN)

#pretty printing
print(json.dumps(json_response, indent=4, sort_keys=True))

Response:

And this is our response: Firstly we can see that we get a 200 Status, which means that everything run smoothly and secondly we can see that our response is a json object which contains the returned tweets:

200
{
 "data": [
 {
 "author_id": "946476703521570816",
 "created_at": "2021–01–06T18:08:57.000Z",
 "id": "1346881457399197697",
 "text": "@HamillHimself i see your bobsleigh dog (seasonally sporty) and i raise you a skateboarding...cat? https://t.co/d7lrCkDIz3"
 },
 { ...

This was it! I hope this was helpful to some of you! Please visit the links in the further material section below to get more insights on the API documentation.