The world’s leading publication for data science, AI, and ML professionals.

Interactive Time Series with Python and API: Generating Heatmap of Taxis in Singapore

Geospatial visualization is a powerful method of storytelling, especially with heatmaps. In this example, I use a time series heatmap to…

Geospatial visualization is a powerful method of storytelling, especially with heatmaps. In this example, I use a time series heatmap to convey the movement of taxis using coordinates and time. With the blended use of heatmaps and live data from an API, users will be able to analyze real-time taxi movement.

Land Transport Authority (LTA) of Singapore has an API that shows live coordinates of all taxis. It is quite a cool API that shows real-time taxi movement that is refreshed every 30 seconds. However, there is little work done with this API after searching up for more information on it. Hence, in this article, I will walk through the steps on how to collect data from this API and present it in a time series heatmap below using Folium!

Taxis movement on a Friday night in Singapore
Taxis movement on a Friday night in Singapore

Step 1: Importing the relevant packages

import folium
import folium.plugins as plugins
import pandas as pd
import json
import requests
import time
from folium.plugins import HeatMapWithTime
import datetime

Step 2: Pulling Data from API and storing it as a dataset

I start by creating an empty data frame so that I can append the live data into it.

cumulative = pd.DataFrame()

Next, using a simple input function, I am able to key in the duration of taxi movement. This will determine the amount of times the loop will run.

mins = input('How many mins do you want to see? ')
number_of_times = (int(mins)*60)/30

Next, write a for loop to continuously pull data from the API. As the API refreshes every 30 seconds, I use Python time Module to rerun the script every 30 seconds. Meanwhile, I only extract the coordinates and the timestamp from the Geojson object and append it into my pandas data frame. This will allow me to compile past data into a data frame for easy manipulation later on.

startTime = time.time()
for i in range(int(number_of_times)):
    url = ("https://api.data.gov.sg/v1/transport/taxi-availability")
    response = requests.get(url)
    data = response.json()
    df = pd.io.json.json_normalize(data['features'])
    coordinateslist = df['geometry.coordinates'].tolist()
    df1 = pd.DataFrame(coordinateslist)
    result = df1.transpose()
    result.columns = ['coordinates']
    result['Timestamp'] = (df['properties.timestamp'][0])
    cumulative = cumulative.append(result) 
    time.sleep(30)
endTime = time.time()
elapsedTime = endTime - startTime
print("Elapsed Time = %s" % elapsedTime,'seconds')

Step 3: Data Wrangling with for loops

Now it is time to manipulate the data so that it is easy for us to utilize it. I start by splitting the coordinates into latitude and longitude columns.

cumulative['coordinates'] = cumulative['coordinates'].astype(str)
Latitude = []
Longitude = []
coordinates = []
for i in cumulative['coordinates']:
    i = i.split(", ")
    lat = i[1][:-1]
    long = i[0][1:]
    Latitude.append(lat)
    Longitude.append(long)
cumulative['Latitude'] = Latitude
cumulative['Longitude'] = Longitude

Your data frame should resemble the one below before we process it into the format we want in order to plot it into Folium.

Resulting data frame after manipulation
Resulting data frame after manipulation

Next, we need to put the latitude and longitude into a nested list in order for us to use folium.plugins.**HeatMapWithTime** to generate the time series heatmap.

lat_long_list = []
for i in cumulative['Timestamp'].unique():
    temp=[]
    for index, instance in cumulative[cumulative['Timestamp'] == i].iterrows():
        temp.append([instance['Latitude'],instance['Longitude']])
    lat_long_list.append(temp)

Step 4: Creating a time index and formatted it

Next, we have to convert the Timestamp into a datetime format so that we can convert it into a neater and more readable layout. This will also be the index that is used in the play button to show the time and date.

#converting it to datetime format
cumulative['Timestamp']= pd.to_datetime(cumulative['Timestamp'])
#creating a time index
time_index = []
for i in cumulative['Timestamp'].unique():
    time_index.append(i)
#formatting the index
date_strings = [d.strftime('%d/%m/%Y, %H:%M:%S') for d in time_index]

Step 5: Plotting the time series heatmap

Finally, we can generate the map! I choose a dark themed map of Singapore so that the heatmap will look more vibrant.

#Choosing the map type 
m = folium.Map(location=[1.352083,103.819839],zoom_start = 11, tiles="https://tiles.stadiamaps.com/tiles/alidade_smooth_dark/{z}/{x}/{y}{r}.png",attr="Stadia.AlidadeSmoothDark")
#Plot it on the map
HeatMapWithTime(lat_long_list,radius=5,auto_play=True,position='bottomright',name="cluster",index=date_strings,max_opacity=0.7).add_to(m)
# Display the map
m
Your map is done! You can zoom it to see the movement in downtown Singapore.
Your map is done! You can zoom it to see the movement in downtown Singapore.

Summary

  • Step 1: Importing the relevant packages
  • Step 2: Pulling Data from API and storing it as a dataset
  • Step 3: Data Wrangling with for loops
  • Step 4: Creating a time index and formatted it
  • Step 5: Plotting the time series heatmap

Feel free to save it as a html file so that you can send to your friends and colleagues. You can check out the full code on my Github here.


Related Articles