Twitter Science

Visualizing Crime and Twitter Data for New Zealand

A visualization app developed using R, Shiny, Python and QGIS3.

Twitterati and law enforcement

New Zealand Police has a goal of increasing citizen engagement while improving its effectiveness in policing. One of the ways of quantifying citizen engagement is to look at their twitter habits determining its correlation with the crime in that location.

The fundamental idea behind this exercise was to visualize the location-specific crime and the sentiment of the people in that area by analyzing their tweets. Evidently, for a thinly populated country like New Zealand, the number of people who opt to share their geolocation was too low for such kind of analysis to be trusted.

Therefore, the total number of tweets sent to https://twitter.com/nzpolice over a period of 5 years and the crime data corresponding to each region of New Zealand was used. This article describes the methodology to develop the visualizations. Check the short video to see how it looks.

Extracting tweets

I tried getting the Twitter developers account but it was refused stating it does not comply with their terms and conditions, which is silly. Because I am neither doing anything for the commercial purposes nor am I trying to steal private data. It is just a small experiment. So I used TweetScraper which actually worked quite well, though it is not bug-free.

I extracted tweets sent to NZ Police from 2015 to early 2020. The following command is a typical example.

scrapy crawl TweetScraper -a "query=@nzpolice near:Auckland since:2014-07-01 until:2020-03-31"

This query was not really useful, because out of 80,000+ tweets in this period, only a few hundred of those users had allowed geolocation access from Auckland. No point in using it for analysis.

Tweet Sentiment

I was looking for deep learning to do proper sentiment analysis but was not keen on doing more research in that direction. That is a…