A Very Simple Method of Weather Forecast Using Markov Model Lookup Table
Weather is a very complex event. Even if you just want to forecast rainfall event, it’s quite difficult. But you know, there is a famous quote that reads
History repeat itself
Based on this we will try to predict the weather using Markov Model. But, the first thing is, we must understand the concept of the Markov Model.
Markov Model
What is the Markov Model? It’s a very simple concept. According to Wikipedia, Markov Model is
In probability theory, a Markov model is a stochastic model used to model randomly changing systems. It is assumed that future states depend only on the current state, not on the events that occurred before it (that is, it assumes the Markov property).
The words we must be that we must be pay attention above are
Future states depend only on the current state
How you can come to the current state is not important, what matters is only our current state. This is very simple. To make it clear, let’s give it an example
1 and 1 and 1 and 1 and 1 and 2 and 1
In the string above, there is 3 unique character i.e. “1”, “and”, and “2” (let’s ignore the fact that space (“ “) is a character too for simplicity) it’s mean we have 3 states. To determine how from 1 state changing to another state, the Markov model has a Markov transition matrix. It’s a matrix that mapping the probability of certain case to another state include itself. For this case, the general form of the matrix will be like this
I think you guys all understand the notation in the matrix above. Prob is short for probability anyway. Now how we determine such a probability in the transition matrix? We obtain it from sampling in the observed data. Let’s map all the event of changing state and it’s frequencies in our example string.
First, here all the changing state event
1 and 1 and 1 and 1 and 1 and 2 and 1
[1 -> and] [and ->1] [1 -> and] [and -> 1] [1 -> and] [and -> 1][1 -> and] [and -> 1][1 -> and][and -> 2][2 -> and][and -> 1]
So we got
“1” ChangeTo “and” (5 of 5 = 100%)
“and” ChangeTo “1” (5 of 6 = 83.33% )
“and” ChangeTo “2” (1 of 6 = 16.66% )
“2” ChangeTo “and” (1 of 1 = 100%)
From this, we obtain the transition matrix
With this matrix, we know the fact that if our current state is “1”, the next state or the next word must be “and”, from “and” there is a big chance (83%) it will end up in “1” and little chance (17%) it will be in “2”. From “2”, it's will end up in “and”. Oh, for your information there are several kinds of Markov Model. This kind of Markov Model where the system is assumed to fully observable and autonomous is called Markov Chain.
Predict Weather Using Markov Model
Now we understand what is the Markov model. We know the relation between the quote (“History repeat itself”) and the Markov Model. Actually, one of the methods to do a weather forecast is called analog method. It’s about examining the past state that is the same with our current state and forecasts the weather based on what happened in the past after this state. And so it’s perfect to use the Markov model to apply the analog methods to forecast the weather.
It’s time to move on to our experiment detail. In the typical example of the Markov Model, the example is always about weather prediction but with simple states such as “Sunny”, “Cloudy”, and “Rainy”. In the real weather report or prediction is not so simple like that. Real report or prediction defines temperature, humidity, visibility, and wind in the current or future time besides the weather.
In this experiment, we will treat every unique combination of temperature, humidity, visibility, wind, and weather as a state. In order to avoid the infinite possibility of combination, we grouping and rounding all parameters except the weather (all of the parameters except the weather is in a real number, range 0 to 1 have an infinite number of real numbers between them). Detail of grouping and rounding we will discuss later. Even if we already doing this stuff, the total possible combination of the parameters will be more than a thousand. That’s why we will build a lookup table to mapping all existing states in the Dataset.
When we found a new unique state, we will encode the combination of all parameters in the current state and register it in the table, the current state, and its future state. When we found that state again as the current state, its future state at that time will be appended to the list of the future state of that state along with other future states of that current state.
When we want to use this table to predict future weather, first encode our current state of the weather parameters as a keyword to search it in our table. When we found the identical encoding state, look at the list of its future state. Pick the most frequent encoding state and decode it and that’s it. You got the weather prediction based on the analog method using Markov Model. Very simple. If you don’t find our current state from the table, it’s mean we can't predict this state. That’s why we need to minimize the possible state and need a big dataset to minimize the possibility of finding a new unique state when we want to predict the weather.
The Dataset we will use to build the transition matrix is METAR data. METAR is an alphanumeric code that contains the weather condition and other parameters in a certain area every 1 hour or half an hour.
For the sake of this experiment, I already decode some METAR data from 1 January 2012 to 31 May 2020 in the JSON format. You can download it here. This JSON data contain wind speed and direction, visibility, temperature, dew point, and weather that extracted from those METAR codes. In this experiment, I am using MongoDB and pymongo.
In the first step, let’s build the transition matrix or the lookup table. We use data from 1 January 2012 to 31 December 2019 to build the table. Here the script
In this experiment, we divide a day into 4 sections, namely Morning section (6 AM-12 AM local time), Afternoon section (12 AM-6 PM local time), Night section (6 PM-12 PM), and Midnight section (12 PM-6 AM local time in the next day). The time resolution of our dataset here is half-hourly. So, we need to summarize every parameter value in every section. For all parameters except the weather, we averaging the parameter's value across every section. After averaging, come grouping and rounding. For weather, we pick the most significant weather that happen in the timeframe of the section.
METAR data using GMT or UTC as a time indicator. Local time of our location is GMT+7 so we need to adjust the way we grabbing data from the database (look at line 77 in the code above).
Previously I said that we will talk about rounding and grouping parameter value later. Now, this is the time. For wind speed, we group it in 5 groups:
- < 5 knot wind (encoded it as “<5KT”)
- 5–10 knot wind (encoded it as “5–10KT”)
- 10–15 knot wind (encoded it as “10–15KT”)
- 15–20 knot wind (encoded it as “<15–20KT”)
- 20–25 knot wind (encoded it as “<20–25KT”)
- > 25 knot wind (encoded it as “>25KT”)
For wind direction, we group it in 8 direction
- < 45° (encoded as “Northeast”)
- 45–90° (encoded as “East”)
- 90–135° (encoded as “Southeast”)
- 135–180° (encoded as “South”)
- 180–225° (encoded as “Southwest”)
- 225–270° (encoded as “West”)
- 270–315° (encoded as “Northwest”)
- 315–360° (encoded as “North”)
Yeah, recently I realized that my classification of wind direction is not quite right. But it doesn't matter because we just want to group the wind direction so it will not generate too much unique state.
For visibility, we round the value in the thousandth value and encoded it the same as that, for a visibility with value of>1000 meters. For 500–100, we treat it as 500 meters (encoded it as “500”), and the last for all visibility <500 meters, we treat it as 0 meters (encoded it as “0”). Special for visibility, it has 2 slot state viz minimum visibility and maximum visibility.
For temperature, we round it to the integers (encoded it the same as that).
For Relative Humidity (RH), it’s a bit special because we don’t have RH data in the database. We prognosis it values from temperature and dew point. After that, we round the value in tithing (encoded it the same as that).
For the weather, we pick the most significant weather in the timeframe of the section. The level of significance of the weather is defined here:
The weather code above is encoded to make it shorter. For example, light rain is encoded as -RA, rain as RA, and thunderstorm as TS. The more descent the more significant the weather. Weather is encoded same as this weather code. When there is no significant weather, we encoded it as “NOSIG”.
After we encoded all the parameters, we concatenate all of them and separated it with “|” character. And that’s how we encoding a single state.
After we get the lookup table, it’s time to try this table to predict the weather. We will use data from 1 January 2020 to 31 May 2020. Here the script
This will resulting a new file, result.dat in CSV format. The first column is the current state, the second column is the next state (the truth value), and the third column is the prediction of the next state.
Here is my result.dat generated from the code above.
Wow, there is so much “UNKNOWN” prediction. “UNKNOWN” means the state is so new so we can't predict it. But, some prediction results close to the actual state, not the same. Looks like we need more data for our table. When I get this result with so much “UNKNOWN”, I concluded that I need more data at least 30 years of data (It’s the standard in meteorology or climatology to see the normal pattern of weather in some area).
That’s the application of the analog method to predict the weather using the Markov Model. Next time, maybe I will try using stability index data, or temperature and RH in the lower and upper layer of the atmosphere which is a pretty decisive parameter and is often used by a forecaster to predict the weather.
UPDATE
You can access the full code here.
References:
https://en.wikipedia.org/wiki/Markov_model accessed on 3 June 2020
http://aviation.bmkg.go.id/web/ accessed on 6 June 2020
http://ww2010.atmos.uiuc.edu/(Gh)/guides/mtr/fcst/mth/oth.rxml accessed on 6 June 2020
