How I tested the Hungarian Election for fraud using Benford’s law
I tested several datasets using Benford’s law and learned about the advantages and pitfalls.
I recently stumbled upon a concept called Benford’s law. I was listening to a danish podcast — and the host introduced the concept that if you sum up the first digit of each number in a dataset, the occurrence of each digit should follow a certain distribution. This, he said, could be used as a way of detecting tax fraud and for flagging fake social media profiles.
What is Benford’s law?
The law basically states that the sum of first digits of any naturally occurring dataset, should follow this distribution: Most numbers, 30%, start with one, 17 % starts with two, and so forth as shown in the first picture.
“That doesn’t sound right,” I thought at first. I learned about Central limit theorem in school and also it just intuitively feels wrong.
Immediately after finishing the podcast, I tested BL on the yearly military spending of all countries in the world since 1970 (as I was looking at that data for other reasons at the time). Taken out of the political context these numbers are seemingly random, and should therefore be a perfect dataset to test. And it turned out to work!
At least the distributions of first digits look quite similar in the figure above. To further test it, I did a chi squared goodness of fit test.
For a significance level of α = 0.05, a comparison with the Benford distribution supports the null hypothesis of no difference between the predicted and observed distributions (χ 2 = 0.970, df = 8, p = 0.9984).
Which means the first digit of military budgets of all countries in the world since 1970 follow Benford’s Law. I still thought it was weird, so I dived into the logic behind it.
Turns out it’s simple math, this type of data tends to conform to a lognormal distribution and the probabilities of first digits can be expressed like this:
I am not going to explain the deeper mathematical reasons why it works but if you are interested you can read about it here , here and here.
How can it detect fraud?
Okay, so the podcast host said that BL could be used for detecting fraud. How does that work?
One of the conditions for a dataset to conform to BL is that the data must be randomly generated, which they are if 1) they are not restricted by max and min, and 2) they are not assigned numbers.
I would suggest it is safe to assume that the lengths of rivers in the world are randomly distributed. The same seems to be the case for the military budgets of the world.
Election data
What about election data then? We would expect it to be random if a specific candidate gets 1000 or 999 votes in a specific polling station. That means election data should follow BL if no manipulation has occurred.
First I wanted to test a legitimate election to compare with the Hungarian election. In Denmark I assume a really low probability for election fraud so I tested the numbers from the last Danish election in 2019. The data was freely available here.
Taking all votes for each party from all polling stations in the country I got a list of 18714 voting numbers.
For a significance level of α = 0.05, a comparison with the Benford distribution supports the null hypothesis of no difference between the predicted and observed distributions (χ 2 = 0.046, df = 8, p = 0.9999).
So the polling station voting data definitely conform to BL.
Let’s take a moment to think about what that means. We would expect a random distribution of numbers to conform to BL but of course these numbers could be manipulated according to BL and fraud or unfairness could also occur in a way that does not affect the first digits. Therefore BL cannot prove the absence of fraud.
So no apparent fraud in the Danish election — as expected — but what about the Hungarian?
Hungarian election data
The EU member Hungary just reelected their prime minister Victor Orban on April 3rd 2022. Some speculated in backlash from voters on account of the war in Ukraine and Orban’s close relationship with Russia’s president Putin, but Orban’s Fidesz/KDNP coalition won an overwhelming majority of 135 out of 199 seats in parliament.
While Orban surely is genuinely favored by a large group of the Hungarian public — several organisations have been raising concerns about the state of democracy in Hungary: Media bias and Election Fraud.
This immense win, and all the concerns about democracy in Hungary makes the election a great candidate for some election forensics.
I didn’t find a publicly available dataset of polling stations for download — but I found constituency voting numbers on the official Hungarian referendum site. To get the data I wrote a small scraper script that can be found on my GitHub page together with the analysis notebook used to create the figures shown in this article. Feel free to try it out for yourself! The scraper gave me a constituency votes dataset of 663 datapoints
This time we see substantial differences, and the null hypothesis is rejected (p value is 0.004 < 0.05).
So in this case the data does not conform to Benford’s distribution. I was so excited when I saw this result. It shows that Victor Orbans win was not genuine. Or what?
Criticism of Benford’s law for election forensics
After finding that data from the Hungarian election didn’t fit BL I did some more research. In 2020 testing elections with Benford’s law got serious criticism from several experts when internet people like myself used it to “prove” that the American presidential election was fraudulent. The heavy criticism was probably also due to the fact that at this time ex-president Donald Trump was trying to discredit the election results, so everyone was on edge about getting it right.
In this article, the Reuters fact checking team contacted several experts in Benford’s law and election forensics. They all agree that BL cannot be used as proof but merely as a red flag that can prompt further investigation. In the specific case of the American 2020 presidential election the small size of the polling stations makes the dataset violate the rule of the data spanning several orders of magnitude. They also said that if you want to use BL you need to do the second digit analysis, as it is less sensitive to constituency size.
After learning this i tested out 2BL i.e. Benford’s law for second digits on the Hungarian election. 2BL works like BL but with a slightly different distribution of digits. Turns out: 2BL analysis conforms very well with with the Hungarian voting data.
So when you look at it this way the Hungarian election conforms pretty well with BL and that indicates there is no reason to raise any red flags on this one.
Reason for the first digit not working
So why did BL test not work on the first digit? Looking at the distribution of votes between the three parties that got seats in parliament. We see that there are a lot of votes between 20K and 40K which would account for the overrepresentation in 2 and 3 as first digits. Next time this is the first plot i will make to see if BL test is even feasible.
Conclusion:
Benford’s law is an interesting and surprising concept — but when using it to detect election fraud you need to be careful — datasets can deviate from Benford’s distribution for other reasons than manipulation and even then it can only be used as a method of raising red flags — not to prove fraud.
I had some fun and learned a lot along the way! Feel free to connect on LinkedIn and GitHub to talk more about election fraud, Data Science or anything in between!
