Why is statistics important in Data Science, Machine learning, and Analytics

Understanding the benefits of having a strong background in statistics as a data scientist

Published in

Towards Data Science

7 min readMay 17, 2022

Statistics, in its broadest sense, refers to a collection of tools and methods for evaluating, interpreting, displaying, and making decisions based on data. Some individuals refer to statistics as the mathematical analysis of technical data.

“A significant constraint on realizing value from Big Data will be a shortage of talent, particularly of people with deep expertise in statistics and machine learning, and the managers and analysts who know how to operate companies by using insights from Big Data “ — McKinsey

In this article, I will attempt to explain why I believe it is essential for data science and machine learning enthusiasts to possess a deeper understanding of statistics. Looking deeper statistics is a form of mathematical analysis that employs multiple quantitative models to produce experimental data or empirical research. Collecting, analyzing, interpreting, and presenting data are all elements of applied mathematics. The mathematical foundations of statistics are linear algebra, differential and integral calculus, and probability theory.

I studied a portion of statistics when completing my python data science qualification, but only later realized that statistics 101 is insufficient. Consequently, I believe that any data scientist worth his or her salt should have a deeper understanding of statistics and be proficient in either “R” or Python.

What exactly is Data Science?

Data science is a field of study that utilizes cutting-edge tools and techniques to uncover hidden patterns and trends, thereby generating valuable insights that can be used to make more informed business decisions. It also encompasses predictive analytics, in which data scientists employ a variety of machine learning or statistical algorithms.

The Data Science Lifecycle

To comprehend the role that statistics play in data science, you must first have a thorough understanding of the data science lifecycle. There are several perspectives on the lifecycle, but I use a simplified one. It consists of the five stages listed below.

Other Lifecycles

The life cycles listed below are more contemporary approaches specific to data science.

The acronym OSEMN, which stands for Obtain, Scrub, Explore, Model, and interpret, represents a five-phase life cycle.
Microsoft TDSP: The Team Data Science Process combines a number of contemporary agile practices . It is comprised of five phases: Business Understanding, Data Acquisition and Understanding, Modeling, Deployment, and Customer Acceptance.

Importance of Statistics for Data Scientists

We can utilize statistical analysis techniques to quantify what we have so instead of sifting through voluminous amounts of data, we can describe it using a few metrics.

Advanced machine learning algorithms in data science utilize statistics to identify and convert data patterns into usable evidence. Data scientists use statistics to collect, evaluate, analyze, and draw conclusions from data, as well as to implement quantitative mathematical models for pertinent variables.
Data science requires both technical skills, such as R and Python programming, and “soft skills,” such as communication and attention to detail.

Listed below are some of the most important skills data scientists must possess to enhance their statistical abilities.

Statistics

Data scientists should make an effort to learn statistics because statistics relate data to the questions organizations face across all disciplines, such as how to increase revenue, limit spending, create efficiencies, and maximize communications, etc.

Data manipulation

Using Excel, R, SAS, Stata, Power Query M, Apache Spark and other systems, data scientists can clean and organize large data sets.

Critical thinking

Data scientists identify and model relationships between dependent and independent variables using linear regression. Data scientists choose procedures with underlying assumptions that are considered during implementation. If assumptions are violated or incorrectly selected, the results will be incorrect.

Organization

Data scientists are constantly inundated with data from a variety of sources and project opportunities. Expertise in statistical functions enables data scientists to work effectively within budget and time constraints. Routine processes also contribute to data security protection.

Problem-solving

In addition to pure computations and fundamental data analysis, data scientists use applied statistics to relate abstract discoveries to real-world problems. Additionally, data scientists utilize predictive analytics to plan future actions. All of this necessitates careful consideration, as well as rational and innovative problem-solving strategies.

A few important use cases:

Statistics in data science, in its roots, find a structure and relations between various unstructured data. Structuring the data helps reveal different valuable insights behind your collected data.

Logistic regression, one of the most widely used classification methods, aids in the prediction of qualitative responses based on observable patterns. The method predicts the values of a currently unknown variable using its relationship and the values of other variables on the graph.

Data analytics and machine learning are built on the understanding of logistic regression, cross-validation, and other techniques that assist the machine in predicting your next move. One such example is when you’re listening to songs on YouTube. You see suggestions of songs you would like even if you’ve never heard them before. The reason behind the suggested songs is statistics.

Clustering is another great example. For example, in the event of a medical emergency, knowing the percentage of people impacted can assist you in devising solutions. In data science, segregating your buyers into various age groups is referred to as clustering. It helps you create adverts and learn more about your intended audience.

Importance of statistics in ML/AI:

Data analysts must understand and establish a comprehensive picture of the data before collecting it in scale for further analysis, such as bivariate, univariate, multivariate, and principal components analysis.

Many machine learning performance measurements, such as precision, accuracy, recall, root mean squared error, f-score, and so on, are based on statistics.

Data exploration is the first and foremost step in the process of data analysis. Data analysts use data visualization and statistical methods to describe dataset characteristics such as size, quantity, and accuracy to better understand the nature of the data.

Data visualization and exploration boost the discovery of new and unexpected insights from data. With this information, statistics help to verify what we already know was debunked, motivating discoveries in different branches of AI.

Data Visualizations and Analytics:

Statistics use tools like pie charts and bar graphs to portray data in a structured format. You can’t reach an accurate and precise conclusion by collecting individual, irrelevant information.

Visualization tools like pie charts, histograms, and bar graphs go a long way in making data more interactive and understandable in extensive data studies. They provide an engaging and easy-to-understand approach to understanding complex data.

Data analytics is the process of analyzing data sets to make decisions based on the available information, which is increasingly done with the help of specialized software and systems. It identifies underlying models and patterns, serves as an input source for Data Visualization, and aids in company improvement by anticipating demands.

We can analyze data by using measures of central tendency. A summary statistic that depicts the dataset’s center point or typical value is known as a measure of central tendency. These measures, often known as the central location of distribution, indicate where most values in a distribution fall. It can be thought of as data tending to gather around a central value. The mean, median, and mode are the three most popular measures of central tendency in statistics. Each of these calculations uses a different method to determine the position of the central point.

These statistical tools aid in the early detection of patterns and make them understandable to even the most inexperienced users. As a result, drawing conclusions and formulating action plans becomes less complicated.

Key Takeaway:

Statistics is essential in real life as well as professional life. It helps you analyze the data given to you and make decisions according to it.

The ability to read pie charts, bar graphs, etc., is facilitated by statistical knowledge, which also aids in data comprehension and, ultimately, leads to enhanced skills in presenting data in a manner that allows not only you but also others to draw conclusions.
It enables you to see trends in any data easily; it enables you to analyze the data effectively; it enables you to reach better and more accurate conclusions.
In ML statistical knowledge allows you to fully understand the effectiveness of your models based on the evaluation. You can simply not understand e.g. R² without it, or any other performance metric.

However, you are not required to attend college or university to take a course in statistics. You could do it easily online. As an executive my perspective and use case for statistics is quite a bit different so I opted for some other courses.

Following are links to some excellent online statistics courses that will help you quickly develop solid foundational skills.

This has been an very high level look at why I think all data scientists should get more into statistics. Let me know what you think in the comments below.