Opinion
How Data Science Will Change the Structure of Non-Tech Companies

By Creative Destruction, we mean the processes that increases the efficiency and leads to replacement or downsizing of a business. For example, invention of cars made the job of a coachman (driver of a horse-drawn carriage) obsolete. The whole industry of coachmen was almost wiped out. Note that the aim of such process is increasing the efficiency, and the replacement or downsizing happens only as a side effect of the process. For example, those who invented cars, were in search of faster and easier modes of transfer. The wipe out of coachmen happened as a side-effect.
It has been about 50 years since Information Technology (IT) has become a necessary part of most businesses. By IT we mean the common usage of it; i.e., a department that – depending on the size and type of the company – have one or more of the job titles such as software development, website design, network administration, database management and report development.
Traditionally, the analytics part of medium and large non-tech businesses, more or less falls under the following schema (see the image below): We have a set of internal (on-premises or Cloud) databases. A set of reports are built on top of this data by the report developers. In some circumstances, third-party tools are connected to these internal databases. Then we have a group of non-technical "analysts" who ingest these reports and using Excel files provide solutions to analytical business problems such as forecasting, demand planning, shipments, etc. By "Analyst" we mean any non-technical person outside IT which provides these types of solutions.

This traditional approach to data and analytics is pretty inefficient. To start, these "analysts" do not know SQL, therefore they cannot get data from internal resources, and cannot transform the data. They have to be spoon-fed by IT for their data. Secondly, they do not know programing languages. Therefore, they cannot get data from external resources (e.g., scraping websites or working with APIs). For the same reason, they cannot automate their solutions and have to rely on manual work all the time. Finally – and this is the most important one – they do not have the required theoretical knowledge to provide optimal solutions for these analytical business problems. The optimal solution to these problems usually requires knowledge of one or more of the skills such as optimization algorithms, mathematical models, statistical analysis and artificial intelligence.
Even if third-party tools are used to help them in providing a more accurate solution, since they do not know the inner-working of these tools (i.e., the algorithms) the provided answers and the process are still far from efficient. These tools are treated as a black box. They feed the data, and get output, but they cannot tune it properly, and also usually there are lots of manual work involved.
One question that arises is that "given all these inefficiencies, why are these non-technical people in charge of providing solutions to analytical business problems? Why are these solutions not being provided by the tech-savvy IT people?" If we look at the traditional IT roles mentioned above (software development, website design, etc.), we see that actually none of these roles have the required theoretical knowledge to provide optimal solutions to the analytical business problems either. In other words, although the IT people are technical, not having the required theoretical knowledge have left IT out of the loop of providing solutions to analytical business problems.
But in the last decade, the emergence of Data Science has changed the game. Data scientists are the perfect candidate for providing efficient solutions for analytical business problems:
- They have the required theoretical knowledge (Math, Stats, AI, …).
- They know SQL, so they can get data from internal resources and transform data.
- They know programming, so they can get data from external resources, and also can automate their solutions.

Data science is causing a paradigm shift. It is a fairly new role in IT which is taking over the role of providing solutions to analytical business problems; something that historically has been provided by the business, outside IT, using Excel files or third-party tools, which as we discussed is inefficient. As a consequence, the role of "analyst" (as defined above) would sooner or later be wiped out in non-tech companies (in tech companies they never existed). Of course, we would still have non-technical roles, but those are not the ones that provide solutions to analytical business problems.
While it seems pretty obvious that data scientists are a better fit for providing solutions to analytical business problems, the actual transition is not going to be easy at all. Non-tech companies have been operating like this for a long time and are used to this state of affairs. This type of structural change is not something that happens overnight. Another issue is that, for this transition to work, you need to demonstrate the inefficiency of the current solutions. But the problem is that the current solutions are mostly in Excel files stored in local storage. This makes it really hard to evaluate the current solutions.
Despite all this, this change would happen sooner or later. In some non-tech companies this transition has already started, and others have to move towards this more efficient state, otherwise they are going to stay behind if they keep doing what they have been doing for the past decades.