A day in the life of a data engineer
Breaking down the main activities of a data engineer in 2021
Data engineering's role in 2021 has been scaling beyond the scope for a better or for worse. Therefore, multiple definitions of the role are popping up. Does the data engineer do more analytics (aka new role definition, analytics engineer), data pipelines, handling more infrastructure (DevOps), or machine learning engineering? Basically, it’s getting a bit blurry on what an average data engineer will spend his time. However, these categories fall into technical activities, and we often forget that it represents just a chunk of the time spent. In this article, we will break down into different activities what a typical day in the life of a data engineer looks like.
Coding — 30 to 40%
Let’s define what do we actually mean by coding:
- Development of a data pipeline/API/microservice.
- Setup/Maintenance infrastructure
- Fixing bugs, improving code base, documentation
Depending on the project phase, you will work on different coding aspects: new features, debugging, maintenance, and stability.
It’s also worth remembering that coding is not only about “more” (adding lines of code) but also about “less” — removing code. A good example is to look at the top committers of Apache Spark here. We can see that most of them actually have a negative ratio; they removed more lines than adding them!
So no, coding is not the main activity! Multiple studies tend to show that a software engineer will spend 30 to 40% coding daily. That number is totally correlated with my experience.
Project and time management — 20 to 30%
This is a challenging part as it's fairly easy to be unproductive with these. Measuring project/time management efficiency is hard, and you are often not the only variable in the equation.
These activities fall mainly into 2 types :
- Writing: tickets grooming, roadmap, etc.
- Meetings: standup, sprint planning, etc.
