tl; dr: The Clean Coder by Robert Martin
I just finished reading The Clean Coder: A Code of Conduct for Professional Programmers by Robert C. Martin. Incredible book, brand-spanking new coder, ten out of ten. I recommend it to everyone who writes code on a daily basis.
I decided to make some notes that combine advice from the book with some of my own experience in data science and machine learning. I tried to write these notes in an actionable form so it will be easy to make habits from them.
- Take responsibility
- Be responsible for your career
- Use TDD when you are going to re-use code
- Don’t be afraid to say “No”
- Define “Done”
- Show commitment by giving time estimation
- Provide time estimates, not guesses
- Decompose big task into several small tasks
- Use PERT methodology
- Ask your teammates to estimate the required time
- Don’t attend every meeting to which you are invited
- Provide a clear agenda and set of goals to achieve
- Make notes during the meeting
- Do pair programming sessions
- Start using Pomodoro
Let’s look at each in detail.
Professionalism
The thin red line of the book is answering the question, “What does it mean to be a software developer professional?”. Think about it for a second yourself. Could you call yourself a professional? Why? If not, what differentiates you from being a professional?
Take responsibility
My father always told me, “Work as if you own it.” Would you write crappy code, do the minimum work to meet requirements, or produce shallow analysis if the company was your own? Probably not. Make a moral obligation to take responsibility for your actions.
It’s a lot easier to be nonprofessional. Non-professionals don’t have to take responsibility for the job they do.
How could we take responsibility? Robert Martin makes several points:
- Do not harm the function. Make yourself sure that your code does what you want it to do.
- Design your solutions to be easy to test. Ask yourself, “If I refactored this piece of code in a year, how easy it would be?” You won’t be afraid to change code if you have reliable tests.
- Design your solutions to be easy to maintain. The code degrades over time; it’s your responsibility to maintain it. The Boy Scout rule: always check in a module cleaner than when you checked it out.
Be responsible for your career
It’s not your employer’s responsibility to make sure that you are marketable. Work on your career.
- Spend n hours per week learning new stuff. Read blog posts, peruse arxiv papers, watch notes from conferences on YouTube, follow top AI influencers on Twitter or LinkedIn — there are countless ways to become aware of what is going on in the data science and machine learning fields. But try not to overwhelm yourself with noise. Treat learning time as a great investment in your career and your future. Make your life a continuous act of learning.
- Practice your skills. Contribute to open software, do your own pet project, participate in machine learning competitions or hackathons. Make practicing your skills a daily routine.
- Become a mentor to someone. Even if you do know the topic, explaining it to someone will structure your knowledge. Collaborate with someone or become a mentor. Contribute to the community.
Test-Driven Development
Test-driven development (TDD) is a software development process in which a coder repeats small cycles of work: write tests for a function, write the function, make the function pass the tests. There are multiple benefits of this approach; below are just a few:
- By writing more tests, you ensure that your program does what you are expecting. TDD allows you to catch bugs and become more confident in your code.
- As time passes, your code starts to “smell”, i.e., code degrades. A good practice is to do code refactoring from time to time. But how can you be sure that your refactoring didn’t break the code? Without tests, it is impossible to know for sure. Lack of tests stops developers from performing necessary refactoring (“If I break it, I need to fix it”).
- TDD forces you to write code correctly. Think about writing tests for a function that does multiple things at once (e.g., splits the data, normalizes it, trains the model, and calculates the score). That would be hard. Thus, you start considering whether the function could be decomposed into several smaller independent functions so that testing would be easier.
- Tests written later are not as incisive as tests written first.
Use TDD when you are going to re-use code
It’s not that obvious if data scientists should use TDD or not. There are multiple arguments against using TDD for data science and machine learning tasks:
- TDD requires more time. In some cases, a data science team need to iterate fast, and TDD would slow the process. Counterargument one: How you could be sure that the results of your iterations are reliable? A pipeline might be rejected because a hidden bug made scores poor. When you find this bug, you have to re-iterate again to be sure of the results. Counterargument two: By writing dirty code now, you will need more time in the future. In many cases, technical debt might be avoided.
- Some of data science tasks can’t be tested. Counterargument: Not being able to do everything doesn’t mean that we shouldn’t do anything. Even 10% test coverage is better than 0%.
- Not every piece of DS code goes to production, thus a lot of work might be useless. Counterargument: If it doesn’t go to production, it doesn’t mean the code isn’t going to be reused.
- Data scientists are often non-software engineers, so very few could write proper tests. Counterargument: start doing it and practice will make it perfect.
Make your own decision to use TDD or not.
I made a simple rule for myself: If I’m planning to use the piece of code more than once, I’ll write tests first. For tasks when you need to iterate quickly, use eye-ball tests (e.g., check the sanity of the output yourself by printing the output or plotting). If we are talking about production, all code must be tested.
Saying No and Saying Yes
Don’t be afraid to say “No”
- We really love the logo you’ve created for us! But our management has some brilliant ideas…
- No.
Management doesn’t have a deep understanding of your field that you do (that’s why they hired you). They often don’t understand how much time a task could take or if it even makes sense to invest resources in some tasks. Your management is counting on you to defend your objectives and not just agree with everything that they say. Be assertive. Both you and your manager need to get to the best possible outcome through negotiation. I’m not saying you should always say “No”, but keep in mind that professionals are expected to disagree.
Define “Done”
It is crucial to understand what you are expected to do, especially in data science, where score improvements might take additional weeks of work. Management expectations might differ from your plan, so to avoid frustration, discuss and define what is meant by “Done”. If you are not sure when you may say “It’s done”, decompose the task into several smaller ones of which you are sure.
Show commitment by giving a time estimate
“I’ll try to do it by the end of next week” — how does that sound to you? The meaning of this phrase might vary from “It would be nice to have; I will do it if I have time” to “This will be done by next week” depending on the person saying it. In some cases, such a phrase might be treated as a commitment to do the task, which might result in a false promise. Be crystal clear about your commitments, give and don’t be afraid to ask for time estimates.
When you say you’ll do it
You mean it
You actually do it
From time to time, you will be hindered by other people or unable to complete the task due to unexpected reasons. That’s fine. Do as much as you can and be professional. If you can’t finish the project in time, notify stakeholders about it as soon as possible.
Time Estimates
“It will take five working days to do the task” — what do you mean by that? Do you mean:
I think it might take five days. Maybe four, maybe six. Maybe even ten. But my final guess is five days.
or do you actually mean something closer to:
I’m committed to making it work in five days or less no matter how much it would cost me. I will work additional time; no family, no friends. Five days, no more.
Provide time estimates, not guesses
Businesses see time estimates as commitments, while developers tend to think of estimates as guesses. Real software developer professionals make commitments and try to achieve them. The problem is that in reality, we don’t know how much time the task will take, thus we need estimates. And people tend to be terrible estimators. However, there are a couple of ways that can help to improve our estimates.
Decompose big task into several small tasks
Imagine you are planning your budget for a vacation in the Bahamas. How much do you need? 1k, 3k, 10k? Could you know for sure?
You may not be perfectly certain, but you can still give a reasonable estimate. All you have to do is to get a price for tickets to the place and back; how much you are planning to spend on food, entertainment, and gifts; and so on and so forth and then sum it up. These estimates are not that hard, are they? The same goes for time planning.
By decomposing the big task into several small ones, you are killing two birds with one stone. First, this breakdown is better for understanding the needed components and the whole architecture of the task. Second, it improves the accuracy of the estimate: it’s much easier to estimate the time needed for a small task than for a big one.
Use PERT methodology
To improve the accuracy of your estimations, start thinking of time estimation not being a single number, but several cases:
● O: Optimistic — the task is done this quickly if absolutely everything goes right.
● N: Normal — this estimate has the greatest chance of success (most probable time). Not everything will be perfect; some errors and blocks will occur.
● P: Pessimistic — everything goes wrong. Worst possible scenario, but still a probable one.
Given these estimations, and assuming the time that is needed for a task can be described with a beta distribution, you may calculate:
- Expected duration of task: m = (O + 4N + P) / 6 (days)
- Standard deviation of task: sigma = (P — O) / 6 (days)
Time which is needed for a sequence of tasks with expected duration m_i and standard deviation sigma_i calculated as:
- Expected duration of a sequence of tasks: M = m_1 + m_2 + … + m_n
- Standard deviation of a sequence of tasks: Sigma = (sigma_1 ^ 2 + sigma_2 ^ 2+ … + sigma_n ^ 2) ^ 0.5
Ask your teammates to estimate the required time
In 1906, Francis Galton, the English statistician, was attending a county fair in Plymouth, a port city located on the south coast of Devon. He came across a weight-guessing lottery wherein participants were asked to guess the weight of the butcher ax. The person that produced the closest guess would win a prize. Galton asked the lottery organizer to write down participants’ guesses.
Based on the notes, Galton found that an average of 800 guesses was within just 500 grams of the actual ax weight, which was far closer than the guess of the winner. This has contributed to the insight in cognitive science that a crowd’s individual judgments can be modeled as a probability distribution of responses, with the median centered near the true value of the quantity to be estimated (see more examples of crowd wisdom in The Wisdom of Crowds).
The simplest and fastest way to get a more accurate estimate of the time required for a task is to independently ask your teammates and then average their answers. You don’t need to ask 800 people as in the ax weight lottery to get a robust prediction. Even with the help of a couple of colleagues, the final estimation would have a much lower variance and thus the estimation would be more accurate (the error of an estimation consists of bias, variance, and noise; reduce one of them and you will get a better prediction).
To improve the prediction even further, you could do the following:
- Guess the time required to do the task. Suppose your estimate is 7 days.
- Ask a colleague if the task will require more or less time. Find the difference between their estimate and yours. Multiply the difference by a small number (0.01 if you have a huge department; 0.5 if you have a small team) and add this result to your estimate.
- Repeat until convergence.
This approach is similar to gradient boosting. In theory, it reduces the bias of the estimation. More information on bias-variance tradeoff, blending, and boosting can be found in An Introduction to Statistical Learning (chapters 2 and 8).
Meetings
The working day is too short, the amount of deep work you could devote to coding even shorter.
Don’t attend every meeting to which you are invited
It is your job to manage your time, not the person who invited you. Sometimes, it’s better to have a short, face-to-face discussion and skip a long meeting.
Provide a clear agenda and set of goals to achieve
This is a simple yet highly effective strategy to improve the efficiency of meetings. Are you going to solve something during the meeting or is it just a random discussion? Write down what points need to be discussed and what decisions should be made. Make it actionable.
Make notes during the meeting
Do you remember what was at the meeting two months ago? Me neither. I don’t know if anyone could. Maybe someone could, but they most likely are counting cards in Vegas.
Take notes yourself or ask someone to take notes during the meeting. Write the most important statements, agreed-upon points, and tasks. Publicly share your notes in Confluence or Google Docs so the people who skipped the meeting can still be updated.
Working as a Team
Do pair programming sessions
Pair programming is a session when two or more (preferably two) people work together on a single task. Usually, one person is writing the code and another talking through the problem and observing the issues, such as syntax, logic, or code style. There are multiple benefits of pair programming:
- It’s probably the best way to review the code.
- Pair programming is the fastest way to transfer knowledge within the company.
- Pair programming helps with team building.
Start using Pomodoro
I started using the Pomodoro technique about six months ago and I noticed that it became easier to focus on the problem and track daily results (daily standups also help). Besides, it is healthier to break your working sessions into intervals so that you can get up from the computer and warm up.
