As a Data Science practitioner and physicist with over 7 years of lab experience, I’ve discovered that some old-school academic practices fit perfectly in a modern data science workflow.
One of these practices is having a properly organised and structured lab notebook.
In this article I’ll share my personal view on using it consistently.
Why should I care about Lab Notebooks?
Lab notebooks have been around for a long time, as long as science itself exists. A Lab Notebook may be defined as a primary record of a researcher. It is used to document hypotheses, experiments, and initial analysis or interpretation of these experiments (see the image below).
To put it simply, a lab notebook is just a notebook that keeps track of your research work, thoughts, observations and activities. There are, however, some guidelines, that may differ from one organization to the other, you can check the ones that MIT suggests.
Why would anyone be bothered with lab notebooks these days you might ask?
Now, in 2022, we have a ton of fancy general-purpose solutions for regular note taking, task/project management, and collaboration: Jira, Confluence, Slack, and virtual whiteboards (Miro) to name a few.
As data scientists, we have a whole set of digital tools of our own to perform and track experiments (Comet, MLflow, Neptune, SageMaker Studio, etc.).
Most of these widely used solutions are great for solving a particular problem in a day-to-day data science workflow, and each of them offers a unique set of features. When combined, these tools seem to cover all my commercial data science needs. But only at a first glance.
With all these tools at our disposal, the number one tool that I rely on (for the better part of the last 12 years actually) is my lab notebook.
This requires a bit of explanation.
How I Used my Lab Notebook While Doing Laser Physics Research
I’ve been practicing applied laser physics for 7 years, and my lab notebook has always been the backbone of the whole multi-dimensional research and work process in general.
First thing in the morning I used to grab a coffee and read through the last couple of weeks of my records. A typical single research day for me takes from 2 to 4 pages of blank paper filled with images and text (like in the picture below).
While working on a particular problem I typically had multiple types of activities: designing experiments, making measurements, doing exploratory data analysis, running numerical simulations, writing scientific articles, etc.
Each activity had been planned and tracked in a very specific way that served me in the best possible way long term.
I would categorize my notes into 5 buckets:
- A brief plan for the day (to-do list, key meetings/presentations/experiments to do)
- Experiment planning and tracking (mostly high-level, but detailed when needed)
- Experimental data analysis and interpretation
- Key questions, insights, and ideas
- Retrospectives
Five types of notes is actually a lot, but it is the bare minimum of what was necessary for me because as an applied physics practitioner you unavoidably have to wear a lot of hats, and most of your activities are far from being trivial and easy to explain.
A lot of these activities immediately put you under some kinds of pressure and stress: operating with unfamiliar tools, struggling to explain phenomena, uncertainty with the next action steps, etc.
A way to handle this stress for me was to lay out the information and results carefully in a notebook, frame the problem, write it down, decompose the problem into actionable steps and start going through the steps one by one.
With note-taking in a lab notebook I’ve been trying to stay concise but make sure that the quality of my writing/drawing is in place. And I mean quality not in a sense of writing neatly and choosing nice pictures, but making sure that I put time and effort into thinking each line through.
And don’t get me wrong, I’ve spent a good half of the day writing software (in both Python and Wolfram Mathematica), processing gigabytes of data, visualizing data, and writing articles – all these using dozens of digital tools.
But my lab notebook has always been above everything else – supervising each activity, guiding me through research life, and allowing me to push through uncertainty.
Side note on analog vs digital: I’ve used analog notebooks because I used to draw schemas and charts, wrote freely and then draw on top of it, etc. At the time there was no way to do it digitally without losing the versatility. Nowadays a lot more paperless options are available, but it is not what this article is about.
Lab notebook, in my view, is a framework for highly structured technical-note taking combined with journaling practices. Digital versus analog – does not matter really.
Over time I’ve realized that my lab notebook became a technical knowledge base on one hand and a personal research diary on the other.
The benefits I’ve been gaining from using lab notebooks consistently were so powerful that I’ve stuck to this habit even after leaving laser physics.
How it All Translated Into My Daily Data Science Practice
Since I left the academy the role of a lab notebook in my work life has evolved, but it is still a very essential piece in my tool belt.
As a data science practitioner and group manager nowadays I do a lot of stuff with data to achieve business goals – I program, perform exploratory data analysis, manage projects, manage teams, meet with product owners and stakeholders and develop data products.
It is exciting as it is, but I’ve managed to adapt to the intense workload and stay productive using my old school academic practices tweaked slightly.
I still keep my daily records in a specifically structured lab notebook manner. Of course, I don’t insert laser beam shapes anymore, and I don’t log laser physics experiments, but I do write down ideas, thoughts, and observations using an updated set of categories (see below).
And again, with all the digital tools at my disposal, the lab notebook becomes a knowledge base, a diary, and a sketchbook that sits on top of every other tool and guides me through my work.
The reasons for keeping records that way, as well as specific guidelines that I follow, I’ve tried to formulate as a set of rules/principles:
Reason 1: Keeping Track of Your Ideas – both Crappy and Brilliant
The role of a data scientist for me personally is a creative one. Especially in a Product Development field – in my opinion, it is all about fresh ideas and creative problem solving.
I’ve always used a lab notebook to do my brainstorming. And not the kind of brainstorming where you just produce absurd stuff in volume, but deep and continuous creative thinking on what can actually work.
Thus, reason number 1: keep and nurture your ideas. Write stuff down. Write down questions, ideas and thoughts. Our human brain is not good at keeping thoughts around for a long time.
Ideas are extremely fragile, Ideas are not predictable in terms of when you’ll have them and how many you gonna have. © Jonathan Ive
Keeping careful records of ideas allows reiterating on them, reassessing, and thinking them through in a really deep manner.
Lab notebook allows me to carefully grow ideas from the very early stages (most fragile) to flourishing implementation without getting sidetracked.
How: I’m using the lightbulb icon to record ideas that might sound interesting. Sometimes it takes me several days to reiterate a single idea, re-formulate, add details, and in this case, I use original record referencing (a page number) to link the latest iteration to the previous one.
Reason 2: Keeping Track of what is going on Around You
As Steven S. Skiena writes in his Data Science Design Manual:
However, the heart of data science lies in doing the simple things right: understanding the application domain, cleaning and integrating relevant data sources, and presenting your results clearly to others
With all my experience I can’t stress enough how valuable good communication might be, especially when it comes to dealing with domain experts.
And not only careful listening is important, in my opinion, but also the overall vigilance and attention to what’s happening on the business side.
Especially when you manage projects, negotiate with stakeholders, and generate product ideas – it is your duty to observe your surrounding and distill each bit of information into key observations.
How: the way I do it is pretty simple: whenever I participate in a conversation, I’m going into a focus mode and listen thoroughly, ask for clarification if needed and then reformulate the key observation/insight into several sentences in the lab notebook.
During the preparation for the meeting I create key points that I plan to cover, and then insert the feedback on each of these points:
The downside – you have to listen actively and pay attention, which is difficult, but the upside is much bigger – over time it creates an impression that not only do you care about other’s opinions, but also that nothing will slip out of your attention (and that is a good thing).
Of course, you can do it without a notebook, but running meeting with a pen in your hand, listening attentively, and digesting input information into key observations sets a solid protocol for these activities.
If I’m not in a position to listen actively I’d rather reschedule a meeting than run it just for sake of being there.
Reason 3: A Perfect Keystone Habit
For me keeping track of my work in a lab notebook became an effective keystone habit.
Keystone habits are the routines and practices by which someone operates. They mark the base level of what you do without any need for willpower or persuasion. The default. Whether positive or negative, each of these habits has a ripple effect across everything you do in life and business. (Forbes article)
Regardless of the problem complexity you are working on, regardless of all other circumstances and chaos, the habit of just starting your work by re-reading your progress and thoughts sets the tone for the day. For me running the lab notebook creates a momentum that I can use to build upon and move stuff forward.
How: just start your work day by reading through the last 3–4 days of records with a cup of coffee. Timestamp the current day, lay out the high-level plan for the day, start thinking about the execution details and go from there, at least it works for me.
Reason 4: Tracking your Personal Progress and Growth
Don’t compare yourself to somebody else – compare yourself to your past self, yes, we’ve heard that multiple times, but it is easier said than done.
With having a proper lab notebook, which is also a personal work journal, your own progress will be obvious to yourself, and you should not underestimate the power of this trick (is it even a trick?).
Just by reading through the pages you’ll see the growth in skillset, problem understanding, and problem-solving.
This matters particularly when you, as a researcher, go through difficult times: lab notebook will always prove that you’ve moved really far from the starting point.
How: what works for me is writing down a list of goals every 3 months. 3 months later I take a page or two of the lab notebook to make a retrospective: I reference the goal-setting page and go through want went well and what did not.
Reason 5: Being Strategic
Lab notebook pushes you to be strategic – it just forces you to think productively and take action, because otherwise you’ll lose your momentum.
The strategy itself is a tricky one to define, but I like the definition by Roger Martin:
Strategy is an integrative set of choices that positions you on a playing field of your choice in a way that you win. Normally, if you just work without leaving timestamped records and thoughts, time passes, and your brain can easily substitute your memories, distort the real sequence of events and fit everything into a more convenient and explainable, but a wrong view.
With a lab notebook, however, you don’t allow this to happen.
First of all, because the notebook is permanently bound, you can’t just rip the pages out and insert new ones (which your brain may easily do with memories).
Secondly, if you make poor choices with research and your work over a significant amount of time and record this, the detrimental impact on the outcome becomes so obvious that it is stunning.
So, naturally, notebook just forces you to think long-term and be strategic.
How: follow the protocol – set the goals every 3 months and perform retrospectives – all in the same notebook. Your daily routines in these 90 days in between will define whether you succeed or failed.
Reason 6: Irrational and Dramatic
Well, that is really controversial and subjective, but I find having a proper notebook a bit dramatic.
Not only great scientists of the present and past had lab notebooks, but explorers too. Well, at this point you might say, that ship captains had nautical logbooks, and arctic researchers had diaries (I’d suggest reading through the diary of Robert Falcon Scott), but I group these different kinds of notebooks into a single family.
When writing into a lab notebook, I think about these brave men and women of the past and present that risked their lives to get to unknown places.
Most of us work in extremely safe environments nowadays, yes, but there is something intangibly dramatic about keeping detailed, journal-like records of your research.
What I say to myself is that yes, I’m not the first human on the North Pole or an astronaut, but I also go to unknown places and look over the fence.
And I do like the feeling of being a captain of my own expedition.
How: I can’t give any specific advice here, rather than finding something that inspires you in a good way and sets the right tone.
Conclusion
To sum it up: for me, lab notebook is a technical log book and a diary at the same time, a tool that takes place above everything else at my disposal. Lab notebook sets rules and protocols for tracking ideas, running meetings, solving problems, and doing work on a day-to-day basis.
In my opinion, having a proper lab notebook is essential in doing any long-term projects in the commercial data science field: it just helps to be a better listener, think deeper, plan strategically and create a real impact.
As a bonus, you get a beautiful diary that tells a unique story of solving problems, a story of grinding and growing.
This is it, thanks for reading, curious to hear your thoughts.