Introducing Reloading: Never Re-Run Your Python Code Again To Print More Details

Modify your code during run-time and save hours of work time

Photo by Brad Neathery on Unsplash

Motivation

While running Python scripts, I have often found myself in situations where I forgot to print all the necessary details to track the pipeline’s progress.

This is typically observed in training machine learning models. More often than not, folks (including me) often forget to:

1. Add necessary logging details.
2. Print essential training details/metrics such as accuracy, error, precision, etc.
3. Save the model after every k epochs, and many more.

I am sure you have been there too.

Of course, the problem is not just limited to machine learning. Many face the same issue in other domains as well, such as Web Scrapping, where people realize after running their codes that they should have scrapped some more details, etc.

Left with no choice, one has to unwillingly stop the code, add the necessary details and re-run the code again. This can be incredibly frustrating if your pipeline has been running for a few hours.

But what if I told you that there is a neat trick to this? In other words, it is actually possible to make changes to an already running code without losing the current progress.

This is what I will explore in this blog.

Let’s begin 🚀!

Reloading

Reloading, as the name suggests, is a Python library that allows you to reload a loop (or a function) from the source before each iteration.

Therefore, you can modify an already running code and add more details to it without losing any current progress. Isn’t that cool?

To install it, use the following command:

pip install reloading

Reloading A Loop

Consider that you have a loop that takes an initial value and halves it after every iteration.

However, we mistakenly forgot to print the iteration number in this loop and now want to modify it.

Of course, without reloading, you have no choice but to re-run it.

However, if you want to reload the body of the for loop before each iteration, wrap the iterator with reloading, as shown below:

Now, you can modify the code during run-time. A demo is shown below:

Demo of Reloading (Gif by Author)

As demonstrated above, we modified the body of the for loop. As a result, we get to see new details in the output panel, while retaining the current progress.

Reloading a Function

Similar to reloading a loop, you can also reload the body of a function after each iteration. Consider the function half_value below:

To reload the body of a function, decorate it with the reloading decorator. This is shown below:

Now, you can modify the function during run-time. A demo is shown below:

Demo of Reloading (Gif by Author)

Perfect!

Conclusion

With this, we come to the end of this blog. I hope you learned something new.

I am confident that this trick will save you tons of time while running some high-run-time codes, such as training machine learning models.

Found this tip interesting?

If you want to learn more such elegant tips and tricks about Data Science and Python, I post an informative tip daily on LinkedIn.

You can find all the tips I have posted in My LinkedIn Post Archive. You can follow me on LinkedIn to see all future posts.

Alternatively, you can also receive them via email:

🚀 Subscribe to the Daily Dose of Data Science. Here, I share elegant tips and tricks on Data Science, one tip a day. Receive these tips right in your inbox daily.

🧑‍💻 Become a Data Science PRO! Get the FREE Data Science Mastery Toolkit with 450+ Pandas, NumPy, and SQL questions.

Get a weekly summary of the top 1% research papers, news, repos, and tweets in Machine Learning.

I like to explore, experiment, and write about data science concepts and tools. You could connect with me on LinkedIn.

Thanks for reading!