The world’s leading publication for data science, AI, and ML professionals.

6 To-Do Tips When Waiting for Models to Train

Share with the community what you do when waiting for models to train!

Model training is time-consuming. Its length could be from hours to days or even weeks. What do data scientists do when waiting for models to train? To answer this question, I interviewed over 20 data scientists combining my own experiences to share this story with you.

How about you? If model training is also a part of your work, what are your favorite activities when waiting? you are welcome to leave a comment and share your story with the community.

To-Do Tip 1 – Write Auto Tuning Scripts

There are millions of hyperparameter combinations from a single machine learning (ML) model. It is suggested to use methods such as Bayesian optimization, evolutionary search in tuning scripts to improve hyperparameter tuning efficiency, which really takes time to learn.

To simplify the script development, there are some helpful hyperparameter optimization tools I would like to recommend:

  • MLlib: a scalable machine learning library from Apache Spark.
  • Hyperopt: a library built for Distributed Asynchronous Hyper-parameter Optimization.
  • DEAP: an evolutionary computation framework with the application of hyperparameter tuning.
  • GpyOpt: a library built for Gaussian process optimization using GPy.
  • AutoSklearn: a subset of Sklearn focusing on algorithm selection and hyperparameter tuning.
  • TPOT: a python automated machine learning tool with the application of hyperparameter tuning.

To-Do Tip 2 – Search Alternatives

Because of "standing on shoulders of giants", one single research tends to be used as a "giant" in multiple places and be branched into alternatives. You probably want to explore similar approaches and compare them with your current work. The alternatives should not be limited to model architecture but also include data vectorization, normalization, feature extraction, etc.

Photo by ThisisEngineering RAEng on Unsplash
Photo by ThisisEngineering RAEng on Unsplash

To-Do Tip 3 – Analyze Data

Some data scientists from interviews commented that they always found something new by simply observing the same dataset for weeks. They could analyze the data in very different views through dimensions mapping, feature visualization, and distribution plot. They claimed that the repetitive data analysis brought them many new innovative ideas, especially when a bottleneck is met.

Maybe? If your agency has human data labelers, you should also try to ask whether they’ve noticed anything interesting.

To-Do Tip 4 – Add Unit Test

"Unit test" is a frequent item used by engineers but I found there were more data scientists starting to introduce unit tests to assure work quality. Unit tests could avoid unexpected output, detect edge cases and prevent broken code from being pushed. Identifying missing data, duplicated data, data range are those common use of unit tests in Data Science. Pytest is probably the most popular unit test tool used by data scientists.

Photo by Ferenc Almasi on Unsplash
Photo by Ferenc Almasi on Unsplash

To-Do Tip 5 – Document Progress

Documentation is not a very attractive task for data scientists, but the future you and your team would appreciate once you have done it. The waiting period for training is the perfect timing to recap the current progress and document it. Keeping documentation up to date is not only helpful to present your work when your supervisor randomly walks in but also to avoid duplicated work from your team members.

To-Do Tip 6 – Start Parallel Training Jobs

Learned from the data scientists working for big tech companies such as Google and Nvidia, it is highly recommended to launch parallel training jobs. It will significantly improve your work productivity.

It would be great if your local computer is powerful enough to run multiple training jobs. Otherwise, you may need to consider additional computing resources from cloud providers (e.g. AWS, GCP). Training model off-site could reduce the workload of your work devices and scale up computing power easily. The only tradeoff is that you may spend a little bit more time on infrastructure setup.

I Won’t Tell You😜 : there were some honest answers such as get coffee and distract colleagues, watch Netflix, play League of Legends, change notebook windows size, pray for success, and so forth. Shuuuu!!! I know what you want to say! Let’s keep the secret low for data scientists.

To speed up the Training

If you need extra computing resources such as cloud machines to speed up training, I would like to introduce you AIbro, a serverless Model Training tool that my friends and I are working hard on. It helps you train Tensorflow-based models on AWS EC2 machines with one line of code. As an early adopter, you will have the access to 4 x V100 GPUs for free. For more details, you may want to check out this story:

https://towardsdatascience.com/train-neural-network-on-cloud-with-one-line-of-code-8ae2e378bc98.

Image by the author
Image by the author

Related Articles