My personal workspace (Shot myself with a Canon-Rp)

Parallel batch processing in Python

Process in batches using joblib and show progress with tqdm

Dennis Bakhuis
8 min readDec 27, 2021

--

Joblib is a great tool for parallelization but sometimes it is better to process the workload in batches and not in the default iterative way. In this article I’ll show:

  1. Standard way to parallelize using joblib and tqdm
  2. Why and when it does not work

--

--

Data Scientist with a passion for natural language processing and deep learning. Python and open source enthusiast. Background in fluid dynamics.