Loading large datasets in Pandas

Effectively using Chunking and SQL for reading large datasets in pandas

Parul Pandey
Towards Data Science
6 min readOct 14, 2020

--

The pandas’ library is a vital member of the Data Science ecosystem. However, the fact that it is unable to analyze datasets larger than memory makes it a little tricky for big data. Consider a situation when we want to analyze a large dataset by using only pandas. What kind of problems…

--

--

Principal Data Scientist @H2O.ai | Author of Machine Learning for High-Risk Applications