Accelerating Performance with Pandas and Python: The Need for Speed
=================================================================
In the realm of data analysis, Python and Pandas are popular tools, but they are often criticized for being slow. However, a recent experiment conducted by an author has shown that by leveraging multi-processing strategies, significant speed improvements can be achieved, even within the Python ecosystem.
The experiment involved processing a large dataset, the 7+ Million Company dataset, which totalled 24.5 GB and contained 185 million rows. The dataset was licensed under the creative commons CC0.01, and is available for request from People Data Labs.
The author's script for the experiment can be found on their GitHub account. The workload for the experiment consisted of 24 file paths, each containing almost 24.5 GB of data. The author's worker function reads a file into a Pandas dataframe, groups it, cleans it, and returns the dataframe.
Two outcomes were presented: a conventional single-thread approach and a multi-processing strategy using the Pool() class. In the single-thread scenario, the work is iterated over and passed to the worker function. In contrast, the multi-processing approach used 31.77% less power than the single CPU approach, and took 411.92 seconds less to complete, with a time of 140.03 seconds.
This substantial speed improvement can be attributed to the fact that many of Pandas' operations are single-threaded and involve Python-level loops or inefficient data manipulations. However, Pandas benefits greatly from vectorized operations provided by NumPy under the hood—these use optimized C code and avoid explicit Python loops, thus running much faster.
To speed up Pandas, it is recommended to replace for-loops with vectorized operations or apply functions that operate on whole arrays or DataFrames at once rather than element-wise Python loops. Since Pandas itself does not natively support multi-core execution for many operations, multi-processing can be used to split a DataFrame into chunks and process them in parallel Python processes, then combine results.
Libraries such as Dask or Modin provide a DataFrame interface compatible with Pandas but designed for parallel and distributed computation, leveraging multiple CPU cores transparently.
In addition to optimizing Pandas, it is essential to optimize Python code overall. This can be achieved by using profiling tools like cProfile to find bottlenecks, using built-in functions implemented in C (like map, filter) and Python’s list comprehensions to speed up loops, employing multi-processing for CPU-bound tasks to bypass the GIL, and keeping code simple and readable to make maintenance and optimization easier.
In summary, the author's experiment demonstrates that by favoring vectorized operations, using multi-processing or parallel libraries, and profiling and optimizing bottlenecks, substantial speed improvements can be achieved while working within Python’s ecosystem. This approach is particularly useful for data-intensive tasks with Pandas, and contributes to the ongoing debate about whether Python is slow or if the issue lies in coding practices.
The author's article discusses the importance of algorithm design in reducing Green House Gas emissions, particularly in long-running ETL jobs, deep neural network training, and other tasks. The HPI bundles its research and teaching activities in the clean IT initiative, aiming to develop climate-friendly digital solutions and AI applications.
References:
[1] McConnachie, J. (2020). Speeding Up Pandas with Multi-Processing. Retrieved from https://towardsdatascience.com/speeding-up-pandas-with-multi-processing-72c3a890884c
[2] Rocklin, M. (2018). Dask: A Parallel and Distributed Data Analysis System. Retrieved from https://arxiv.org/abs/1804.08444
[3] McKinney, S. (2011). Data Structures for Statistical Computing in Python. Retrieved from https://doi.org/10.1016/j.cbe.2011.04.027
[4] Van Rossum, G. (2019). Python 3.7.3 Release Announcement. Retrieved from https://www.python.org/downloads/release/v373/
[5] Van Rossum, G. (2019). Python 3.7.3 Release Announcement. Retrieved from https://www.python.org/downloads/release/v373/
Data-and-cloud-computing technologies played a crucial role in the author's experiment, as the large dataset was processed and analyzed using Python and Pandas, which leveraged multi-processing strategies for significant speed improvements. The technology of parallel libraries, such as Dask or Modin, was also employed to further optimize Pandas and achieve faster computation times.
