PLEASE STOP USING PANDAS FOR GIANT DATA SETS.

@lachinemearning.com

20.9K followers · usually 8.3K views a post

Views

77.3K

Likes

8.2K

Comments

64

Shares

307

Selling · SaaS › Other SaaS

Polars

Why it worked

The video addresses a common pain point for data scientists and programmers by offering a practical solution to a performance issue with a widely used tool (Pandas). It highlights a superior alternative (Polars) with clear benefits and a compelling explanation of its underlying technology.

Summary

The video explains that while Pandas is a reliable data manipulation tool, it is inefficient for large datasets due to its single-core processing and memory hoarding. It introduces Polars as a faster alternative written in Rust, which uses lazy evaluation and multi-core processing for significant performance gains on big workloads.

Structure

1Introduction to the problem with Pandas for large datasets
2Explanation of Pandas' limitations (single core, memory)
3Introduction of Polars as a faster alternative
4Explanation of Polars' advantages (Rust, Apache Arrow, lazy evaluation, multi-core)
5Caveats for using Pandas with smaller datasets or specific libraries
6Recommendation to switch to Polars for large-scale data processing

Original caption

it definitely gets easier to use the syntax if youre familiar with the messy world of data cleaning, play around with it!! bear in mind, pandas is still good for relatively small to medium datasets where polars isnt needed, they can perform the same #statistics #machinelearning #computerscience #python #programming

More from @lachinemearning.com

Their other posts in the index, biggest breakout first.

Product placement

Pandas: A library that simplifies tasks of data manipulation in Python. It is part of the 'big four' Python data science libraries (NumPy, Pandas, Matplotlib, and Scikit-learn). It acts as a data manipulation tool. Removing it would change the video's subject. Polars: A data manipulation library written in Rust, built on Apache Arrow, designed for performance and efficient handling of large datasets. It acts as a data manipulation tool. Removing it would change the video's subject. NumPy: Part of the 'big four' Python data science libraries. Merely appears. Matplotlib: Part of the 'big four' Python data science libraries. Merely appears. Scikit-learn: Part of the 'big four' Python data science libraries. It expects Pandas dataframes. Merely appears.

On-screen text

PLEASE STOP USING PANDAS FOR GIANT DATA SETS. IT IS EXTREMELY INEFFICIENT. IF YOU'RE NEW TO DATA SCIENCE OR YOU'VE BEEN WORKING, YOU KNOW THAT PANDAS IS AMONGST THE FAMILY OF THE BIG FOUR- NUMPY, PANDAS, MATPLOTLIB, AND SKLEARN AND THEY ALL NATIVELY WORK WITH EACH OTHER. IT'S RELIABLE. IT'S BEEN THE DEFAULT FOR OVER A DECADE. HOWEVER, EVERYTHING RUNS ON A SINGLE CORE AND IT LOVES TO HOARD MEMORY. EVEN AT WORK. I TRIED TO USE IT FOR A COUPLE MILLION ROWS AND MY RAM RED LINED. HOWEVER, THERE IS A QUICKER ALTERNATIVE, AND IT'S CALLED POLARS, AND IT'S WRITTEN IN RUST, BUILT ON APACHE ARROW, AND IT IS BLAZING FAST. WE'RE TALKING FIVE TO 10 TIMES FASTER ON BIG WORKLOADS. AND THE REASON IS BECAUSE IT USES SOMETHING CALLED LAZY EVALUATION, WHERE INSTEAD OF EXECUTING YOUR CODE LINE BY LINE AND MAKING A BUNCH OF EXPENSIVE DATA COPIES, IT WAITS, OPTIMIZES YOUR ENTIRE QUERY PLAN, AND THEN RUNS IT ACROSS ALL YOUR CPU CORES AT ONCE. THIS DOESN'T MEAN THROW PANDAS OUT OF YOUR DATA SCIENCE TOOLKIT. BECAUSE IF YOUR DATA SET FITS COMFORTABLY IN MEMORY, OR IF YOU'RE USING LIBRARIES LIKE SCIKIT-LEARN THAT EXPECTS PANDAS DATA FRAMES, YOU MIGHT AS WELL STICK WITH PANDAS. EVEN THOUGH POLARS DOES ALLOW YOU TO CONVERT THINGS TO PANDAS DATA FRAMES. HOWEVER, IF YOU'RE TRYING TO PROCESS MILLIONS OF ROWS OF DATA, OR IF YOU'RE BUILDING PRODUCTION PIPELINES, YOU MIGHT AS WELL SWITCH TO POLARS! IT WILL SAVE YOU A LOT OF WORKLOAD

Transcript

PLEASE STOP USING PANDAS FOR GIANT DATA SETS. IT IS EXTREMELY INEFFICIENT. IF YOU'RE NEW TO DATA SCIENCE OR YOU'VE BEEN WORKING, YOU KNOW THAT PANDAS IS AMONGST THE FAMILY OF THE BIG FOUR- NUMPY, PANDAS, MATPLOTLIB, AND SKLEARN AND THEY ALL NATIVELY WORK WITH EACH OTHER. IT'S RELIABLE. IT'S BEEN THE DEFAULT FOR OVER A DECADE. HOWEVER, EVERYTHING RUNS ON A SINGLE CORE AND IT LOVES TO HOARD MEMORY. EVEN AT WORK. I TRIED TO USE IT FOR A COUPLE MILLION ROWS AND MY RAM RED LINED. HOWEVER, THERE IS A QUICKER ALTERNATIVE, AND IT'S CALLED POLARS, AND IT'S WRITTEN IN RUST, BUILT ON APACHE ARROW, AND IT IS BLAZING FAST. WE'RE TALKING FIVE TO 10 TIMES FASTER ON BIG WORKLOADS. AND THE REASON IS BECAUSE IT USES SOMETHING CALLED LAZY EVALUATION, WHERE INSTEAD OF EXECUTING YOUR CODE LINE BY LINE AND MAKING A BUNCH OF EXPENSIVE DATA COPIES, IT WAITS, OPTIMIZES YOUR ENTIRE QUERY PLAN, AND THEN RUNS IT ACROSS ALL YOUR CPU CORES AT ONCE. THIS DOESN'T MEAN THROW PANDAS OUT OF YOUR DATA SCIENCE TOOLKIT. BECAUSE IF YOUR DATA SET FITS COMFORTABLY IN MEMORY, OR IF YOU'RE USING LIBRARIES LIKE SCIKIT-LEARN THAT EXPECTS PANDAS DATA FRAMES, YOU MIGHT AS WELL STICK WITH PANDAS. EVEN THOUGH POLARS DOES ALLOW YOU TO CONVERT THINGS TO PANDAS DATA FRAMES. HOWEVER, IF YOU'RE TRYING TO PROCESS MILLIONS OF ROWS OF DATA, OR IF YOU'RE BUILDING PRODUCTION PIPELINES, YOU MIGHT AS WELL SWITCH TO POLARS! IT WILL SAVE YOU A LOT OF WORKLOAD