modin

熊猫简介：(Brief Introduction to Pandas:)

Pandas is an open-source library providing high performance, easy to use data structures and data analytics tools for the Python language. Pandas is a fundamentally high-level building block for doing real-world data analysis in Python. It has the broader goal of becoming the most powerful and flexible open-source data analysis and manipulation tool.

Pandas是一个开源库，为Python语言提供了高性能，易于使用的数据结构和数据分析工具。 Pandas是从根本上高层的构建基块，可用于在Python中进行实际数据分析。 它的广泛目标是成为最强大，最灵活的开源数据分析和操纵工具。

Pandas library has 40+ API’s, some of the most used are pandas DataFrame, pandas Series, pandas plotting, pandas Input/Output. Important highlights of pandas libraries are:

熊猫图书馆有40多个API ，其中一些最常用的是熊猫DataFrame，熊猫系列，熊猫绘图，熊猫输入/输出。 熊猫库的重要亮点是：

A fast and efficient DataFrame object for data manipulation with integrated indexing

具有集成索引的快速高效的DataFrame对象，用于数据处理

Tools for reading and writing data

读写数据的工具

Intelligent data alignment and integrated handling of missing data

智能数据对齐和丢失数据的集成处理

Intelligent label-based slicing, fancy indexing, and subsetting of large datasets

基于智能标签的切片，花式索引和大型数据集子集

Aggregating or transforming data with a powerful group by engine.

通过强大的引擎来聚合或转换数据。

High-performance merging and joining of data sets

数据集的高性能合并和联接

Time series-functionality

时间序列-功能

Pandas library is highly used in academics and commercial domains including Finance, Neuroscience, Economics, Statistics, Advertising, Web Analytics, and more.

熊猫图书馆在包括金融，神经科学，经济学，统计学，广告，网络分析等在内的学术和商业领域得到了广泛使用。

Pandas being such a great library that enables data scientists/data analysts to interact with the data. But it takes time to interact with large size data. Pandas mainly use 1 core of CPU to process instructions and return the result. Here Modin comes in the picture, that uses distributed computing to accelerate Pandas workflow.

熊猫是一个很棒的图书馆，可让数据科学家/数据分析师与数据进行交互。 但是，与大型数据进行交互需要花费时间。 熊猫主要使用1个CPU内核来处理指令并返回结果。 图中是Modin，它使用分布式计算来加快Pandas工作流程。

Source), Modin logo 来源)，Modin徽标

如何加快熊猫工作流程：(How to accelerate Pandas workflow:)

Modin is an early access multi-processing data frame library with identical API to pandas, that bridges the gap between data analytics and large size data. It provides an effortless way to speed up pandas notebooks, scripts, and libraries and manages data partitioning and shuffling so that data scientists/data analysts can interact with the data efficiently.

Modin是一个早期访问的多处理数据框架库，具有与熊猫相同的API，从而弥合了数据分析和大数据之间的鸿沟。 它提供了一种轻松的方式来加快熊猫笔记本，脚本和库的速度，并管理数据分区和改组，以便数据科学家/数据分析师可以与数据进行高效交互。

Modin speed-up to 4x on a laptop with 4 cores.

在4核笔记本电脑上，Modin的速度提高了4倍。

Modin accelerates Pandas queries, only requiring users to change a single line of code in their notebooks. Pandas use just 1 core of CPU for processing instructions and the rest of the cores are not utilized. Whereas, Modin utilizes all the cores of available in the system, to speed up the Pandas workflow.

莫丁 加快了Pandas查询的速度，只需要用户在笔记本中更改一行代码即可。 熊猫仅使用1个CPU内核来处理指令，其余的内核则未使用。 鉴于Modin利用系统中的所有可用核心，以加快Pandas工作流程。

Source), Plots comparing time utilization to read CSV file in Pandas vs Modin, (Image 1) 来源)，比较熊猫与莫丁之间的时间利用率以读取CSV文件的图，(图1)

Modin的工作原理： (Working of Modin under the hood:)

As discussed above, Modin accelerates Pandas queries by distributed computing. It utilizes all the cores of the system. For a 4 core system, the below image compares the utilization of Pandas and Modin libraries.

如上所述，Modin通过分布式计算加快了Pandas查询的速度。 它利用了系统的所有核心。 对于4核系统，下图比较了Pandas和Modin库的利用率。

Source), CPU cores utilization in Pandas and Modin, (Image 2) 来源)，Pandas和Modin中的CPU内核利用率，(图2)

On a large server with comparatively, more number of cores compared to a standard Laptop, the performance of Modin exponentially increases. From (Image 1), it is observed it takes few seconds to load 18gb of CSV file.

与标准笔记本电脑相比，在具有相对更多内核的大型服务器上，Modin的性能成倍增加。 从(图1)可以看出，加载18gb的CSV文件需要花费几秒钟的时间。

安装： (Installation:)

Modin is open-source and you can find the GitHub repository here. Install Modin from PyPI:

Modin是开源的，您可以在此处找到GitHub存储库。 从PyPI安装Modin：

pip install modin

If you don’t have Ray or Dask installed, you will need to install Modin with one of the targets:

如果未安装Ray或Dask ，则需要使用以下目标之一安装Modin ：

# Install Modin dependencies and Ray to run on Raypip install modin[ray]# Install Modin dependencies and Dask to run on Daskpip install modin[dask]# Install all of the abovepip install modin[all]

Modin will automatically detect which engine you have installed and use for scheduling computation.

Modin将自动检测您已安装的引擎并用于计划计算。

实现方式： (Implementation:)

Use Modin to accelerate your Pandas queries with just one line of code, that is only by importing the Modin library.

仅通过导入Modin库，使用Modin只需一行代码即可加速您的Pandas查询。

import modin.pandas as pd

Modin与熊猫的比较： (Comparison of the Modin with Pandas:)

Left: Pandas, 左：Pandas， Right: Modin, Comparing of compilation time to read 150 MB CSV file in Pandas vs Modin 右：Modin，在Pandas vs Modin中读取150 MB CSV文件的编译时间比较

一些限制：(Some Limitations:)

Pandas is a massive library with a large number of APIs, some of the famous ones being DataFrame, Series, etc. Pandas DataFrame and Series have 280+ functions each. Modin does not support all the Pandas functions, but it supports the most famous ones.

Pandas是一个具有大量API的大型库，其中一些著名的是DataFrame，Series等。PandasDataFrame和Series分别具有280多个函数。 Modin不支持所有Pandas功能，但支持最著名的功能。

Source), Pandas API Coverage, (Image 3) 来源)，Pandas API覆盖率(图3)

The developers of Modin figured out the most popular Pandas methods used in Kaggle competitions. A list of the top 20 is mentioned in (Image 4).

Modin的开发人员找出了Kaggle比赛中使用的最受欢迎的Pandas方法。 (图4)提到了前20名的列表。

Source), Top 20 Most Used Pandas methods in Kaggle, (Image 4) 来源)，Kaggle中20种最常用的熊猫方法，(图4)

They started implementing distributed computing Modin functions in order of their popularity.

他们按照流行程度开始实现分布式计算Modin功能。

The functions that are not implemented in Modin, is defaulted to pandas. In case you try to use a function not distributed in Modin, the library uses the default implementation of that function in Pandas, by converting the Modin data frame to Pandas data frame, performing the operations, then reconverting to the Modin data frame.

Modin中未实现的功能默认为pandas。 如果您尝试使用未在Modin中分发的功能，则库将在熊猫中使用该功能的默认实现，方法是将Modin数据帧转换为Pandas数据帧，执行操作，然后再转换为Modin数据帧。

Source), Defaulting to Pandas in Modin for unsupported functions, (Image 5) 来源)，默认使用Modin中的Pandas以获得不支持的功能，(图像5)

结论： (Conclusion:)

You can use Modin in replacement of Pandas, as it parallelizes the implementation and reduces the computation time, even if some pandas API is not parallelized, it is still defaulting to pandas.

您可以使用Modin替换Pandas，因为它可以并行化实现并减少计算时间，即使某些pandas API没有并行化，它仍然默认为pandas。

Modin uses Ray or Dask to parallelize as much of the pandas API as is possible. It is intended to be used as a drop-in replacement for pandas. In the long-term, Modin is planned to become a DataFrame library that supports the popular APIs (SQL, pandas, etc.) and runs on a variety of compute engines and backends.

Modin使用Ray或Dask尽可能多地并行化pandas API。 它旨在用作熊猫的简易替代品。 从长远来看，Modin计划成为一个DataFrame库，该库支持流行的API(SQL，Pandas等)，并在各种计算引擎和后端上运行。

翻译自: https://towardsdatascience.com/modin-speed-up-your-pandas-notebooks-scripts-and-libraries-c2ac7de45b75

modin