python julia

介绍(Introduction)

This blog post is about working with Python and Julia together and aimed mainly for Pythonists Data scientists who want to quickly accelerate their run time performance and do it by another programming language that gains its popularity within universities, companies, and users.

这篇博客文章是关于与Python和Julia一起工作的，主要针对希望快速提高运行时性能并通过另一种在大学，公司和用户中广受欢迎的编程语言来实现的Pythonists数据科学家。

image: Julia Computing (GitHub stars for Julia) 图片：Julia计算(Julia的GitHub明星)

By the end of this post, you will gain:

在这篇文章的结尾，您将获得：

An environment of docker image that also contains Julia and Python

也包含Julia和Python的docker映像环境 A straightforward comparison between Julia and Python in terms of speed.

Julia和Python在速度方面的直接比较。Three different methods to embed Julia code into Python.

三种将Julia代码嵌入Python的方法。

I hope you will find it informative!

希望您能从中获得启发！

安装 (Installation)

To set the environment, I used this docker image containing Julia, Python, and Jupyter:

为了设置环境，我使用了这个包含Julia，Python和Jupyter的docker镜像：

https://hub.docker.com/r/jupyter/datascience-notebook

https://hub.docker.com/r/jupyter/datascience-notebook

A step by step guide for dockers can be found in the references section.

可在参考资料部分中找到有关docker的分步指南。

JuliaVS。 用于循环的Python速度测试 (Julia VS. Python speed test on for loops)

Before I demonstrate how to embed Julia code into Python to boost the performance, I want to convince you that there is a value to use Julia, and it can work much faster than Python when using for loops.So, let’s run a simple for loop in Julia and Python and compare their running times:

在演示如何将Julia代码嵌入Python以提高性能之前，我想说服您使用Julia具有价值，并且在使用for循环时它的运行速度比Python快得多。因此，让我们运行一个简单的for循环在Julia和Python中比较它们的运行时间：

In Python:

在Python中：

def for_loop(iterations = 100): a = 0 for i in range(iterations): a = a+1 return a

It took about 1 second.

花了大约1秒钟。

%%timeres = for_loop(iterations = 10_000_000)>>> CPU times: user 953 ms, sys: 0 ns, total: 953 ms>>> Wall time: 951 ms

And in Julia:

在Julia：

function for_loop(iterations = 100) a = 0 for i in 1:iterations a = a+1 end return aend

It took 0.000001 seconds:

花了0.000001秒：

@time res = for_loop(10_000_000)>>> 0.000001 seconds (1 allocation: 16 bytes)

Quite a difference!

完全不同！

But it was just an extreme case; usually, the differences won’t be that significant.

但这只是一个极端的情况。 通常，差异不会那么大。

将Julia代码嵌入Python (Embed Julia code into Python)

So, after we are convinced that there are some potential here, let’s look at a real example from the NLP area in Machine Learning.We will pick a task that involves for loop to use Julia’s strength.We will use the Stemming task (the readers who are not familiar with this task can read about is here: https://en.wikipedia.org/wiki/Stemming).

因此，在我们确信这里有潜力之后，让我们看一下机器学习中NLP领域的一个真实示例。我们将选择一个涉及for循环的任务来利用Julia的力量。我们将使用Stemming任务(读者不熟悉此任务的人可以在这里阅读： https ： //en.wikipedia.org/wiki/Stemming)。

数据 (Data)

We will be using the data from the Shakespeare-hamlet dataset on the NLTK package. It can be accessed like this:

我们将使用NLTK软件包中的莎士比亚哈姆雷特数据集的数据。 可以这样访问：

from nltk.corpus import gutenbergdata = gutenberg.raw('shakespeare-hamlet.txt')data = data.replace("\n", " ")data = data.replace(" ", " ")

This is how the data looks:

数据如下所示：

"[The Tragedie of Hamlet by William Shakespeare 1599] Actus Primus. Scoena Prima. Enter Barnardo and Francisco two Centinels. Barnardo. Who's there? Fran. Nay answer me: Stand & vnfold your selfe Bar. Long liue the King Fran. Barnardo? Bar. He Fran. You come most carefully vpon your houre Bar. 'Tis now strook twelue, get thee to bed Francisco Fran. For this releefe much thankes: 'Tis bitter cold, And I am sicke at heart Barn. Haue you had quiet Guard? Fran. Not a Mouse stirring ..."

“ [[莎士比亚的《哈姆雷特的悲剧》] ActusPrimus。ScoenaPrima。进入Barnardo和Francisco两个城堡。Barnardo。谁在呢？Fran。 Bar。He Fran。您来得很仔细，请Bar。'Tis现在吃惊了，让您去Francisco Fran睡觉。后卫？弗兰。没有老鼠在搅动……”

使用Python进行词干 (Stemming using Python)

We will also use NLTK to do the Stemming part:

我们还将使用NLTK来执行“阻止”部分：

import nltkfrom nltk.stem import PorterStemmerporter = PorterStemmer()%%timestem_words = []nltk_tokens = nltk.word_tokenize(data)for token in nltk_tokens: new_token = porter.stem(token) stem_words.append(new_token)>>> CPU times: user 1.52 s, sys: 0 ns, total: 1.52 s>>> Wall time: 1.57 s

It took us 1.57 seconds.

我们花了1.57秒。

使用Julia进行词干 (Stemming using Julia)

Now for the interesting part.

现在开始有趣的部分。

There are few methods to embed your Julia code into Python, and we will cover three of them:

很少有将Julia代码嵌入Python的方法，我们将介绍其中三种：

Run whole expressions of Julia

运行Julia的整个表情 Run Julia with “magic” command

使用“ magic”命令运行JuliaRun Julia with a script

使用脚本运行Julia

First, we will import the relevant modules:

首先，我们将导入相关模块：

from julia.api import Juliajl = Julia(compiled_modules=False)from julia import Mainjl.using("TextAnalysis")

(We imported the relevant packages from Python and Julia).

(我们从Python和Julia导入了相关的软件包)。

运行整个表达式 (Run a whole expression)

Let’s run the code, and then understand what happened here:

让我们运行代码，然后了解这里发生的情况：

%%timeMain.data = dataMain.token_data = jl.eval("tokens_data = TokenDocument(data) ; return(tokens_data)")stem_list = jl.eval("stem!(tokens_data);stem_tokens_data = tokens(tokens_data) ; return(stem_tokens_data)")>>> CPU times: user 747 ms, sys: 112 µs, total: 747 ms>>> Wall time: 741 ms

So, in the first row, we store the data variable in the Main.data variable so Julia can read this variable.

因此，在第一行中，我们将数据变量存储在Main.data变量中，以便Julia可以读取此变量。

Main is a module that we imported from Julia. We use it here to store the data variable in Julia’s format to use it later (when we call the TokenDocument(data)).Please note that if we didn’t write that line (Main.data = data), Julia can’t recognize any variable with the name ‘data’.Next, we tokenize the data in Julia and store it in Main.token_data variable. (Note: we should use the return command to store the variable).Then, we do the stemming on the tokenized data and store it in a regular python list.

Main是我们从Julia导入的模块。 我们在这里使用它以Julia的格式存储数据变量以备后用(当我们调用TokenDocument(data)时使用)。请注意，如果我们不写那行(Main.data = data)，Julia不能识别名称为“ data”的任何变量。接下来，我们在Julia中标记数据并将其存储在Main.token_data变量中。 (注意：我们应该使用return命令来存储变量)。然后，对标记化数据进行词干处理并将其存储在常规python列表中。

So, all we need to do to run Julia’s expression in Python is to use the jl.eval function, concatenate Julia’s expressions with “;” between them, and finish with the return command.

因此，要在Python中运行Julia的表达式，我们要做的就是使用jl.eval函数，将Julia的表达式与“ ; ”连接起来。 ”，然后以return命令结束。

It took us about 740 milliseconds — about half a time from Python.

我们花了大约740毫秒的时间-大约是Python的一半。

使用“魔术”命令运行 (Run with “magic” command)

%load_ext julia.magic

To use the magic command, we first need to load it.Please note that this method will work only in Jupyter Notebooks.%%time

要使用magic命令，首先需要加载它。请注意，此方法仅在Jupyter Notebooks中有效。%% time

tokens = %julia TokenDocument($data)%julia stem!($tokens)stem_list = %julia tokens($tokens)>>> CPU times: user 849 ms, sys: 3.39 ms, total: 852 ms>>> Wall time: 845 ms

So, after we load the julia.magic, we run those three lines of codes.

因此，在加载julia.magic之后，我们运行了这三行代码。

We should pay attention to two things:

我们应该注意两件事：

put the % symbol before using syntax from Julia.

在使用Julia的语法之前，请先放置％符号。

Use the $ sign before referring to a python variable.

在引用python变量之前，请使用$符号。

And like before, we first tokenized the data, then stem it, and finely store it as a Python list.

和以前一样，我们首先标记数据，然后将其提取出来，然后将其作为Python列表进行精细存储。

使用Julia脚本运行 (Run with Julia script)

In this method, we can write an entire independent Julia script and call it from python.

在这种方法中，我们可以编写一个完整的独立Julia脚本并从python调用它。

So, if we write a script named julia_stemming.jl, and store there this Julia function:

因此，如果我们编写一个名为julia_stemming.jl的脚本并在其中存储以下Julia函数：

function stemming_document(document_string) tokens_data = TokenDocument(document_string) stem!(tokens_data) stem_tokens_data = tokens(tokens_data) return stem_tokens_dataend

We can just call this function from our Python script/notebook:

我们可以从Python脚本/笔记本中调用此函数：

%%timejl.eval('include("julia_stemming.jl")')Main.data = datastem_list = jl.eval("stemming_document(data)")>>> CPU times: user 602 ms, sys: 3.24 ms, total: 606 ms>>> Wall time: 602 ms

First, we include this script to make it usable, and then we call the function inside it.

首先，我们包含此脚本以使其可用，然后在其中调用该函数。

最终速度测试 (Final speed test)

We showed that for this specific dataset (the shakespeare-hamlet.txt), Julia is more than two times faster than Python. But what happens when we use larger datasets?

我们证明，对于这个特定的数据集( shakespeare-hamlet.txt )，Julia比Python快两倍以上。 但是，当我们使用更大的数据集时会发生什么？

We executed our stemming function multiple times with varying dataset length (duplicating our original dataset), and plotted the results:

我们使用可变的数据集长度(复制原始数据集)多次执行了词干函数，并绘制了结果：

Image by Author 图片作者

As you can see, the largest the dataset, the more significant the difference between the run time — which makes Julia very useful when dealing with large datasets and for loops.

如您所见，数据集越大，运行时间之间的差异就越明显-这使得Julia在处理大型数据集和for循环时非常有用。

We used this code to calculate the difference:

我们使用以下代码来计算差异：

time_list_python = []time_list_julia = []dataset_length = []duplicates_list = [1,2,5,10]for t in duplicates_list: data_repeat = ' '.join([data] * t) dataset_length.append(len(data_repeat)) time_start_python = time.perf_counter() stemming_python(data_repeat) time_end_python = time.perf_counter()query_time_python = time_end_python - time_start_python time_list_python.append(query_time_python) time_start_julia = time.perf_counter() Main.data_repeat = data_repeat jl.eval("stemming_document(data_repeat)") time_end_julia = time.perf_counter()query_time_julia = time_end_julia - time_start_julia time_list_julia.append(query_time_julia)

And this code to plot:

并绘制以下代码：

import pandas as pdimport matplotlib.pyplot as pltdf = pd.DataFrame({"running_time_python" : time_list_python, "running_time_julia" : time_list_julia, "dataset_length" : dataset_length})df.set_index("dataset_length", inplace=True)df.plot(figsize = (10,6), title = "Python vs Julia running time comparison")plt.ylabel('seconds', fontsize = 20)plt.xlabel('dataset_length', fontsize = 20)plt.show()

概要 (Summary)

I hope I convinced you about the advantages of connecting Julia and Python.

我希望我使您相信连接Julia和Python的优势。

From what I have seen, there are still multiple areas that pure Python is much better — like working with vectorized computing (such as NumPy), so we need to make sure that switching to Julia is worth the trouble.

从我所看到的情况来看，纯Python仍然有很多方面要好得多-例如使用矢量化计算(例如NumPy)，因此我们需要确保切换到Julia值得解决。

Thank you for reading, and sure, I would be glad to hear your response.

感谢您的阅读，当然，我很高兴听到您的回复。

翻译自: https://towardsdatascience.com/how-to-embed-your-julia-code-into-python-to-speed-up-performance-e3ff0a94b6e

python julia