fastai ulmfit

Last month, I started the fastai MOOC(2019) - Deep Learning for Coders. It covers various topics like Computer Vision, Natural Language Processing, Collaborative Filtering, etc. The most fascinating part I found was the application of Transfer Learning in the field of Natural Language Processing. The approached used by fastai is Universal Language Model Fine-tuning (ULMFiT), which is a transfer learning methodology applied in the field of Natural Language Processing.

上个月，我开始了Fastai MOOC(2019)-面向程序员的深度学习。 它涵盖了各种主题，例如计算机视觉，自然语言处理，协作过滤等。我发现最有趣的部分是转移学习在自然语言处理领域的应用。 fastai使用的方法是通用语言模型微调( ULMFiT ) ，这是一种应用于自然语言处理领域的转移学习方法。

According to Wikipedia, Transfer learning (TL) is a research problem in machine learning (ML) that focuses on storing knowledge gained while solving one problem and applying it to a different but related problem. For example, knowledge gained while learning to recognize cars could apply when trying to recognize trucks.

根据Wikipedia的说法，转移学习(TL)是机器学习(ML)中的一个研究问题，其重点是存储在解决一个问题并将其应用于其他但相关的问题时获得的知识。 例如，在尝试识别卡车时可以应用在学习识别汽车时获得的知识。

To show the use case of ULMFiT, I’ll apply the same on the Real or Not? NLP with Disaster Tweets competition on Kaggle. To give you an overview of the data, the data set contains two CSV files train.csv and test.csv representing the training data set and the test data set respectively. The training set contains the tweet data in text column and target value in target column, the value of which is 1 if it is a real disaster or 0 if it is not a real disaster. The test set contains only tweet data and no target values. The task is to predict whether a tweet represents a real disaster or not.

为了显示ULMFiT的用例，我将在Real或Not上应用相同的内容吗？ NLP与Kaggle的Disaster Tweets比赛。 为了给您数据的概览，数据集包含两个CSV文件train.csv 和test.csv代表训练数据集和测试数据集。 训练集在text列中包含推文数据，在target列中包含target ，如果这是真正的灾难，则值为1如果不是真正的灾难，则0 。 测试集仅包含tweet数据，而没有目标值。 任务是预测一条推文是否代表真正的灾难。

The paragraph given below is taken from Understanding building blocks of ULMFiT:

以下给出的段落摘自了解ULMFiT的构建基块：

High level idea of ULMFiT is to train a language model using a very large corpus like Wikitext-103 (103M tokens), then to take this pre-trained model’s encoder and combine it with a custom head model, e.g. for classification, and to do the good old fine tuning using discriminative learning rates in multiple stages carefully. Architecture that ULMFiT uses for it’s language modeling task is an AWD-LSTM. The name is an abbreviation of ASGD Weight-Dropped LSTM.

ULMFiT的高级概念是使用超大型语料库(如Wikitext-103(103M令牌))训练语言模型，然后采用此经过预先训练的模型的编码器，并将其与自定义头部模型结合起来，例如用于分类，并进行使用区分学习率在多个阶段仔细地进行良好的旧微调。 ULMFiT用于其语言建模任务的体系结构是AWD-LSTM 。 该名称是ASGD减重LSTM的缩写。

Refer to this paper written by Jeremy Howard and Sebastian Ruder if you want to read more about ULMFiT.

如果您想了解有关ULMFiT的更多信息，请参考Jeremy Howard和Sebastian Ruder撰写的本文。

In Computer Vision problems, transfer learning is used to help in classification directly but in the case of NLP, we first build a language model that basically predicts the next word of a sentence(the model has to understand the language in which the text is written(e.g., English, etc.)) and then build our classification model using the language model’s encoder and vocabulary.

在计算机视觉问题中，转移学习可用于直接帮助分类，但是在NLP的情况下，我们首先建立一个语言模型，该模型基本上可以预测句子的下一个单词(该模型必须了解书写文字的语言(例如英语等)，然后使用语言模型的编码器和词汇表来构建我们的分类模型。

As mentioned in the paper, we will be using AWD-LSTM architecture pre-trained on Wikipedia. We could directly use this pre-trained language model to build our disaster tweet classifier but the gist here is that the English language of Wikipedia would be different from the English language of the tweets. So we will fine-tune our pre-trained language model using the tweet data and then build our classifier on top of that. As explained in the Deep Learning for Coders with Fastai and PyTorch book where they used the same pre-trained architecture on the IMDb Reviews data set to classify if the review is positive or negative. They explain that the IMDb Reviews English is more informal, contains names of movies, directors, actors, etc. than the regular Wikipedia English on which the architecture is pre-trained on.

如本文所述，我们将使用在Wikipedia上经过预训练的AWD-LSTM体系结构。 我们可以直接使用这种经过预先训练的语言模型来构建灾难鸣叫分类器，但是要点在于，维基百科的英语将与鸣叫的英语有所不同。 因此，我们将使用tweet数据微调我们的预训练语言模型，然后在此基础上构建分类器。 正如使用Fastai和PyTorch编写的《面向程序员的深度学习》中所解释的那样，他们对IMDb评论数据集使用了相同的经过预训练的体系结构来对评论是肯定还是否定进行分类。 他们解释说，IMDb评论英语比预训练该体系结构的常规维基百科英语更为非正式，包含电影，导演，演员等的名字。

让我们开始吧！！ (Let’s get started!!)

We import our test.csv and train.csv to get our training data set and test data set. I cannot share the data set as it is from a Kaggle Competition, you can download it yourself by logging into your account and adhering to the competition rules.

我们导入test.csv和train.csv以获得训练数据集和测试数据集。 我无法共享Kaggle竞赛中的数据集，您可以通过登录帐户并遵守竞赛规则自行下载。

# Importing Pandasimport pandas as pd# Importing fastai libraries for text and callbacksfrom fastai.text import *from fastai.callbacks import *train = pd.read_csv('train.csv')test = pd.read_csv('test.csv')

I won’t be going into details on how the language model is built and you should really check out the MOOC and also the Deep Learning for Coders with Fastai and PyTorch book for further reading. The basic idea is that text data cannot be directly fed to the model and it needs to be converted to numbers so that we can apply our mathematical functions to it. This is done using Tokenization and Numericalization. In Tokenization, we convert the text into a list of tokens and in Numericalization we convert them into numbers based on their index. You should refer to the book if you want to dive deeper.

我不会详细介绍语言模型的构建方式，您应该真正阅读MOOC以及带有Fastai和PyTorch的《面向程序员的深度学习》一书，以进一步阅读。 基本思想是文本数据不能直接输入模型，需要将其转换为数字，以便我们可以将数学函数应用于该模型。 这是通过符号化和Numericalization完成。 在符号化，我们将文本转换成标记列表，并在Numericalization我们把它们转换成基于其索引号。 如果您想更深入地学习，请参考这本书。

We’ll be using the Data Block API that does the above for us under the hood and then we will feed the databunch created to our language model.

我们将使用在后台为我们执行上述操作的数据块API ，然后将创建的数据集提供给我们的语言模型。

data_lm = (TextList.from_df(pd.concat([train[['text']], test[['text']]], ignore_index=True, axis=0)) .split_by_rand_pct(0.15) .label_for_lm() .databunch(bs=128))data_lm.show_batch()

We ignore the labels and take the text corpus from both training and test data. Remember that we are making a language model and not a classification model right now. We are just including text data as much as we can for our language model to predict the next word of a sentence. Next, we use our data_lm databunch to make our language model.

我们忽略标签，并从训练和测试数据中获取文本语料库。 请记住，我们现在正在制作语言模型，而不是分类模型。 我们只是尽可能多地包含文本数据，以便我们的语言模型预测句子的下一个单词。 接下来，我们使用data_lm databunch创建语言模型。

learn = language_model_learner(data_lm, AWD_LSTM, drop_mult = 0.5)learn.lr_find()learn.recorder.plot(suggestion = True) learn.fit_one_cycle(1, 1e-2, moms=(0.8,0.7))

We’ll be unfreezing the model and we’ll fit more. We’ll use callbacks to select the best model.

我们将解冻模型，并且我们将使其更合适。 我们将使用回调来选择最佳模型。

Better model found at epoch 0 with accuracy value: 0.4097544550895691.Better model found at epoch 1 with accuracy value: 0.4404464364051819.Better model found at epoch 2 with accuracy value: 0.4609375.Better model found at epoch 3 with accuracy value: 0.47495537996292114.Better model found at epoch 4 with accuracy value: 0.48810267448425293.Better model found at epoch 5 with accuracy value: 0.49515628814697266.Better model found at epoch 6 with accuracy value: 0.4975222945213318.Better model found at epoch 9 with accuracy value: 0.49756699800491333.

We’ll select the best accuracy which is at epoch number 9. We’ll then save the language model and the encoder.

我们将选择最佳精度，即第9个时间点。然后，我们将保存语言模型和编码器。

learn.save('fine_tuned')learn.save_encoder('fine_tuned_enc')

Next, we’ll make our classifier. For that, we’ll need to create a new databunch. We’ll take the validation set as 10% and we’ll keep our vocabulary same as the vocabulary of the language databunch. We’ll also add our test data in the separate add_test parameter.

接下来，我们将进行分类。 为此，我们需要创建一个新的数据束。 我们将验证集设为10％，并将词汇表与语言数据绑定的词汇表保持一致。 我们还将测试数据添加到单独的add_test参数中。

data_clas = (TextList.from_df(df, vocab=data_lm.vocab) .split_by_rand_pct(0.1) .label_from_df('target') .add_test(TextList.from_df(test['text'], vocab=data_lm.vocab)) .databunch(bs=128))

We’ll build the classifier using the same encoder as our language model.

我们将使用与我们的语言模型相同的编码器来构建分类器。

learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5, metrics=[accuracy, FBeta(beta=1)])learn.load_encoder('fine_tuned_enc')

We’ll do a lr_find() check and then plot to see the graph.

我们将进行一次lr_find()检查，然后绘图以查看该图。

Let’s fit one cycle. We see that we get an accuracy of 77.66%.

让我们适应一个周期。 我们看到我们得到的准确度是77.66 ％。

learn.fit_one_cycle(1, 1e-3, moms=(0.8,0.7))

Let’s unfreeze the last 2 layers and train for one cycle. Our accuracy increases to 79.5%.

让我们解冻最后两层并训练一个周期。 我们的准确性提高到79.5 ％。

learn.freeze_to(-2)learn.fit_one_cycle(1, slice(1e-3/(2.6**4),1e-2), moms=(0.8,0.7))

We’ll unfreeze the last 3 layers now and train again for one more cyle. Our accuracy increases to 81.73%!

现在，我们将解冻最后三层，并再次训练一个循环。 我们的准确性提高到81.73％ ！

learn.freeze_to(-3)learn.fit_one_cycle(1, slice(5e-3/(2.6**4),5e-3), moms=(0.8,0.7))

What we used here were discriminative learning rates which were introduced in ULMFiT. As explained in the article 10 New Things I Learnt from fast.ai v3:

我们在这里使用的是歧视性学习率，这是ULMFiT中引入的。 如文章10我从fast.ai v3中学到的新知识中所述：

Discriminative learning rates for pre-trained modelsTrain earlier layer(s) with super low learning rate, and train later layers with higher learning rate. The idea is to not drastically alter the almost-perfect pre-trained weights except for minuscule amounts, and to be more aggressive with teaching the layers near the outputs. Discriminative learning rate was introduced in ULMFiT.

预训练模型的判别学习率训练具有极低学习率的较早层，并训练具有较高学习率的较后层。 这样做的想法是，除了微小的数量外，不要大幅度地改变几乎完美的预训练权重，并且在示教输出附近的层时要更加积极。 区分学习率是在ULMFiT中引入的。

We’ll unfreeze all layers, train, and use callbacks to select our best model.

我们将解冻所有层，进行训练，并使用回调来选择最佳模型。

callbacks = SaveModelCallback(learn,monitor="accuracy", mode="max", name="best_classification_model") Better model found at epoch 0 with accuracy value: 0.8160315155982971.Better model found at epoch 1 with accuracy value: 0.8173456192016602.Better model found at epoch 2 with accuracy value: 0.822601854801178.Better model found at epoch 9 with accuracy value: 0.8239158987998962.

We get an accuracy of 82.39%!!

我们的准确性为82.39％ ！

结论 (Conclusion)

In Transfer Learning we use knowledge from a source and apply it to our target. Implementation of the same in the field of Natural Language Processing has provided an extraordinary state of the art results with minimal training as we use a pre-trained network. You can check out my notebook on Kaggle if you would like to see the code in action: NLP - Disaster Prediction ULMFiT

在转移学习中，我们使用来自来源的知识并将其应用于目标。 由于我们使用的是预训练网络，因此在自然语言处理领域中实现相同功能已提供了非凡的最新结果，而无需进行最少的培训。 如果您想查看运行中的代码，可以在Kaggle上查看我的笔记本： NLP-灾难预测ULMFiT

Note: I have used fastai V1 here. The fastai V2 along with the new MOOC came out on 21st August. Check that out here

注意：我在这里使用fastai V1。 Fastai V2和新的MOOC于8月21日问世。 在这里检查

翻译自: https://towardsdatascience.com/fastai-disaster-prediction-using-ulmfit-4dd244d3889c

fastai ulmfit

相关资源：FX_NaiveChartPredictions_using_fastai：使用最新的图像识别算法完成的外汇交易多头头寸预测-源码