辍学的名人
Recent research and our own experiments have shown that it is possible to create neural machine translators that achieve relatively high BLEU scores with small datasets of parallel text.
最近的研究和我们自己的实验表明，可以使用较小的并行文本数据集来创建获得相对较高的BLEU分数的神经机器翻译器。
The trick is to train a smaller model for the smaller dataset.
诀窍是为较小的数据集训练较小的模型。
Training a large model on a small dataset is comparable to estimating a regression model with a large number of parameters on a dataset with few observations: It leaves you with too few degrees of freedom. The model thus becomes over-fit and does not make good predictions.
在较小的数据集上训练大型模型与估算几乎没有观测值的数据集上具有大量参数的回归模型具有可比性：它使您的自由度过低。 该模型因此变得过度拟合，并且无法做出良好的预测。
Reducing the vocabulary with subword-splitting training a smaller network and setting a high-dropout parameter reduce over-fitting. And self-attentional neural networks also reduce over-fitting because (compared to recurrent and convolutional networks) they are less complex. They directly model the relationships between words in a pair of sentences.
通过子词拆分来减少词汇量，训练一个较小的网络并设置一个高辍学参数可以减少过度拟合的情况。 自注意神经网络也减少了过度拟合，因为(与循环和卷积网络相比)它们不那么复杂。 他们直接模拟一对句子中单词之间的关系。
This combination of splitting, dropout and self-attention achieved a BLEU score of 20.3 on English-to-Sicilian translation and 21.4 on Sicilian-to-English with only 13,839 lines of parallel training data containing 190,072 Sicilian words and 195,372 English words.
分裂，辍学和自我注意的结合在英语到西西里语的翻译中获得了20.3的BLEU评分，在西西里语到英语中获得了21.4的BLEU评分，仅13,839行并行训练数据包含190,072个西西里语单词和195,372个英语单词。
And because the networks were small, each model took a little over five hours to train on CPU.
而且由于网络很小，每种型号都花了五个多小时来训练CPU。
Our success is an implementation of the best practices developed by Sennrich and Zhang (2019) with the self-attentional Transformer model developed by Vaswani et al. (2017).
我们的成功是由Sennrich和Zhang(2019)开发的最佳实践与Vaswani等人开发的自注意变压器模型的实施。 (2017) 。
For training, we used the Sockeye toolkit by Hieber et al. (2017) running on a server with four 2.40 GHz virtual CPUs.
为了进行培训，我们使用了Hieber等人的Sockeye工具包。 (2017)在具有四个2.40 GHz虚拟CPU的服务器上运行。
In their best practices for low-resource NMT, Sennrich and Zhang suggest the byte-pair encoding (i.e. subword-splitting) developed by Sennrich, Haddow and Birch (2016), a smaller neural network with fewer layers, smaller batch sizes and larger dropout parameters.
在针对低资源NMT的最佳实践中，Sennrich和Zhang建议由Sennrich，Haddow和Birch(2016)开发的字节对编码(即子字拆分) ，这是一种较小的神经网络，具有较少的层，较小的批处理大小和较大的落差参数。
Using those best practices in the “BiDeep RNN” architecture proposed by Miceli Barone et al. (2017), they achieved a BLEU score of 16.6 on German-to-English translation with only 100,000 words of parallel training data.
使用Miceli Barone等人提出的“ BiDeep RNN”架构中的最佳实践。 (2017) ，他们在仅有10万个单词的并行训练数据的德语到英语翻译中获得了BLEU得分16.6。
Their largest improvements in translation quality came from the application of a byte-pair encoding (i.e. subword-splitting) that reduced the vocabulary from 14,000 words to 2000 words. But their most successful training also occurred when they set high dropout parameters.
他们在翻译质量上的最大改进来自字节对编码(即子词拆分)的应用，该词对将词汇量从14,000个单词减少到2000个单词。 但是，当他们设置较高的辍学参数时，他们也获得了最成功的训练。
During training, dropout randomly shuts off a percentage of units (by setting it to zero), which effectively prevents the units from adapting to each other. Each unit therefore becomes more independent of the others because the model is trained as if it had a smaller number of units, thus reducing over-fitting (Srivastava et al. (2014)).
在训练期间，辍学会随机关闭一定百分比的单位(通过将其设置为零)，这有效地防止了单位之间的相互适应。 因此，每个单元变得彼此之间更加独立，因为模型被训练得好像单元数量较少，从而减少了过度拟合( Srivastava等人(2014) )。
Subword-splitting and high dropout parameters helped us achieve better than expected results with a small dataset, but it was the Transformer model that pushed our BLEU scores into the double digits.
子单词拆分和高辍学参数帮助我们使用较小的数据集获得了比预期更好的结果，但是正是Transformer模型将BLEU得分推到了两位数。
Compared to recurrent neural networks, the self-attention layers in the Transformer model more easily learn the dependencies between words in a sequence because the self-attention layers are less complex.
与递归神经网络相比，Transformer模型中的自我注意层更容易学习序列中单词之间的依存关系，因为自我注意层不那么复杂。
Recurrent networks read words sequentially and employ a gating mechanism to identify relationships between separated words in a sequence. By contrast, self-attention examines the links between all the words in the paired sequences and directly models those relationships. It’s a simpler approach.
循环网络按顺序读取单词，并采用门控机制来识别顺序中分离单词之间的关系。 相反，自我注意会检查配对序列中所有单词之间的链接，并直接对这些关系进行建模。 这是一种更简单的方法。
Combining these three features — subword-splitting, dropout and self-attention — yields a trained model that makes relatively good predictions. And as we add more parallel text to our dataset, the translation quality will improve even more.
结合这三个功能(子词拆分，辍学和自我关注)，可以得出经过训练的模型，可以做出相对较好的预测。 而且，随着我们向数据集中添加更多并行文本，翻译质量将进一步提高。
In the meantime, we invite you to see the results at Napizia.
同时，我们邀请您在Napizia上查看结果。
翻译自: https://medium.com/@eryk_8028/just-split-dropout-and-pay-attention-c185a6374c9d
辍学的名人
