bigbird丁鹏
Last year, BERT was released by researchers at Google, which proved to be one of the efficient and most effective algorithm changes since RankBrain. Looking at the initial results, BigBird is showing similar signs!
去年，BERT被Google研究人员发布，这被证明是自RankBrain以来最有效，最有效的算法更改之一。 观察初步结果，BigBird表现出相似的迹象！
In this article, I’ve covered:
在本文中，我介绍了：
A brief overview of Transformers-based Models, 基于变压器的模型的简要概述， Limitations of Transformers-based Models, 基于变压器的模型的局限性 What is BigBird, and 什么是BigBird，以及 Potential applications of BigBird. BigBird的潜在应用。Let’s begin!
让我们开始！
Natural Language Processing (NLP) has improved quite drastically over the past few years and Transformers-based Models have a significant role to play in this. Still, there is a lot to uncover.
在过去的几年中，自然语言处理(NLP)取得了巨大的进步，基于变压器的模型在其中发挥着重要作用。 尽管如此，仍有很多发现。
Transformers — a Natural Language Processing Model launched in 2017, are primarily known for increasing the efficiency of handling & comprehending sequential data for tasks like text translation & summarization.
变形金刚—一种于2017年推出的自然语言处理模型，主要以提高处理和理解诸如文本翻译和摘要之类的任务的顺序数据的效率而闻名。
Unlike Recurrent Neural Networks (RNNs) that process the beginning of input before its ending, Transformers can parallelly process input and thus, significantly reduce the complexity of computation.
与递归神经网络(RNN)在输入结束之前处理输入的开始不同，变形金刚可以并行处理输入，因此可以显着降低计算的复杂性。
BERT, one of the biggest milestone achievements in NLP, is an open-sourced Transformers-based Model. A paper introducing BERT, like BigBird, was published by Google Researchers on 11th October 2018.
BERT是NLP的最大里程碑成就之一，它是一个基于Transformers的开源模型。 Google研究人员于2018年10月11日发表了一篇介绍BERT的论文 ，例如BigBird。
Bidirectional Encoder Representations from Transformers (BERT) is one of the advanced Transformers-based models. It is pre-trained on a huge amount of data (pre-training data sets) with BERT-Large trained on over 2500 million words.
变压器的双向编码器表示(BERT)是基于变压器的高级模型之一。 它经过大量数据的预训练(预训练数据集)，而BERT-Large则接受了超过25亿个单词的训练。
Having said that, BERT, being open-sourced, allowed anyone to create their own question answering system. This too contributed to its wide popularity.
话虽如此，BERT是开源的，它允许任何人创建自己的问题回答系统。 这也为它的广泛普及做出了贡献。
But BERT is not the only contextual pre-trained model. It is, however, deeply bidirectional, unlike other models. This is also one of the reasons for its success and diverse applications.
但是BERT并不是唯一的上下文预训练模型。 但是，它是双向的，与其他模型不同。 这也是其成功和多样化应用的原因之一。
Source资源The results of this pre-trained model are definitely impressive. It was successfully adopted for many sequence-based tasks such as summarization, translation, etc. Even Google adopted BERT for understanding the search queries of its users.
经过预训练的模型的结果绝对令人印象深刻。 它已成功用于许多基于序列的任务，例如摘要，翻译等。甚至Google也采用BERT来理解其用户的搜索查询。
But BERT, like other Transformers-Based Models, has its own limitations.
但是，像其他基于变压器的模型一样，BERT也有其自身的局限性。
While Transformers-Based Models, especially BERT, are much improved and efficient than RNNs, they come with a few limitations.
尽管基于变压器的模型(尤其是BERT)比RNN有了很大的改进和效率，但它们也有一些局限性。
BERT works on a full self-attention mechanism. This leads to a quadratic growth of the computational and memory requirements for every new input token. The maximum input size is around 512 tokens which means this model cannot be used for larger inputs & for tasks like large document summarization.
BERT致力于一种完整的自我注意机制 。 这导致每个新输入令牌的计算和内存需求呈二次方增长。 最大输入大小约为512个令牌，这意味着该模型不能用于较大的输入以及诸如大型文档摘要之类的任务。
This basically means a large string has to be broken into smaller segments before applying them as input. This content fragmentation also causes a significant loss of context which makes its application limited.
这基本上意味着，在将大字符串应用为输入之前，必须将其分成较小的段。 内容碎片还会导致上下文的大量丢失，从而使其应用受到限制。
So, what is Big Bird and how is it different from BERT or any other transformers-based NLP models?
那么， 什么是Big Bird？它与BERT或任何其他基于变压器的NLP模型有何不同？
As mentioned earlier, one of the major limitations of BERT and other transformers-based NLP models was because they ran on a full self-attention mechanism.
如前所述，BERT和其他基于变压器的NLP模型的主要限制之一是因为它们基于完全的自我注意机制。
This changed when researchers at Google published a paper on arXiv titled “Big Bird: Transformers for Longer Sequences”.
当Google的研究人员在arXiv上发表题为“大鸟：更长序列的变形金刚”的论文时，这种情况发生了变化。
BigBird runs on a sparse attention mechanism that allows it to overcome the quadratic dependency of BERT while preserving the properties of full-attention models. The researchers also provide instances of how BigBird supported network models surpassed the performance levels of previous NLP models as well as genomics tasks.
BigBird在稀疏的注意力机制上运行，该机制使其能够克服BERT的二次依赖性，同时保留全注意力模型的属性。 研究人员还提供了BigBird支持的网络模型如何超越以前的NLP模型的性能水平以及基因组学任务的实例。
Source 资源Before we move onto the possible applications of BigBird, let’s look at the key highlights of BigBird.
在继续介绍BigBird的可能应用之前，让我们看一下BigBird的主要亮点。
Here are some of the features of BigBird that make it better than previous transformer-based models.
这是BigBird的一些功能，这些功能使其比以前的基于变压器的模型更好。
Sparse Attention Mechanism
稀疏注意机制
Let’s say that you are given a picture and are asked to create a relevant caption for it. You will start by identifying the key object in that picture, say a person throwing a “ball”.
假设您得到一张图片，并被要求为其创建相关标题。 您将从识别图片中的关键对象开始，比如说一个人扔了一个“球”。
Identifying this main object is easy for us, as humans, but streamlining this process for computer systems is a big deal in NLP. Attention mechanisms were introduced to reduce the complexity of this entire process.
作为人类，对于我们而言，确定主要目标很容易，但是简化计算机系统的此过程对NLP来说意义重大。 引入了注意机制以降低整个过程的复杂性。
BigBird uses Sparse Attention Mechanism which enables it to process
BigBird使用稀疏注意力机制，使其能够处理
sequences of length up to 8x more than what was possible with BERT. Keep in mind that this result can be achieved using the same hardware as of BERT.
序列的长度是BERT的8倍。 请记住，可以使用与BERT相同的硬件来获得此结果。
In the said paper of BigBird, researchers show how the Sparse Attention mechanism used in BigBird is as powerful as the full self-attention mechanism (used in BERT). Besides this, they also show “how Sparse encoder-decoders are also Turing Complete”.
在BigBird的上述论文中，研究人员展示了BigBird中使用的稀疏注意力机制与完整的自我注意力机制(在BERT中使用)一样强大。 除此之外，他们还展示了“稀疏编码器-解码器如何同时实现图灵完成”。
In simpler words, BigBird uses the Sparse Attention mechanism which means the attention mechanism is applied token by token, unlike BERT where the attention mechanism is applied to the entire input just once!
用简单的话来说，BigBird使用稀疏注意机制，这意味着注意机制是逐个令牌地应用的，而BERT只是一次将注意机制应用于整个输入！
Can Process Up to 8x Longer Input Sequence
可以处理多达8倍的较长输入序列
One of the key features of BigBird is its capability to handle 8x Longer Sequences than what was previously possible.
BigBird的主要功能之一是其处理8倍长序列的能力比以前可能的高。
The team of researchers designed BigBird to meet all the requirements of full transformers like BERT.
研究人员团队设计了BigBird来满足BERT等完整变压器的所有要求。
Using BigBird and its Sparse Attention mechanism, the team of researchers decreased the complexity of O(n²) (of BERT) to just O(n). This means that the input sequence which was limited to 512 tokens is now increased to 4096 tokens (8 * 512).
使用BigBird及其稀疏注意力机制，研究人员团队将BERT的O(n²)的复杂度降低到O(n)。 这意味着限制为512个令牌的输入序列现在增加到4096个令牌(8 * 512)。
Philip Pham, one of the researchers who created BigBird, says in a Hacker News discussion — “In most of our paper, we use 4096, but we can go much larger 16k+.”
创建BigBird的研究人员之一菲利普·范(Philip Pham)在一次Hacker News讨论中表示 ： “在我们的大多数论文中，我们使用4096，但我们可以将其放大到16k +。”
Pre-trained Larger Data-set
预先训练的较大数据集
Google researchers used 4 different datasets in pre-training of BigBird — Natural Questions, Trivia-QA, HotpotQA-distractor, & WikiHop.
Google研究人员在BigBird的预训练中使用了4个不同的数据集- 自然问题 ， Trivia-QA ， HotpotQA-distractor 和 WikiHop 。
While the collective pre-training data-set of BigBird is not nearly as large as that of GPT-3 (trained on 175 billion parameters), Table 3 from the research paper shows that it performs better than RoBERTa — A Robustly Optimized BERT Pretraining Approach, and Longformer — A BERT-like model for long documents.
尽管BigBird的集体预训练数据集不及GPT-3的集体训练数据集(经过1750亿个参数训练)，但该研究论文的表3显示，它的性能优于RoBERTa —鲁棒优化的BERT预训练方法和Longformer —类似于BERT的长文档模型。
When a user asked Philip Pham to compare GPT-3 to BigBird, he said — “GPT-3 is only using a sequence length of 2048. BigBird is just an attention mechanism and could actually be complementary to GPT-3.”
当用户要求Philip Pham将GPT-3与BigBird进行比较时，他说- “ GPT-3仅使用2048的序列长度。BigBird只是一种关注机制，实际上可以与GPT-3互补。”
A paper introducing BigBird was introduced very recently — Jul 28, 2020. As such the full potential of BigBird is yet to be determined.
最近介绍了介绍BigBird的论文— 2020年7月28日。因此，BigBird的全部潜力尚未确定。
But here are a few possible areas where it can be applied. A few of these applications are also proposed by the creators of BigBird in the original research paper.
但是这里有一些可能适用的领域。 BigBird的创建者还在原始研究论文中提出了其中的一些应用。
Genomics Processing
基因组学处理
There has been an increase in the usage of deep learning for genomics data processing. The encoder takes fragments of DNA sequence as input for tasks such as — methylation analysis, predicting functional effects of non-coding variants, and more.
深度学习在基因组数据处理中的使用有所增加。 编码器将DNA序列的片段作为诸如甲基化分析，预测非编码变体的功能效应等任务的输入。
Creators of BigBird say that: “we introduce a novel application of attention-based models where long contexts are beneficial: extracting contextual representations of genomics sequences like DNA”.
BigBird的创建者说： “我们引入了一种基于注意力的模型的新颖应用，在这种应用中，长情境是有益的：提取基因组序列(如DNA)的情境表示”。
Upon using BigBird for Promoter Region Prediction, the paper claim to have improved the accuracy of the final results by 5%!
使用BigBird进行启动子区域预测后，该论文声称将最终结果的准确性提高了5％！
Long Document Summarization & Question Answering
长文档摘要和问题解答
Since BigBird can now handle up to 8x longer sequence lengths, it can be used for NLP tasks such as summarization of longer document form & question answering. During the creation of BigBird, the researchers also tested its performance for these tasks and witnessed “state-of-the-art results”.
由于BigBird现在可以处理多达8倍的更长序列长度，因此可以用于NLP任务，例如总结更长的文档格式和回答问题。 在创建BigBird的过程中，研究人员还测试了其在这些任务中的性能，并见证了“最新成果”。
BigBird for Google Search
Google搜寻的BigBird
Google started using BERT in October 2019 for understanding search queries and displaying more relevant results for their users. The ultimate goal of updating search algorithms by Google is to understand search queries better than usual.
Google于2019年10月开始使用BERT ，以了解搜索查询并为用户显示更多相关结果。 Google更新搜索算法的最终目标是比平常更好地了解搜索查询。
With BigBird outperforming BERT in Natural Language Processing (NLP), it makes sense to start using this newly founded and more effective model to optimize search result queries by Google.
由于BigBird在自然语言处理(NLP)中的表现优于BERT，因此有必要开始使用这种新近建立的更有效的模型来优化Google的搜索结果查询。
Web & Mobile App Development
网络和移动应用开发
Natural Language Processing has progressed significantly over the decade. With a GPT-3 powered platform that can turn your simple statements into a functioning web app (along with code) already in place, AI developers can truly transform the way you develop your web & web apps.
在过去的十年中，自然语言处理取得了长足的进步。 借助GPT-3驱动的平台，该平台可以将您的简单语句转换为已经可用的Web应用程序(以及代码)， AI开发人员可以真正改变您开发Web和Web应用程序的方式。
演示地址
Since BigBird can handle longer input sequences than GPT-3, it can be used with GPT-3 to efficiently & quickly create web & mobile apps for your business.
由于BigBird比GPT-3可以处理更长的输入序列，因此可以与GPT-3一起使用，以高效，快速地为您的企业创建Web和移动应用程序 。
While there is a lot about BigBird that is left yet to explore, it definitely has the capability of completely revolutionizing Natural Language Processing (NLP) for good. What are your thoughts on BigBird and its contribution to the future of NLP?
尽管还有很多关于BigBird的东西尚待探索，但它肯定具有彻底彻底改变自然语言处理(NLP)的能力。 您对BigBird及其对NLP未来的贡献有何看法？
[2]Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv.org
[2] Jacob Devlin，Ming-Wei Chang，Kenton Lee，Kristina Toutanova，BERT：用于语言理解的深度双向变压器的预训练， arXiv.org
翻译自: https://towardsdatascience.com/understanding-bigbird-is-it-another-big-milestone-in-nlp-e7546b2c9643
bigbird丁鹏
相关资源：四史答题软件安装包exe