贝叶斯定理
During World War II, cryptoanalysts in both the United States and the UK were frantically trying to decipher encrypted communications sent by the Axis powers’ military branches. Among these efforts, the British code-breaking center at Bletchley Park probably stands out in its fame as the birthplace of the British Bombe, the machine that would decode the infamous German cipher Enigma between 1939 and the end of the war. While the Bombe and its successor models have received most of the public’s attention, their efforts would often have been futile without the support of statistical methods. The goal of this article is to cast light on one of these methods, namely Bayesian inference, first by introducing the theory behind it and then by outlining how it was used to crack Axis ciphers.
第二次世界大战期间，美国和英国的加密分析师都在疯狂地试图破译由轴心国军事分支发送的加密通信。 在这些努力中，位于布莱奇利公园(Bletchley Park)的英国密码破解中心可能以英国邦贝(Bombbe)的发源地而出名，该机器可解码1939年至战争结束之间臭名昭著的德国密码《谜》。 尽管孟买及其后继模型受到了大多数公众的关注，但如果没有统计方法的支持，他们的努力往往是徒劳的。 本文的目的是首先介绍一种方法，即贝叶斯推理，首先介绍其背后的理论，然后概述其如何用于破解Axis密码。
Bletchley Park, photographed by Toby Oxborrow. Source: Flickr. This file is licensed under the 布莱奇利公园的小屋1，由Toby Oxborrow拍摄。 资料来源：Flickr。 此文件已获得 Creative Commons 知识共享 Attribution-Share Alike 2.0 Generic license 署名-相同方式共享2.0通用许可的许可Part I will be dedicated to the breaking of the Japanese Naval cipher JN 25, whereas Part II will outline the role Bayesian inference played in cracking Enigma. Large parts of this report are based on an article written by Edward Simpson, one of the code-breakers who worked at Bletchley Park. I have tried to make the topic more accessible by reviewing the Bayesian formalism and carrying out most of the math that is only implied in Simpson’s article. If at the end of this post, you find yourself intrigued by the topic, I encourage you to read the original article by Simpson for a fascinating first-hand report of his time at Bletchley Park.
第一部分将致力于打破日本海军密码JN 25，而第二部分将概述贝叶斯推理在破解《谜》中的作用。 该报告的大部分内容基于爱德华·辛普森(Edward Simpson)撰写的一篇文章，该文章是在Bletchley Park工作的代码破坏者之一。 我试图通过回顾贝叶斯形式主义并执行仅在Simpson文章中隐含的大部分数学运算，来使该主题更易于访问。 如果在本文结尾处，您发现自己对该主题感兴趣，我鼓励您阅读辛普森(Simpson)的原始文章，以获得有关他在布莱奇利公园(Bletchley Park)的时间的有趣的第一手报道。
Like so many other famous theories, Bayes’ theorem is surprisingly simple:
像许多其他著名理论一样，贝叶斯定理非常简单：
The formula itself is easily derived and has many applications outside of Bayesian statistics. However, its simplicity can be deceiving as most of the theorem’s power lies in the interpretation of the probabilities P involved. The true controversy that has stuck with Bayes’ theorem for centuries lies in the way its usage challenges the more traditional frequentist approach. While so-called frequentists define the probability of an event as the “limit of its relative frequency in many trials” [1]. Bayesians interpret probability as a measure of personal belief.
该公式本身易于导出，并且在贝叶斯统计之外具有许多应用。 但是，它的简单性可能是骗人的，因为大多数定理的能力在于对涉及的概率P的解释。 几个世纪以来一直存在于贝叶斯定理的真正争议在于它的用法挑战了更传统的常客主义方法。 所谓的常客主义者将事件的可能性定义为“在许多试验中其相对频率的极限” [1]。 贝叶斯主义者将概率解释为个人信念的量度。
One may ask how they dare include something as subjective as belief in a mathematical theory. And voicing this critique, one would certainly be in good company. Many heavyweights of statistics, including Fisher and Pearson, have discarded the Bayesian interpretation of probability based on similar arguments [4].
有人可能会问，他们怎么敢在数学理论中包含主观信念。 说出这种批评，肯定会有好伙伴。 许多统计重量级人物，包括Fisher和Pearson，都基于相似的论点放弃了对概率的贝叶斯解释[4]。
To really understand the difference between the two approaches and be able to pass fair judgment, let us consider the following example:
要真正理解两种方法之间的区别并能够通过公正的判断，让我们考虑以下示例：
Imagine a friend challenges you to a game of flipping coins and promptly produces a coin that she would like to use. Before agreeing to this game you, being rather suspicious by nature, would like to ascertain that the coin she handed you is fair, i.e. the chances of it landing on head and tail are equal.
想象一个朋友向您挑战掷硬币游戏，并立即产生她想使用的硬币。 在同意此游戏之前，您本性相当可疑，所以想确定她递给您的硬币是否公平，即它落在头和尾上的机会是相等的。
A frequentist will frame this problem as a hypothesis test with null hypotheses H₀: the coin is fair, and alternative hypotheses H₁ that it is not. She will then decide on a number of trials (say, 𝑛=100), and a confidence level (e.g., 𝛼=0.05). After flipping the coin 𝑛 times, the outcomes are recorded. If we let 𝑘 denote the number of times we observe heads and 𝑝 the probability that the coin lands on heads then, following a binomial distribution, the probability of a particular outcome is given as
一个常客将这个问题作为假设检验，用零假设H₀进行解释：硬币是公平的，而替代假设H₁不是。 然后，她将决定进行多次试验(例如𝑛= 100)和置信度(例如𝛼 = 0.05)。 翻转硬币𝑛次后，将记录结果。 如果我们用𝑘表示观察到正面的次数，并且let使硬币落在正面上的概率，则按照二项式分布，给出特定结果的概率为
Given the confidence level 𝛼 she can then compute a rejection region, meaning intervals of 𝑘 for which the null hypotheses H₀:𝑝=0.5 can be safely rejected. This region can be obtained by solving
给定置信度𝛼之后，她可以计算拒绝区域，这意味着可以安全拒绝否定假设H₀：𝑝= 0.5的间隔𝑘。 可以通过求解获得该区域
for 𝑘*. Thankfully people have dealt with these kinds of problems in the past so rather than actually solving the equation herself she can simply look up its solution in a table or use any statistical software to give her the answer. It turns out that for this problem, 𝑘*=10, so H₀ can be rejected if |𝑘−50|>10, or equivalently if 𝑘<40 or 𝑘>60.
𝑘*。 值得庆幸的是，过去人们已经解决了这类问题，因此与其直接自己解决方程式，不如直接在表中查找其解决方案或使用任何统计软件为她提供答案。 事实证明，对于此问题，𝑘* = 10，因此如果| 𝑘−50 |> 10，或者等效地如果𝑘<40或𝑘> 60，则可以拒绝H₀。
Let us assume that out of 100 coin flips, (𝑘=) 73 came back as heads. The frequentist can now conclude that the coin is rigged and that the chance that her conclusion is wrong is less than 5% (the confidence level).
让我们假设在100次硬币翻转中，(𝑘=)73成为正面。 现在，常客可以得出结论：该硬币已被操纵，并且得出错误结论的可能性小于5％(置信度)。
You, a firm supported of Bayesian statistics, enter the scene. You are appalled by the “waste of time” and suggest that all this could have been done with much fewer trials. To understand how you then proceed, let us first revisit Bayes’ theorem:
支持贝叶斯统计的公司您进入了现场。 您对“时间浪费”感到震惊，并建议所有这些工作都可以通过更少的试验来完成。 要了解您如何进行，请首先回顾一下贝叶斯定理：
Notice that we have replaced the variables A, B by more meaningful symbols 𝜃 and 𝐷. 𝜃 denotes the model’s parameters, which in our case is just the probability of heads 𝑝, a characteristic that is inherent to the coin used and the unknown value we are trying to infer. 𝐷 stands for the observed data, i.e. the number of times the coin lands on heads and tails.
注意，我们已经用更有意义的符号𝜃和replaced替换了变量A，B。 𝜃表示模型的参数，在我们的情况下，该值仅是正面probability的概率，这是所用硬币固有的特性以及我们试图推断的未知值。 𝐷代表观察到的数据，即硬币落在正面和反面的次数。
𝑃(𝜃|𝐷) is the probability of 𝜃 given the evidence or data 𝐷. In other words, the probability that our coin’s 𝑝 has a particular value after having landed on heads a given number of times. This is called the posterior.
𝑃(𝜃 |𝐷)是给定证据或数据𝐷的概率。 换句话说，我们的硬币𝑝落在头部上一定次数后具有特定值的概率。 这称为后验。
𝑃(𝐷|𝜃) answers the following question: what is the likelihood of observing the data 𝐷 given 𝜃? This is something the frequentist computes as well and it is called the likelihood.
𝑃(𝐷| 𝜃)回答以下问题：观察给定𝐷的数据what的可能性是多少？ 这也是常客计算的东西，称为可能性。
𝑃(𝜃) is the statistician’s belief in how likely different values for 𝜃 are before observing any data. It is commonly known as the model prior
𝑃(𝜃)是统计学家对any在观察任何数据之前different的不同值具有多大可能性的信念。 通常称为先验模型
𝑃(𝐷) is usually called the marginal and describes the probability of observing the data, independent of the model parameters. It ensures that the posterior distribution is normalized, but we can often find ways to avoid its direct computation.
𝑃(𝐷)通常称为边际函数，它描述观测数据的概率，而与模型参数无关。 它可以确保后验分布是规范化的，但是我们经常可以找到避免对其进行直接计算的方法。
The first thing we need to do is to decide on a model to describe the coin-tossing experiment. Just as before, the binomial distribution is our function of choice:
我们需要做的第一件事是确定一个描述投币实验的模型。 和以前一样，二项分布是我们选择的功能：
The data D comprises both the number of tosses 𝑛 and the number of heads 𝑘. Having defined the likelihood, we now have to pick a prior. Remember, the prior characterizes the statistician’s belief in parameter 𝜃 before seeing any data. So, if we are dealing with a trustworthy friend, we, as Bayesians, can pick a prior that has a sharp peak around 𝜃=0.5 (the value for a fair coin), as shown by the orange curve. If our friend has tried to pull tricks on us in the past, we might be more inclined to pick the blue curve as prior. It is wider and therefore puts less belief in any particular value for 𝜃.
数据D包括抛掷数𝑛和头部数𝑘。 确定了可能性之后，我们现在必须选择一个先验。 请记住，先验特征是统计学家在看到任何数据之前对参数belief的信念。 因此，如果我们要与一个值得信赖的朋友打交道，我们作为贝叶斯人可以选择一个先验先验，先验先验在𝜃 = 0.5(公平硬币的价值)附近有一个尖峰，如橙色曲线所示。 如果我们的朋友过去曾试图欺骗我们，我们可能会更倾向于选择蓝色曲线。 它更宽，因此对any的任何特定值的信任程度降低。
As you’ve probably noticed, we didn’t write down any equations for the prior. Unfortunately, the math involved in Bayesian inference can be rather tricky. Oftentimes integrals cannot be computed analytically and one has to resort to numerical tools such as Monte-Carlo sampling. For that reason, we will rely on graphs to develop an intuitive understanding of Bayesian statistics.
正如您可能已经注意到的那样，我们没有写下任何先前的方程式。 不幸的是，贝叶斯推理中涉及的数学可能非常棘手。 通常，积分无法通过分析来计算，因此必须求助于诸如蒙特卡洛采样之类的数值工具。 因此，我们将依靠图来发展对贝叶斯统计的直观理解。
Let’s say out of 6 tosses, the coin lands on heads 4 times. Using the “Low Trust” prior, the posterior distribution 𝑃(𝜃|𝐷) will look like this:
比方说，在6次抛掷中，硬币正面落了4次。 使用“低信任度”先验，后验分布𝑃(𝜃 |𝐷)将如下所示：
As you can see, the distribution has shifted to larger values of 𝜃 because heads appeared twice as often as tails. We can interpret the likelihood term as a filter acting on the prior distribution, only letting through values for 𝜃 that are more or less consistent with the data 𝐷.
如您所见，由于头部出现的频率是尾部的两倍，因此分布已转移到更大的𝜃值。 我们可以将似然项解释为作用于先验分布的过滤器，仅让𝜃的值与数据more或多或少地保持一致。
The distribution is still rather wide, so we conclude that to make any reliable inference, we need more samples. After 20 tosses and having observed heads 15 times, the posterior looks like this:
分布仍然相当广泛，因此我们得出结论，为了做出可靠的推断，我们需要更多的样本。 经过20次抛掷并观察了15次头部之后，后部看起来像这样：
We can conclude with high certainty that the coin is rigged, and we have been able to do so with 80 fewer coin tosses than the frequentist. Talk about efficient use of resources!
我们可以高度肯定地得出该硬币被操纵的结论，而且我们能够比普通人少投掷80枚硬币。 谈谈有效利用资源！
Now should you agree to play with our friend? Depends. If she lets you pick either heads or tails, bet on heads! If however, she insists on picking heads for herself, maybe it’s time you find a new friend…
现在您应该同意和我们的朋友一起玩吗？ 依靠。 如果她允许您选择正面还是反面，请押头！ 但是，如果她坚持要自己挑头，也许是时候找到一个新朋友了……
One can actually calculate the expected gain for the favorable version of this game: If we bet one dollar on heads our expected return is
实际上，可以计算出该游戏的有利版本的预期收益：如果我们押一美元，我们的预期收益为
As we do not know the exact value for 𝜃 we need to integrate over its probability distribution:
由于我们不知道𝜃的确切值，因此需要对它的概率分布进行积分：
For the last expression, we don’t actually need to compute any integrals. To get a rough approximation for the expectation value (the mean) of 𝜃 under the posterior distribution, it is enough to look at said distribution to find ⟨𝜃⟩≈0.7 (the exact value is about 0.708). So
对于最后一个表达式，我们实际上不需要计算任何积分。 为了在后验分布下获得the的期望值(均值)的粗略近似，只需查看所述分布即可找到⟨𝜃⟩≈0.7(精确值约为0.708)。 所以
With an expected ROI of about 40% per coin toss I would say “go for it”; but before you bet all your money, make sure you’re aware of gambler’s ruin.
每次抛硬币的预期投资回报率约为40％，我会说“去吧”。 但是在押注所有资金之前，请确保您已意识到赌徒的破产。
The USS Arizona (BB-39) burning after the Japanese attack on Pearl Harbor , Public Domain 日本袭击珍珠港后， 美国亚利桑那号(BB-39)燃烧，公共领域The way the Japanese navy encoded their messages during World War II was rather straightforward. The sender would use a code book (I) to transform every word of his message into a five-digit number. As a safety measure, the sum of these five digits would always be divisible by 3. While giving the Japanese a way to make sure their messages had been transmitted without error, this measure would greatly help the allied forces decode intercepted messages; but more about that later.
日本海军在第二次世界大战期间编码信息的方式非常简单。 发件人将使用密码簿(I)将其消息的每个单词转换为五位数的数字。 作为安全措施，这五个数字的总和总是可以被3整除。虽然给日本人提供了一种确保他们的消息已经无误传输的方法，但是这种措施将极大地帮助盟军解码被拦截的消息。 但是稍后会更多。
Having encoded the plain text, the sender would then consult an enciphering table for so-called additives (II). These five-digit numbers would be added (non-carrying) to the code to produce the final enciphered message (III). The receiver would have access to the same code book and additives and could simply decode the message by reversing the above procedure.
对纯文本进行编码后，发送者将查阅加密表以获取所谓的添加剂(II) 。 这些五位数字将被添加(非携带)到代码中以产生最终的加密消息(III) 。 接收者可以访问相同的代码簿和附加信息，并且可以通过颠倒上述过程来简单地解码消息。
Being presented with a set of intercepted messages, the allied cryptoanalysts’ task would be to figure out the right additives. At the same time, so-called book builders would try to reproduce the Japanese code book by relying on a combination of linguistic and combinatoric skills.
收到一组截获的消息后，联盟的密码分析师的任务是找出正确的添加剂。 同时，所谓的书籍制作者将尝试依靠语言和组合技巧的结合来复制日语代码书。
As the cryptoanalysts only had access to a limited amount of evidence, i.e. a given number of intercepted messages, the best they could hope for was to make a probabilistic statement about the most likely additives underlying a message. Therefore, (Bayesian) statistics was particularly suited to tackle this problem.
由于加密分析师只能获得有限的证据，即给定数量的被拦截消息，因此他们所希望的最好是对消息背后最可能的添加剂做出概率陈述。 因此，(贝叶斯)统计数据特别适合解决此问题。
The deciphering process would start with a set of intercepted messages (IV) that were known to have the same additives. These messages were said to be “in depth” and one would speak of a “depth” of messages.
解密过程将从一组已知具有相同添加剂的侦听消息(IV)开始。 据说这些消息是“深度”的，人们会说这是消息的“深度”。
The five-digit ciphers were differenced against each other and the result was recorded (always picking the number below 5555). The logic behind this procedure was that taking differences between encoded messages, the underlying additives would cancel out.The allied forces already had knowledge of part of the Japanese code book, and many of the code groups were known. These were referred to as “good groups” and were tabulated along with their (non-carrying) differences (V). If 50 groups were known, 1225 differences had to be calculated and recorded.
五位数密码彼此不同，并记录结果(始终选择5555以下的数字)。 该程序背后的逻辑是，利用编码消息之间的差异，潜在的添加剂将被抵消。盟军已经掌握了日语代码书的一部分，并且许多代码组都是已知的。 这些被称为“好群体” ，并与它们的(非携带)差异(V)一起制成表格。 如果已知50组，则必须计算和记录1225个差异。
Cryptoanalysts would compare the differences calculated from the intercepted messages and those from good groups. If a match was found (in our case 22571, shown in red), a hypothetical additive was calculated (shown in green).
密码分析人员将比较从截获的消息计算得出的差异与从好的组得出的差异。 如果找到匹配项(在我们的案例中为22571，以红色显示)，则计算出假设的添加剂(以绿色显示)。
The final and most important task was to test this hypothetical additive for its validity. As a first step, the additive was subtracted from its corresponding encoded word in every message (VI). If the resulting code (shown in blue) broke the “divisible by 3” rule, the additive could be quickly discarded, saving the Allies a large amount of time. Usually, multiple potential additives would pass this test and hence statistical analysis was used to determine their relative strengths.As evidence for a true additive, the cryptoanalyst would use the presence of any other good groups in the above test. The math works out like this:
最后也是最重要的任务是测试这种假想添加剂的有效性。 第一步，从每个消息(VI)中相应的编码词中减去添加剂。 如果生成的代码(以蓝色显示)违反了“被3整除”的规则，则可以Swift丢弃添加剂，从而为盟军节省了大量时间。 通常，多种潜在的添加剂会通过此测试，因此使用统计分析来确定其相对强度。作为真正添加剂的证据，密码分析家会在上述测试中使用任何其他良好基团的存在。 数学计算如下：
Let us denote 𝐴 as the event that the additive found is true, and ¬𝐴 that it is not. We want to determine the posterior probability given the data of good groups after the additives were added 𝑃(𝐴|𝐷).
让我们将note表示为发现的添加剂为真，而𝐴则为不表示。 我们想根据添加了𝑃(𝐴|𝐷)的良好组的数据来确定后验概率。
If we are only interested in the the odds ratio (true vs. false), the marginal 𝑃(𝐷) in the denominator cancels out:
如果我们只对优势比感兴趣(真与假)，则分母中的边际𝑃(𝐷)抵消：
Not having any prior knowledge about the probability for A, one can always just use a prior that assigns equal probability to both events, i.e. 𝑃(𝐴)=0.5. The last term in Eq. thereby cancels out and one is left with determining the likelihood ratio, which is fairly straightforward to obtain. (Note that if we had any reason to believe that we are dealing with a correct additive, based on evidence other than the presence of good groups, we could always modify the prior to reflect this belief.)
没有关于A的概率的任何先验知识，总是可以使用为两个事件分配相等概率的先验，即𝑃(𝐴)= 0.5。 等式中的最后一项。 从而抵消，剩下的就是确定似然比，这很容易获得。 (请注意，如果我们有任何理由相信我们是根据正确的添加剂进行处理的，则除了存在良好群体以外的证据以外，我们总是可以修改先验知识以反映这一信念。)
As an example, let us consider the good group 32151 which can be recovered in Message 2. If 98213 is not a true additive, then 32151 is just a random five-digit number. The probability of this number occurring would be 1/10⁵, however, as the number needs to check the “divisible by 3” rule, we obtain 𝑃(32151|¬𝐴)=3/10⁵.
例如，让我们考虑可以在消息2中恢复的良好组32151。如果98213不是真正的加法器，则32151只是一个随机的五位数。 该数字出现的可能性为1 /10⁵，但是，由于该数字需要检查“ 3分频”规则，因此我们获得𝑃(32151 |¬𝐴)= 3 /10⁵。
If 98213 is a true additive, then the likelihood 𝑝(32151|𝐴) is given by the relative frequency of occurrence for 32151 (“Fleet”). Certainly, words such as “Ship” or “Weather” occur more often than others like “Attack”. This needs to be taken into account in a correct statistical treatment, as more frequent terms add more evidence towards the hypothesis. For the sake of this example, let us assume that every 80th intercepted word is “Ship” and every 250th is “Fleet”, then the total evidence for additive 98213 would be calculated as:
如果98213是真正的加法器，则似然𝑝(32151 |𝐴)由32151的相对出现频率给出(“车队”)。 当然，诸如“ Ship”或“ Weather”之类的词比诸如“ Attack”之类的词出现得更多。 在正确的统计处理中需要考虑到这一点，因为更频繁的用语为该假设增加了更多的证据。 对于此示例，让我们假设每个第80个截获单词是“ Ship”，每250个捕获单词是“ Fleet”，那么加法运算符98213的总证据将计算为：
Technically, codes that are not contained in the list of good groups also add factors to the product above. However, as these factors are only slightly less than one, they can be safely ignored for the sake of saving time.
从技术上讲，商品组列表中未包含的代码也会为上述产品添加因素。 但是，由于这些因素仅略小于一个，因此可以节省时间，可以安全地忽略它们。
As a last simplifying trick, the likelihoods would be converted into their logarithmic values, trading complicated multiplication for simple addition. The log-likelihoods for every good group would be tabulated in advance, so that staff could simply look up values and the process of testing additives could be streamlined. One of the main advantages of this procedure was that the somewhat labor-intensive and repetitive task of testing additives could be outsourced to staff with only limited math knowledge. Expert opinion would merely need to be consulted for fringe cases or messages of great importance.
作为最后的简化技巧，可能性将被转换为对数值，将复杂的乘法交易为简单的加法。 每个好组的对数似然率将预先制成表格，以便员工可以简单地查找值并简化添加剂的测试过程。 此程序的主要优点之一是，测试添加剂的工作量大且重复性高，可以将其外包给只有有限数学知识的人员。 对于边缘案例或非常重要的消息，只需征询专家意见。
I hope that I was able to cast some light on Bayesian inference and show how it was used at Bletchley Park to decipher JN 25. If you want to know more about the topic, check out Edward Simpson’s article here [3]. Stay tuned for Part II, where I will dissect the inner workings of the German Enigma and discuss how Bayes’ theorem helped break it.
我希望我能对贝叶斯推断有所启发，并说明如何在布莱奇利公园使用它来解释JN25。如果您想进一步了解该主题，请在此处查看爱德华·辛普森的文章[3]。 请继续关注第二部分，在该部分中，我将剖析German Enigma的内部工作原理，并讨论贝叶斯定理如何帮助打破它。
翻译自: https://towardsdatascience.com/how-bayes-theorem-helped-win-the-second-world-war-7f3be5f4676c
贝叶斯定理
