混合精度训练支持什么显卡
Discover a way to efficiently utilize your GPU
探索有效利用GPU的方法
What is Mixed Precision Training
什么是混合精密训练
Why MPT is Important
为什么MPT很重要
How MPT reduces memory
MPT如何减少内存
Frameworks with AMP (Automatic Mixed Precision)
带有AMP(自动混合精度)的框架
Mixed precision training is a technique used in training a large neural network where the model’s parameter are stored in different datatype precision (FP16 vs FP32 vs FP64). It offers significant performance and computational boost by training large neural networks in lower precision formats. With release of 30X series of GPUs it becomes even more important to utilize these features.
混合精度训练是一种用于训练大型神经网络的技术，其中模型的参数以不同的数据类型精度( FP16与FP32与FP64 )存储。 通过以较低的精度格式训练大型神经网络，它提供了显着的性能和计算能力。 随着30X系列GPU的发布，利用这些功能变得更加重要。
For instance, In Pytorch, the single precision float mean float32 and by default the parameters takes float32 datatype. Now if we have a parameter (W) which could be stored in FP16 while ensuring that no task specific accuracy is affected by this movement between precision, then why should we use FP32 or FP64?
例如，在Pytorch中，单精度float均值float32 ，默认情况下参数采用float32数据类型。 现在，如果我们有一个参数(W)可以存储在FP16中，同时确保精度之间的这种移动不会影响特定于任务的精度，那么为什么要使用FP32或FP64？
Notations
记号
FP16 — Half-Precision, 16bit Floating Point-occupies 2 bytes of memory
FP16 —半精度，16位浮点占用2个字节的内存
FP32 — Single-Precision, 32bit Floating Point-occupies 4 bytes of memory
FP32 —单精度32位浮点占用4个字节的内存
FP64— Double-Precision, 64bit Floating Point-occupies 8 bytes of memory
FP64 —双精度64位浮点占用8个字节的内存
Since the introduction of Tensor Cores in the Volta and Turing architectures (NVIDIA), significant training speedups are experienced by switching to mixed precision — up to 3x overall speedup on the most arithmetically intense model architectures. The ability to train deep learning networks with lower precision was introduced in the Pascal architecture and first supported in CUDA® 8 in the NVIDIA Deep Learning SDK.
自从在Volta和Turing架构(NVIDIA)中引入Tensor Core以来，通过切换到混合精度，可显着提高培训速度-在算术强度最高的模型架构上，总体速度可提高3倍。 Pascal架构引入了以较低的精度训练深度学习网络的能力，并首次在NVIDIA Deep Learning SDK的CUDA®8中得到支持。
Requires less memory, enabling training and deploying large neural networks
需要更少的内存，可以训练和部署大型神经网络
Transfer is faster since the bandwidth required is reduced for transmission of data
传输速度更快，因为减少了数据传输所需的带宽
FP16 requires 2 bytes, as the number of bytes are reduced to capture the same numerical entity, it reduces the amount of memory required to train a model and helps in increasing the batch size for our training. And also the data transfer of type FP16 is faster compared to FP32 and FP64.
FP16需要2个字节，因为减少了字节数以捕获相同的数字实体，所以它减少了训练模型所需的内存量，并有助于增加训练的批处理大小。 与FP32和FP64相比，FP16类型的数据传输也更快。
Smaller Is Better 越小越好For 1 million parameters:
对于一百万个参数：
FP32 — 1000,000 * 4 Bytes — 4 MB
FP32 — 1000,000 * 4字节— 4 MB
FP16 — 1000,000 * 2 Bytes — 2 MB
FP16 — 1000,000 * 2字节— 2 MB
Though its half the amount of memory in FP16, few folks many consider 2MB is not worth the headache for moving to mixed precision, then
尽管其容量只有FP16的一半，但很少有人认为2MB转向混合精度不值得头疼，然后
Consider following situation of using Resnet50, The 50-layer ResNet network has ~26 million weight parameters and computes ~16 million activation’s in the forward pass. If you use a 32-bit floating-point value to store each weight and activation this would give a total storage requirement of 168 MB. By using a lower precision value to store these weights and activation’s we could halve or even quarter this storage requirement, i.e. for 42 million transactions, FP16 requires 84MB.
考虑以下使用Resnet50的情况，该50层ResNet网络具有约2600万个权重参数，并在前向通道中计算约1600万个激活量。 如果使用32位浮点值存储每个权重和激活，则总存储需求为168 MB。 通过使用较低的精度值来存储这些权重和激活值，我们可以将该存储需求减少一半甚至四分之一，即对于4,200万笔交易，FP16需要84MB。
There is clearly significant improvement in memory required for the same number of parameters.
相同数量的参数所需的内存显然有了显着改善。
Iteration Vs Mixed Precision vs Loss scaling 迭代与混合精度对比损耗定标Porting the model to use FP16 wherever possible
尽可能移植模型以使用FP16
Adding loss scaling to preserve small gradient values.
添加损耗定标以保留较小的梯度值。
First point, Porting model to use FP16 is simple, we access the model parameters and move it float16 or half-precision as widely known. It similar to changing the dtype of a variable.
首先，使用FP16移植模型很简单，我们访问模型参数并将其移动为众所周知的float16或Half-precision。 它类似于更改变量的dtype。
Scaling Parameter 标度参数Second point, Adding loss scaling to preserve small gradient values, it refer to scaling the parameter by multiplying alpha value to it, before back propagation and then unscale it by dividing the gradient by alpha before updating the weight. Loss scaling is done to avoid the gradient exploding/vanishing.
第二点，添加损耗定标以保留较小的梯度值，它是指在反向传播之前，通过将参数值乘以参数值来定标参数，然后在更新权重之前通过将梯度除以alpha来取消定标。 进行损耗定标可避免梯度爆炸/消失。
Interestingly, there is adaptive scaling technique is introduced for layer wise update of parameters by alpha.
有趣的是，引入了自适应缩放技术，用于按字母逐层更新参数。
Less time required for inference, execution time can be sensitive to memory or arithmetic bandwidth. Half-precision halves the number of bytes accessed, thus reducing the time spent in memory-limited layers. Nvidia GPUs offer up to 8x more half precision arithmetic throughput when compared to single-precision, thus speeding up math-limited layers.
推理所需的时间更少，执行时间可能对内存或算术带宽敏感。 半精度将访问的字节数减半，从而减少了内存受限层所花费的时间。 与单精度相比，Nvidia GPU的半精度算术吞吐量最高可提高8倍，从而加快了数学受限层的速度。
The term Mixed Precision Training is realized because the training utilizes both the half-precision and single precision representations.
之所以称其为“混合精度训练”，是因为该训练同时利用了半精度和单精度表示形式。
Based on research, certain operations tends to cause over or underflow of a parameter or a variable and some act intact within FP16, ultimately a few lists are generated like AllowList, DenyList and InferList, which basically mentions which ops should take place with FP16 and vice-versa.
根据研究，某些操作往往会导致参数或变量的上溢或下溢，并且某些操作会在FP16中完好无损，最终会生成一些列表，例如AllowList，DenyList和InferList，这些列表基本上提到应该在FP16中进行哪些操作，反之-反之亦然。
AllowList operations are operations that take advantage of GPU Tensor Cores. DenyList operations are operations that may overflow the range of FP16, or require the higher precision of FP32. InferList operations are operations that are safely done in either FP32 or FP16. Typical ops included in each list are:
AllowList操作是利用GPU Tensor Core的操作。 DenyList操作是可能会使FP16的范围溢出或需要FP32的较高精度的操作。 InferList操作是在FP32或FP16中安全完成的操作。 每个列表中包括的典型操作是：
AllowList: Convolutions, Fully-connected layers
AllowList：卷积，全连接层
DenyList: Large reductions, Cross entropy loss, L1 Loss, Exponential
DenyList：大幅减少，交叉熵损失，L1损失，指数
InferList: Element-wise operations (add, multiply by a constant)
InferList：逐元素运算(加，乘以常数)
With recent updates in deep learning frameworks, a technique called Automatic Mixed Precision has been introduced. It helps the developers in performing these casting and scaling operations automatically
随着深度学习框架的最新更新，引入了一种称为自动混合精度的技术。 它可以帮助开发人员自动执行这些转换和缩放操作
Automatic loss scaling and master weights integrated into optimizer classes.
自动损耗定标和主权重已集成到优化器类中。
Automatic casting between float16 and float32 to maximize speed while ensuring no loss in task-specific accuracy.
在float16和float32之间自动转换以最大化速度，同时确保不损失特定于任务的精度。
In those frameworks with automatic support, using mixed precision can be as simple as adding one line of code or enabling a single environment variable. Currently, the frameworks with support for automatic mixed precision are TensorFlow, PyTorch, and MXNet.
在那些具有自动支持的框架中，使用混合精度可以像添加一行代码或启用单个环境变量一样简单。 当前，支持自动混合精度的框架是TensorFlow，PyTorch和MXNet 。
Interesting point, Nvidia’s tensor cores are designed in such fashion that keeping dimension of matrix as multiple of 8 helps in faster calculation. Do read the NVIDIA’s Mixed Precision article to understand,
有趣的是，Nvidia的张量核心的设计方式是将矩阵的维数保持为8的倍数有助于更快地计算。 请阅读NVIDIA的混合精度文章以了解，
how to set the dimension in CNN.
如何在CNN中设置尺寸。
how to choose mini-batch size.
如何选择小批量。
how to choose linear layer.
如何选择线性层。
how to pad the vocabulary in sequence based model
如何在基于序列的模型中填充词汇
Details related to How to set the alpha value, How to scale & unscale a parameter, how the performance is affected if poor scaling factor is chosen, how to match the performance of FP32 using FP16, check out the reference link to NVIDIA Mixed Precision.
有关如何设置alpha值，如何缩放和不缩放参数，如果选择较差的缩放系数，如何影响性能，如何使用FP16匹配FP32的性能等详细信息，请查看NVIDIA Mixed Precision的参考链接。
I have tried AMP in Pytorch, the neural network ran only for one epoch, there was a time difference of more than 1 minute in training the network and with moderate affect on loss, it was done without the loss scaling though.
我曾在Pytorch中尝试过AMP，神经网络只运行了一个纪元，训练网络时差超过1分钟，并且对损失的影响适中，尽管这样做没有损失缩放。
Without AMP vs AMP 没有AMP与AMP翻译自: https://medium.com/analytics-vidhya/mixed-precision-training-fd08f4c8e72d
混合精度训练支持什么显卡
相关资源：混合精度训练官方PPT.zip