生成对抗网络(Generative Adversarial Network)

Generative Adversarial Network (GAN) is a generative model that is able to generate new content. The topic has become really popular in the machine learning community due to its interesting applications such as generating synthetic training data, creating arts, style-transfer, image-to-image translation, etc.

生成对抗网络(GAN)是一种能够生成新内容的生成模型。 由于其有趣的应用程序，例如生成综合训练数据，创建艺术，样式转换，图像到图像的翻译等，该主题在机器学习社区中变得非常流行。

GAN Architecture [Image by Author] GAN Architecture [作者提供的图片]

GAN consisted of 2 networks, the generator, and the discriminator. The generator will try to generate fake samples and fool the discriminator into believing it to be real samples. The discriminator will try to detect the generated samples from both the real and fake samples. This interesting adversarial concept was introduced by Ian Goodfellow in 2014. There are already a lot of resources available to learn GAN, hence I will not explain GAN to avoid redundancy.

GAN由2个网络，生成器和鉴别器组成。 生成器将尝试生成假样本，并欺骗鉴别器以为它是真实样本。 鉴别器将尝试从真实样本和假样本中检测生成的样本。 这个有趣的对抗性概念是由Ian Goodfellow在2014年提出的。已经有很多学习GAN的资源，因此，为了避免重复，我不会解释GAN。

I recommend reading this beautiful article by Joseph Rocca for understanding GAN.

我建议阅读Joseph Rocca的这篇漂亮文章，以了解GAN。

样式GAN2 (StyleGAN2)

The StyleGAN paper, “A Style-Based Architecture for GANs”, was published by NVIDIA in 2018. The paper proposed a new generator architecture for GAN that allows them to control different levels of details of the generated samples from the coarse details (eg. head shape) to the finer details (eg. eye-color).

NVIDIA在2018年发布了StyleGAN论文“基于GAN的基于样式的体系结构”。该论文提出了GAN的新生成器体系结构，该体系结构使他们可以从粗略细节中控制生成样本的不同细节级别(例如头部形状)到更精细的细节(例如，眼睛颜色)。

StyleGAN also incorporates the idea from Progressive GAN, where the networks are trained on lower resolution initially (4x4), then bigger layers are gradually added after it’s stabilized. By doing this, the training time becomes a lot faster and the training is a lot more stable.

StyleGAN还结合了Progressive GAN的想法，该网络最初以较低的分辨率(4x4)进行网络训练，然后在稳定后逐渐添加较大的层。 这样，培训时间变得更快，并且培训也更加稳定。

[Source: Sarah Wolf] [资料来源：莎拉·沃尔夫(Sarah Wolf)]

StyleGAN improves it further by adding a mapping network that encodes the input vectors into an intermediate latent space, w, which then will have separate values be used to control the different levels of details.

StyleGAN通过添加一个将输入矢量编码到中间潜在空间w的映射网络来进一步改善它，然后将使用单独的值来控制细节的不同级别。

StyleGAN Generator Architecture [Image by Author] StyleGAN生成器体系结构[作者提供的图像]

Why add a mapping network? One of the issues of GAN is its entangled latent representations (the input vectors, z). For example, let’s say we have 2 dimensions latent code which represents the size of the face and the size of the eyes. In this case, the size of the face is highly entangled with the size of the eyes (bigger eyes would mean bigger face as well). On the other hand, we can simplify this by storing the ratio of the face and the eyes instead which would make our model be simpler as unentangled representations are easier for the model to interpret.

为什么要添加地图网络？ GAN的问题之一是纠缠的潜在表示(输入向量z )。 例如，假设我们有2个维度的潜在代码，分别代表脸的大小和眼睛的大小。 在这种情况下，脸的大小与眼睛的大小高度纠缠在一起(大眼睛也意味着更大的脸)。 另一方面，我们可以通过存储脸部和眼睛的比例来简化此过程，这将使我们的模型更加简单，因为无纠缠的表示更易于模型解释。

With entangled representations, the data distribution may not necessarily follow the normal distribution where we want to sample the input vectors z from. For example, the data distribution would have a missing corner like this which represents the region where the ratio of the eyes and the face becomes unrealistic.

使用纠缠表示，数据分布可能不一定遵循我们要从中采样输入向量z的正态分布。 例如，数据分布将具有这样的缺失角，该缺失角表示眼睛和面部的比例变得不现实的区域。

[Source: Paper] [资料来源：论文]

If we sample the z from the normal distribution, our model will try to also generate the missing region where the ratio is unrealistic and because there Is no training data that have this trait, the generator will generate the image poorly. Therefore, the mapping network aims to disentangle the latent representations and warps the latent space so it is able to be sampled from the normal distribution.

如果我们从正态分布中采样z，我们的模型将尝试生成比率不切实际的缺失区域，并且由于没有训练数据具有该特征，因此生成器将生成较差的图像。 因此，映射网络旨在解散潜在表示并扭曲潜在空间，以便能够从正态分布中进行采样。

[Source: Paper] [资料来源：论文]

Additionally, Having separate input vectors, w, on each level allows the generator to control the different levels of visual features. The first few layers (4x4, 8x8) will control a higher level (coarser) of details such as the head shape, pose, and hairstyle. The last few layers (512x512, 1024x1024) will control the finer level of details such as the hair and eye color.

另外，在每个级别上具有单独的输入向量w允许生成器控制视觉特征的不同级别。 前几层(4x4、8x8)将控制更高级别(更粗略)的细节，例如头部形状，姿势和发型。 最后几层(512x512、1024x1024)将控制细节级别，例如头发和眼睛的颜色。

[Source: Paper] [来源：论文] [Source: Paper] [来源：纸张]

For full details on StyleGAN architecture, I recommend you to read NVIDIA's official paper on their implementation. Here is the illustration of the full architecture from the paper itself.

有关StyleGAN架构的完整详细信息，我建议您阅读NVIDIA关于其实现的官方论文。 这是论文本身对完整体系结构的说明。

[Source: A Style-Based Architecture for GANs Paper] [来源：GANs的基于样式的体系结构]

随机变化 (Stochastic Variation)

StyleGAN also allows you to control the stochastic variation in different levels of details by giving noise at the respective layer. Stochastic variations are minor randomness on the image that does not change our perception or the identity of the image such as differently combed hair, different hair placement and etc. You can see the effect of variations in the animated images below.

StyleGAN还允许您通过在相应层上发出噪声来控制不同细节级别的随机变化。 随机变化是图像上的微小随机性，不会改变我们的感知或图像身份，例如梳理不同的头发，放置不同的头发等。您可以在下面的动画图像中看到变化的影响。

[Source: Paper] [来源：论文] [Source: Paper] [来源：论文]

StyleGAN also made several other improvements that I will not cover in these articles such as the AdaIN normalization and other regularization. You can read the official paper, this article by Jonathan Hui, or this article by Rani Horev for further details instead.

StyleGAN还进行了其他一些我在这些文章中不会涉及的改进，例如AdaIN标准化和其他正则化。 您可以阅读正式文章，乔纳森·许(Jonathan Hui)的文章或拉尼·霍列夫(Rani Horev)的本文，以获取更多详细信息。

截断技巧 (Truncation Trick)

When there is an underrepresented data in the training samples, the generator may not be able to learn the sample and generate it poorly. To avoid this, StyleGAN uses a “truncation trick” by truncating the intermediate latent vector w forcing it to be close to average.

当训练样本中的数据不足时，生成器可能无法学习样本并生成不良样本。 为了避免这种情况，StyleGAN通过截断中间潜矢量w使其接近平均水平来使用“截断技巧”。

The 𝚿 (psi) is the threshold that is used to truncate and resample the latent vectors that are above the threshold. Hence, with higher 𝚿, you can get higher diversity on the generated images but it also has a higher chance of generating weird or broken faces. For this network 𝚿 value of 0.5 to 0.7 seems to give a good image with adequate diversity according to Gwern. Though, feel free to experiment with the threshold value.

psi(psi)是用于截断和重新采样高于阈值的潜矢量的阈值。 因此，使用较高的higher，可以在生成的图像上获得更高的多样性，但也更有可能生成怪异或折断的面Kong。 对于这个网络，根据Gwern的描述，0.5值为0.5到0.7似乎可以提供具有足够多样性的良好图像。 不过，请随意尝试该阈值。

3x3 Grid Images generated with 0.3 psi (left) vs 0.7 psi (middle) vs 1.3 psi (right) 以0.3 psi(左)与0.7 psi(中)和1.3 psi(右)生成的3x3网格图像

生成动漫人物(Generating Anime Characters)

I will be using the pre-trained Anime StyleGAN2 by Aaron Gokaslan so that we can load the model straight away and generate the anime faces. So, open your Jupyter notebook or Google Colab, and let’s start coding.

我将使用Aaron Gokaslan预先训练好的Anime Style2，以便我们可以立即加载模型并生成动漫面Kong。 因此，打开Jupyter笔记本或Google Colab，让我们开始编码。

Note: You can refer to my Colab notebook if you are stuck

注意：如果卡住了，可以参考我的Colab笔记本

So first of all, we should clone the styleGAN repo.

因此，首先，我们应该克隆styleGAN存储库。

$ git clone https://github.com/NVlabs/stylegan2.git

If you are using Google Colab, you can prefix the command with ‘!’ to run it as a command: !git clone https://github.com/NVlabs/stylegan2.git

如果您使用的是Google Colab，则可以在命令前加上“！” 将其作为命令运行：！git clone https://github.com/NVlabs/stylegan2.git

Next, we would need to download the pre-trained weights and load the model. Make sure you are running with GPU runtime when you are using Google Colab as the model is configured to use GPU.

接下来，我们将需要下载预训练的权重并加载模型。 当您将Google Colab配置为使用GPU时，请确保在GPU运行时下运行。

this notebook 此笔记本中的代码已修改

Now, we need to generate random vectors, z, to be used as the input fo our generator. Let’s create a function to generate the latent code, z, from a given seed.

现在，我们需要生成随机向量z，以用作生成器的输入。 让我们创建一个函数，以从给定的种子生成潜在代码z。

Then, we can create a function that takes the generated random vectors z and generate the images.

然后，我们可以创建一个函数，该函数采用生成的随机向量z并生成图像。

Now, we can try generating a few images and see the results.

现在，我们可以尝试生成一些图像并查看结果。

The function will return an array of PIL.Image. In Google Colab, you can straight away show the image by printing the variable. Here is the first generated image.

该函数将返回一个PIL.Image数组。 在Google Colab中，您可以通过打印变量直接显示图像。 这是第一个生成的图像。

Image by Author 图片作者

Let’s show it in a grid of images, so we can see multiple images at one time.

让我们将其显示在图像网格中，以便一次可以看到多个图像。

And then we can show the generated images in a 3x3 grid.

然后我们可以在3x3的网格中显示生成的图像。

Image by Author 图片作者

One of the nice things about GAN is that GAN has a smooth and continuous latent space unlike VAE (Variational Auto Encoder) where it has gaps. Hence, when you take two points in the latent space which will generate two different faces, you can create a transition or interpolation of the two faces by taking a linear path between the two points.

GAN的优点之一是，GAN具有不同于VAE(可变自动编码器)的平滑且连续的潜伏空间，该空间存在间隙。 因此，当您在潜在空间中获取两个点将生成两个不同的面时，可以通过在两个点之间采用线性路径来创建两个面的过渡或插值。

[Source: Joseph Rocca] [来源：Joseph Rocca]

Let’s implement this in code and create a function to interpolate between two values of the z vectors.

让我们在代码中实现它，并创建一个函数在z向量的两个值之间进行插值。

Let’s see the interpolation results. You can see that the first image gradually transitioned to the second image.

让我们看一下插值结果。 您可以看到第一张图像逐渐过渡到第二张图像。

Image by Author 图片作者

Now that we’ve done interpolation. We can finally try to make the interpolation animation in the thumbnail above. We will use the moviepy library to create the video or GIF file.

现在，我们已经完成了插值。 我们最终可以尝试在上面的缩略图中制作插值动画。 我们将使用moviepy库创建视频或GIF文件。

When you run the code, it will generate a GIF animation of the interpolation. You can also modify the duration, grid size, or the fps using the variables at the top.

运行代码时，它将生成插值的GIF动画。 您也可以使用顶部的变量修改持续时间，网格大小或fps。

Generated StyleGAN2 Interpolation GIF [Image by Author] 生成的StyleGAN2插值GIF [作者提供的图像]

If you made it this far, congratulations! You have generated anime faces using StyleGAN2 and learned the basics of GAN and StyleGAN architecture.

如果您做到了这一点，那么恭喜！ 您已经使用StyleGAN2生成了动漫面Kong，并学习了GAN和StyleGAN体系结构的基础知识。

下一步是什么？ (What’s Next?)

Now that we have finished, what else can you do and further improve on? Here are a few things that you can do.

现在我们已经完成了，您还能做什么并进一步改进？ 这是您可以做的几件事。

Other DatasetsObviously, StyleGAN is not limited to anime dataset only, there are many available pre-trained datasets that you can play around such as images of real faces, cats, art, and paintings. Check out this GitHub repo for available pre-trained weights. On the other hand, you can also train the StyleGAN with your own chosen dataset.

其他数据集显然，StyleGAN不仅限于动漫数据集，还可以使用许多可用的经过预先训练的数据集，例如真实面Kong，猫，艺术品和绘画的图像。 查看此GitHub存储库以获取可用的预训练权重。 另一方面，您也可以使用自己选择的数据集训练StyleGAN。

Conditional GANCurrently, we cannot really control the features that we want to generate such as hair color, eye color, hairstyle, and accessories. Conditional GAN allows you to give a label alongside the input vector, z, and hence conditioning the generated image to what we want. Alternatively, you can try making sense of the latent space either by regression or manually. If you want to go to this direction, Snow Halcy repo maybe be able to help you, as he done it and even made it interactive in this Jupyter notebook.

有条件的GAN 当前，我们无法真正控制想要生成的功能，例如头发颜色，眼睛颜色，发型和配饰。 有条件的GAN允许您在输入向量z旁边加上标签，从而将生成的图像调整为我们想要的图像。 另外，您可以尝试通过回归或手动了解潜在空间。 如果您想朝这个方向发展， Snow Halcy仓库可能会为您提供帮助，因为他做到了，甚至使它在此J upyter笔记本中具有交互性。

致谢 (Acknowledgment)

I’d like to thanks Gwern Branwen for his extensive articles and explanation on generating anime faces with StyleGAN which I strongly referred to in my article. I fully recommend you to visit his websites as his writings are a trove of knowledge. Additionally, check out ThisWaifuDoesNotExists website which hosts the StyleGAN model for generating anime faces and a GPT model to generate anime plot.

我要感谢Gwern Branwen 他在我的文章中强烈提到的大量文章以及有关使用StyleGAN生成动漫面Kong的说明。 我完全建议您访问他的网站，因为他的著作非常丰富。 此外，请访问ThisWaifuDoesNotExists网站，该网站托管用于生成动漫面Kong的StyleGAN模型和用于生成动漫情节的GPT模型。

If you enjoy my writing, feel free to check out my other articles!

如果您喜欢我的文章，请随时查看我的其他文章！

翻译自: https://towardsdatascience.com/generating-anime-characters-with-stylegan2-6f8ae59e237b

相关资源：Python-GAN实战对抗生成网络深度学习随书代码