aws最佳实践

TL; DR; (TL;DR;)

I created a repo to deploy Airflow on AWS following software engineering best practices. You can go straight there if you don’t feel like reading this post. But I do describe some things you might find useful here.

我按照软件工程最佳实践创建了一个可在AWS上部署Airflow的存储库 。 如果您不想阅读这篇文章，可以直接去那里 。 但是我确实描述了一些您可能会在这里发现有用的东西。

Run a docker-compose command and voíla, you have Airflow running on your local environment, and you are ready to develop some DAGs. After some time you have your DAGs (and Airflow) prepared for deployment on a production environment. Then you start searching for instructions on how to deploy Airflow on AWS. Here’s what you’ll probably find:

运行docker-compose命令并执行voíla，您在本地环境中运行了Airflow，并准备开发一些DAG。 一段时间后，您已准备好将DAG(和Airflow)部署到生产环境中。 然后，您开始搜索有关如何在AWS上部署Airflow的说明。 您可能会发现以下内容：

No instructions on Airflow documentation.

没有关于气流文档的说明。

Some posts, like this one, teach you how to deploy on AWS ECS. Quite an interesting approach. The problem is that the whole tutorial is based on creating resources by point-and-click on the AWS console. Trust me; you don’t want to go that route for deploying in production. Just imagine the nightmare for creating three different environments (dev, staging and production) and having to repeat the process three times. Now imagine updating the environments and keeping them in sync. Picture how you could easily spend a whole week fixing a bug caused by a resource that was deleted by mistake.

诸如此类的一些文章，教您如何在AWS ECS上进行部署。 非常有趣的方法。 问题在于，整个教程都基于在AWS控制台上单击鼠标并创建资源的基础。 相信我; 您不想走这种路线在生产中进行部署。 试想一下创建三个不同环境(开发，暂存和生产)并不得不重复该过程三遍的噩梦。 现在想象一下更新环境并使它们保持同步。 想象一下如何轻松地花费整整一周的时间来修复由错误删除的资源引起的错误。

This other post is relatively recent and has a superior approach, but it still creates resources (like the ECS cluster) using AWS CLI. A little better than console point-and-click, but still not production-ready.

另一篇文章是相对较新的，并且具有更好的方法，但是它仍然使用AWS CLI创建资源(如ECS集群)。 比控制台点击好一点，但仍未准备就绪。

Other articles, might mention using Infrastructure as Code, which would solve the problems mentioned above. However, they are very shallow in technical details and implementation. So, despite offering a good overview and best practices, they are not practical for someone without DevOps experience.

其他文章可能提到使用基础结构作为代码，这将解决上述问题。 但是，它们在技术细节和实施方面非常肤浅。 因此，尽管提供了很好的概述和最佳实践，但对于没有DevOps经验的人来说，它们并不实用。

This repo on GitHub is probably the closest you’ll get from a proper implementation of Airflow on AWS following software engineering best practices. But it still lacks some basic stuff like autoscaling of webservers and workers or a way to configure settings such as RDS instance type without having to dig through Terraform code.

遵循软件工程最佳实践， 此GitHub仓库可能是您从AWS上正确实施Airflow可获得的最接近的仓库。 但是它仍然缺少一些基本的东西，例如Web服务器和工作程序的自动缩放，或者无需浏览Terraform代码即可配置RDS实例类型等设置的方法。

You might have a look on page 2 of your Google search, but if a good result doesn’t show up on the first page, then it probably means that it doesn’t exist.

您可能会在Google搜索的第2页上看到外观，但是如果首页上未显示好的结果，则可能意味着该结果不存在。

入门 (Getting started)

Just clone the repo, follow the few instructions and install the requirements described on the README file and then run:

只需克隆存储库 ，按照一些说明进行操作，并安装README文件中描述的要求，然后运行：

make airflow-deploy

This will deploy an Airflow instance to your AWS account!

这会将Airflow实例部署到您的AWS账户！

Done! You’ll have Airflow deployed on AWS ECS with autoscaling enabled. Image by the Author, print screen of the Airflow user interface. 做完了！ 您将在启用自动缩放的情况下在AWS ECS上部署Airflow。 图片由作者提供，Airflow用户界面的打印屏幕。

它如何帮助您遵循最佳做法？ (How does it help you follow best practices?)

一个service.yml文件 (A service.yml file)

This repo implements all infrastructure using AWS Cloudformation. Inside the /cloudformation directory you’ll find all templates to create the infrastructure needed to run Airflow. The good thing is you don’t need to worry about learning Cloudformation to do a simple deploy because in the root there is a service.yml to help you.

此仓库使用AWS Cloudformation实施所有基础架构。 在/ cloudformation内部 您将在目录中找到所有模板，以创建运行Airflow所需的基础结构。 好处是您无需担心学习Cloudformation即可进行简单部署，因为在根目录中有一个service.yml。 来帮你。

Let’s say you want to whitelist specific IPs (such as your office IP) to have access to Airflow UI. The only thing you need to do is change a few lines in service.yml:

假设您想将特定IP(例如您的办公室IP)列入白名单以访问Airflow UI。 您唯一需要做的就是更改service.yml中的几行：

whitelistedIPs: - 123.456.789.123/32 - 987.654.321.987/32

If you want to change the Airflow database instance type, you also go to service.yml:

如果要更改Airflow数据库实例类型，还可以转到service.yml ：

metadataDb: instanceType: db.t3.micro port: 5432 dbName: airflow engine: postgres engineVersion: 11.7 family: postgres11 deletionProtection: false enableIAMDatabaseAuthentication: true allocatedStorage: 20 parameters: maxConnections: 100

You want to just the Airflow workers CPU and memory? Fine-tune autoscaling? service.yml is the one-stop-shop.

您只想要Airflow Worker的CPU和内存吗？ 微调自动缩放？ service.yml是一站式商店。

service: logGroupName: airflow/ecs/fargate cidrBlock: 10.0.0.0 baseUrl: http://localhost:8080 port: 80 workers: port: 8793 cpu: 1024 memory: 2048 desiredCount: 2 autoscaling: maxCapacity: 8 minCapacity: 2 cpu: target: 70 scaleInCooldown: 60 scaleOutCooldown: 120 memory: target: 70 scaleInCooldown: 60 scaleOutCooldown: 120

在AWS Secrets Manager中管理的密码 (Passwords Managed within AWS Secrets Manager)

Don’t hardcode your passwords is Software Engineering 101. But sometimes it is hard to automate deployments and create passwords at runtime without hardcoding then somewhere. In this Airflow repo, we use AWS Secrets Manager to help us solve that.

不要对您的密码进行硬编码是Software Engineering101。但是有时候，在没有进行硬编码的情况下，很难自动执行部署并在运行时创建密码。 在此Airflow存储库中，我们使用AWS Secrets Manager来解决问题。

Airflow Metadata DB

气流元数据数据库

Our Postgres database that will hold Airflow metadata is one of the resources that require an admin username and password. Using AWS Secrets Manager, a strong random password is created at deploy time and attached to the cluster. To get the password value, you have to log in to your AWS account and go to Secrets Manager.

我们将存储Airflow元数据的Postgres数据库是需要管理员用户名和密码的资源之一。 使用AWS Secrets Manager，将在部署时创建一个强随机密码并将其附加到集群。 要获取密码值，您必须登录到您的AWS账户并转到Secrets Manager。

MetadataDBSecret: Type: AWS::SecretsManager::Secret Properties: Description: 'This is Airflow Metadata DB secret' Name: "{{ serviceName }}-{{ ENVIRONMENT }}-metadata-db-credentials" GenerateSecretString: SecretStringTemplate: '{"username": "airflow_admin"}' GenerateStringKey: 'password' PasswordLength: 32 ExcludePunctuation: true KmsKeyId: !ImportValue encryption-KMSKeyId Tags: - Key: Name Value: "{{ serviceName }}-{{ ENVIRONMENT }}-metadata-db-secret"

It is also possible to implement automatic password rotation with Secrets Manager, but it was not implemented for this project.

也可以使用Secrets Manager实施自动密码轮换，但该项目尚未实现。

Fernet Key

铁网钥匙

Airflow uses a Fernet Key to encrypt passwords (such as connection credentials) saved to the Metadata DB. This deployment generates a random Fernet Key at deployment time and adds it to Secrets Manager. It is then referenced in the Airflow containers as an environment variable.

Airflow使用Fernet密钥对保存到元数据数据库中的密码(例如连接凭据)进行加密。 此部署在部署时会生成一个随机的Fernet密钥，并将其添加到Secrets Manager中。 然后在“气流”容器中将其作为环境变量引用。

自动缩放启用 (Autoscaling Enabled)

One of the biggest challenges of putting Airflow in production is dealing with resources management. How to avoid crashing the webserver if there is a usage peak? Or what to do if a particular daily job requires more CPU/memory?

将Airflow投入生产的最大挑战之一是处理资源管理。 如果出现使用高峰，如何避免使Web服务器崩溃？ 或者，如果一项特定的日常工作需要更多的CPU /内存，该怎么办？

Autoscaling solves those issues for you. In this repository, you can easily configure thresholds and rest assured that your infrastructure will scale up and down to meet demand.

自动缩放可为您解决这些问题。 在此存储库中，您可以轻松配置阈值，并放心您的基础架构将根据需求扩展和缩小。

易于部署到不同的环境 (Easy Deploy to Different Environments)

In a production setup, you will want to deploy your code to different environments. Let’s say you’ll need: prod, stage and dev.

在生产设置中，您将需要将代码部署到不同的环境。 假设您需要：生产，开发和开发。

This repo allows you to deploy the same code to different environments by just changing one environment variable, that could be automatically inferred on you CI/CD pipeline. To change the environment, do:

此回购允许您仅更改一个环境变量即可将相同的代码部署到不同的环境，该变量可以在您的CI / CD管道上自动推断。 要更改环境，请执行以下操作：

export ENVIRONMENT=dev; # this will deploy airflow to dev environmentmake airflow-deploy;

在不涉及基础架构的情况下更新DAG (Update DAGs Without Touching on Infrastructure)

The beauty of Airflow is the ability to write workflows as code. It means that you will change DAGs code much more often than you change infrastructure. With this deployment of Airflow, you will submit changes to your DAGs, and it won’t try to redeploy the infrastructure for you.

Airflow的优点在于可以将工作流程编写为代码。 这意味着您更改DAG代码的频率将比更改基础架构的频率高得多。 通过部署Airflow，您将向DAG提交更改，并且不会尝试为您重新部署基础结构。

The only thing you want to do is build a new Airflow image, push it to ECR and then update your ECS service to load the latest image. To achieve that, just run:

您唯一要做的是构建一个新的Airflow映像，将其推送到ECR，然后更新您的ECS服务以加载最新的映像。 为此，只需运行：

make airflow-push-image;

It is not the case here, but you could even have your DAGs sitting on a separate repository. It would separate infrastructure from software even more.

这里不是这种情况，但是您甚至可以将DAG放在单独的存储库中。 它将使基础架构与软件进一步分离。

标记您的资源 (Tag your Resources)

Tagging resources will allow us to create automated alerts, identify ownership and track infrastructure costs easily. That’s why this Airflow repository tags all resources.

标记资源将使我们能够创建自动警报，识别所有权并轻松跟踪基础架构成本。 这就是为什么此Airflow存储库标记所有资源的原因。

Tags: - Key: Name Value: "{{ serviceName }}-{{ ENVIRONMENT }}-ecs-cluster"

你该怎么办？ (What should you do on your side?)

You should add the deployment process to your CI/CD pipeline. To run some automated tests, I’m using GitHub Actions (but your company might be using other tools such as CircleCI or Jenkins).

您应该将部署过程添加到CI / CD管道中。 为了运行一些自动化测试，我使用GitHub Actions(但是您的公司可能正在使用CircleCI或Jenkins等其他工具)。

You can follow a similar process to automate your deploy and tests. Have a look at the tests workflow to get some inspiration.

您可以遵循类似的过程来自动化部署和测试。 查看测试工作流程以获取一些启发。

它要多少钱？ (How much does it cost?)

Running Airflow on AWS with the default configurations (also considering cluster is not scaling up) should cost from 5 to 7 US dollars per day. Depending on the region you are deploying.

使用默认配置在AWS上运行Airflow(还要考虑群集未扩展)的成本每天为5至7美元 。 根据您要部署的区域。

This cost can be further reduced by lowering CPU and memory on service.yml and also changing the minimum number of workers. The default settings would allow Airflow to run quite a few dags before needing to increase resources. Fine-tune and find what works best for your use case.

可以通过降低service.yml上的CPU和内存并更改最小工作线程数来进一步降低此成本。 默认设置将允许Airflow在需要增加资源之前运行很多dag。 微调并找到最适合您的用例的内容。

I hope this post (and the repository) helps you to easily productionize Airflow. If you have any questions, suggestions or requests, please reach out to me on LinkedIn, or open an issue on the repo. Also, you are welcome to open PRs and collaborate with the project!

我希望这篇文章( 和资料库 )能帮助您轻松生产Airflow。 如果您有任何疑问，建议或要求，请在LinkedIn上与我联系，或在仓库中打开问题。 此外，欢迎您打开PR，并与该项目合作！

翻译自: https://towardsdatascience.com/how-to-deploy-airflow-on-aws-best-practices-63778d6eab9e

aws最佳实践

相关资源：微信小程序源码-合集6.rar