Over the last few weeks, I’ve been working on improving the overall performance of our SaaS platform (OnCare) for user experience gains and profit. Hopefully, some of my fresh musings could be useful to you.
在过去的几周中，我一直在努力改善SaaS平台( OnCare )的整体性能，以期获得用户体验并获得收益。 希望我的一些新想法对您有用。
TLDR; Focus on building and marketing features, then fix performance issues if and when they emerge.
TLDR； 着重于构建和营销功能，然后在出现性能问题时予以解决。
We’ve been running the business lean, taking in requirements, building solutions from first principles and iterating on feedback from customers once it’s been shipped. It’s a tricky balance spinning all the technical plates while growing your business, one of which is performance. Performance is something we focus on sparingly during our early product lifecycles as premature optimisation is a killer if you build something people don’t end up wanting or using.
我们一直在精简业务，接受需求，从第一条原则构建解决方案，并在发货后根据客户的反馈进行迭代。 在发展业务的同时，平衡所有技术板块是一个棘手的平衡，其中之一就是性能。 在产品的早期生命周期中，我们很少关注性能，因为如果您构建了人们最终不想要或不使用的产品，那么过早的优化将成为杀手。
Given that performance can take a back seat, tech debt will need to be paid off from time to time, here’s how I tackle it.
鉴于性能可能会退居次要位置，因此需要不时偿还技术债务，这就是我要解决的方法。
TLDR; find a cloud provider where you can profile live traffic. Typically known as application performance management (APM).
TLDR； 查找可以在其中描述实时流量的云提供商。 通常称为应用程序性能管理(APM)。
Our infrastructure provider AWS provides a useful live profiling tool called Xray that’s been fairly core to learning what the bottlenecks you have and to some extent why they are a problem.
我们的基础架构提供商AWS提供了一个有用的实时分析工具Xray ，该工具对于了解您所遇到的瓶颈以及在某种程度上导致问题的原因非常重要。
Minutiae on Xray is that it’ll only profile a fraction of your traffic so occasionally I’ll increase samples for certain endpoints I’ve earmarked for improvements to ensure I’m looking at statistically significant traces.
Xray上的细节之处在于，它只会分析您流量的一小部分，因此我偶尔会增加一些我已指定用于改进的端点的样本，以确保我查看的是统计上有意义的痕迹。
Once I’ve got a handle on slow requests, it’s time to dig deeper and look at the waterfall charts in individual traces. There tends to be a theme at this stage; N+1 inefficiencies or slow SQL queries (complex joins) all originating at the DB level. Here’s an example:
一旦我处理了缓慢的请求，就该深入挖掘并查看各个轨迹中的瀑布图了。 在这个阶段往往有一个主题。 N + 1个效率低下或SQL查询(复杂联接)缓慢的情况都起源于数据库级别。 这是一个例子：
Beyond AWS I’ve found DataDog and New Relic to be useful services with their APM offerings but I favour the convenience, cost and consolidation AWS provides.
除AWS之外，我发现DataDog和New Relic在其APM产品中是有用的服务，但我更喜欢AWS提供的便利性，成本和整合。
Locally Django Silk has been of some use but given how much of a slowdown it incurs on response times, it’s seldom used. It’s also not representative of users problems as our dev environments are highly unoptimised and have comparably little data to work off.
在本地Django Silk已有一定用处，但考虑到它会在响应时间上造成很大的影响，因此很少使用。 它也不能代表用户的问题，因为我们的开发环境高度未优化，并且工作所需的数据相对较少。
TLDR; optimise queries by taking advantage of Django’s slightly hidden ORM helpers (prefetch_related, _id & annotate).
TLDR； 通过利用Django稍微隐藏的ORM帮助器( prefetch_related, _id和annotate )优化查询。
I’ll tend to chip away at N+1 queries first as they can often be resolved with a prefetch_related or select_related on the queryset.
我倾向于先解决N + 1个查询，因为通常可以使用查询集上的prefetch_related或select_related来解决它们。
Word of caution here; overzealously “optimising” querysets with prefetch_related and select_related can lead to worse results as you may overfetch and have inefficient memory allocation.
谨慎的话虚假的热情“优化”查询集与prefetch_related和select_related可以导致更糟糕的结果，因为你可能overfetch和有效率低下的内存分配。
Fixing inefficient queries
修复效率低下的查询
This is a big topic that probably deserves a book by an expert in this field but there’s a few simple things you can do to get by on;
这是一个重要的主题，可能值得该领域的专家撰写一本书，但是您可以做一些简单的事情来继续前进。
Use _id over .pk on your foreign key model lookups. Idiomatic Django would suggest using code like this to get related IDs:
在外键模型查询上使用_id .pk 。 惯用的Django建议使用类似这样的代码来获取相关ID：
reports = Report.objects.all()# Get all reports Creator IDs[report.creator.pk for report in reports]Django will be lazy here and assume you want to get a creator instance so will iterate over each report and do a separate lookup of each creator to just pluck out its PK.
Django在这里会很懒惰，并假设您想获得一个创建者实例，因此将遍历每个报告并对每个创建者进行单独查找以仅抽取其PK。
Instead, just use the somewhat hidden _id convention:
相反，只需使用有些隐藏的_id约定：
reports = Report.objects.all()# Get all reports Creator IDs[report.creator_id for report in reports]Taking this foreign key example one step further and let’s say you wanted to get a list of all the related creator email addresses you could start with this:
再以这个外键示例为例，假设您想获得所有相关创建者电子邮件地址的列表，可以从此开始：
reports = Report.objects.all()# Get all reports Creator emails[report.creator.email for report in reports]This, of course, will lead to the same N+1 problem as above but now we can’t simply use the _id trick as we don’t want the ID. Instead, bring in the select_related method:
当然，这将导致与上述相同的N + 1问题，但是现在我们不能简单地使用_id技巧，因为我们不需要ID。 相反，请引入select_related方法：
reports = Report.objects.select_related('creator').all()# Get all reports Creator emails without N+1 lookups[report.creator.email for report in reports]This will be much faster if you’re covering a big dataset. You can take this one step further, though and be explicit on which related fields you’re pulling through:
如果要覆盖大型数据集，这将更快。 不过，您可以进一步迈出这一步，并明确指出要涉及的相关领域：
reports = Report.objects.select_related( 'creator').only 'creator__email')# Get all reports Creator emails without N+1 lookups[report.Or
要么
reports = Report.objects creator_email=F('creator__email'))[report.TLDR; GraphQL will create inefficient queries, use a library to help improve it.
TLDR； GraphQL将创建效率低下的查询，并使用库来改进它。
We’ve embraced the benefits of GraphQL to simplify our interfaces when getting and mutating data but it’s come with a cost; maintaining efficient queries.
我们已经接受了GraphQL的好处，以简化获取和修改数据时的界面，但是这是有代价的。 维持有效的查询。
Using Graphene, one of the most popular Python GraphQL frameworks it won’t take you too long until you hit N+1 issues (e.g “Is there `n+1` queries issue?”). This is inherent because Graphene will cleverly traverse your models and lazily fetch related M2M and foreign keys through Django’s models but provides no mechanism out the box to annotate what fields could be optimised.
使用Graphene (最流行的Python GraphQL框架之一)，它不会花很长时间，直到您遇到N + 1个问题(例如，“是否存在n + 1个查询问题？ ”)。 这是固有的，因为Graphene将巧妙地遍历您的模型并通过Django的模型懒惰地获取相关的M2M和外键，但没有提供开箱即用的机制来注释可以优化哪些字段。
I found there was a good library available to help with most of the Graphene performance issues; graphene-django-optimizer. The main benefits we’ve taken from this library are the resolver hints to have more explicit control over optimisations. E.g:
我发现有一个很好的库可用来解决大多数Graphene性能问题。 graphene-django-optimizer 。 我们从该库中获得的主要好处是，解析器提示可以对优化进行更明确的控制。 例如：
import grapheneimport graphene_django_optimizer as gql_optimizerclass ItemType(gql_optimizer.OptimizedDjangoObjectType): name = graphene.String()@gql_optimizer.resolver_hints( select_related=('product', 'shipping'), only=('product__name', 'shipping__name'), ) def resolve_name(root, info): return '{} {}'.format(root.product.name, root.shipping.name)TLDR; anticipate that any features that gain adoption will have a performance tax you’ll need to pay after a few weeks or months.
TLDR； 预计数周或数月后，您将需要支付的性能税将被采用。
Performance issues tend to sneak up on you at an exponential rate and carry on wreaking havoc until they’re addressed.
性能问题往往以指数级的速度潜伏在您身上，并造成严重破坏，直到得到解决。
Here’s the scenario: you release a new feature that will demand a constant read & write throughput on the system (end clients through to the DB). During development, testing and to some extent live early adoption you won’t spot many performance issues as the corpus is too small to put a strain on the system. E.g you’ll have very few rows in a DB table for a slow query to be noticeable.
这是场景：您发布了一项新功能，该功能将要求系统(最终客户端到数据库)上具有恒定的读写吞吐量。 在开发，测试和某种程度上可以实时采用的过程中，由于语料库太小而无法给系统带来压力，因此不会发现很多性能问题。 例如，您在数据库表中将只有很少的几行，以使缓慢的查询引人注目。
Over time as new users are signed up to the new feature, the corpus will grow and you’ll get a slow but steady degradation of the system as DB transactions put locks on tables, CPU loads rise etc which will all lead to requests backing up.
随着时间的推移，随着新用户注册新功能，语料库将增长，并且由于数据库事务在表上锁定，CPU负载增加等原因，您的系统将缓慢但稳定地降级，所有这些都会导致请求备份。
The way I’ve handled this challenge is by setting up alarms (AWS’s CloudWatch alarms) around spikes in metrics such as average response time, sustained DB CPU load and general anomaly detection. Once alarms are triggered, typically I’m looking for isolated faults but serendipitously I’ll discover high load on services in the same way you’d spot in your OS process monitor. Once a performance issue is detected, it’s noted and triaged with the product & bug backlog and in the meantime, it’s normally best to horizontally or vertically scale to temporarily stymie the issue.
我处理此挑战的方法是围绕平均响应时间，持续的DB CPU负载和常规异常检测等指标的峰值设置警报(AWS的CloudWatch警报)。 触发警报后，通常我会寻找孤立的故障，但是偶然地，我会以与在OS进程监视器中发现的相同方式发现服务的高负载。 一旦检测到性能问题，便将其记录下来并与积压的产品和bug进行分类，同时，通常最好水平或垂直缩放以暂时解决问题。
TLDR; Enable verbose DB logging and when you spot more activity than normal, you know you’ve got a problem.
TLDR； 启用详细的数据库日志记录，当发现活动超出正常水平时，您就知道有问题。
I’ve found simply logging SQL queries on dev machines to be the most effective indicator of inefficient lookups. Simply add:
我发现仅在开发人员机器上记录SQL查询是低效率查找的最有效指示。 只需添加：
LOGGING = { 'loggers': { 'django.db.backends': { 'level': 'DEBUG', 'handlers': ['console'], } }}Now when calling any API endpoints or whole pages, you’ll see a lot of activity when things are bad.
现在，当调用任何API端点或整个页面时，当情况变坏时，您将看到很多活动。
The Django toolbar is also great for surfacing problems with queries but you have to be looking at it consciously and it doesn’t work with API (GraphQL & REST) requests so I favour the serendipitous discovery through a noisy terminal.
Django工具栏也非常适合解决查询问题，但是您必须自觉地查看它，并且它不适用于API(GraphQL和REST)请求，因此我赞成通过一个嘈杂的终端进行偶然发现。
TLDR; When faced with a performance issue, it might be cheaper and more reliable to re-architect whole parts of the system.
TLDR； 遇到性能问题时，重新架构系统的整个部分可能会更便宜且更可靠。
It’s helpful to have a grasp of the end-to-end service when approaching performance issues as you can find solutions that scale well into the future.
在处理性能问题时，掌握端到端服务很有帮助，因为您可以找到可以很好地扩展到未来的解决方案。
A recent example was our mobile app’s data fetching lifecycle. Without dropping to our domain model, let’s use a fictional app called InstaCalendar which gets all your friend’s and their calendar events for the day. We were (analogously) loading a list of all your friends, then for each of those friends, loading their calendar events. Performance issues aren’t felt if you have few friends and those friends have little going on in their lives but the popular ones (the power users), will quickly feel the performance hit as they’ll have a lot of friends and a lot of events to fetch because they’re also busy.
最近的一个例子是我们的移动应用程序的数据获取生命周期。 在不使用域模型的情况下，让我们使用一个名为InstaCalendar的虚构应用程序，该应用程序可以获取当天您所有好友及其日历事件。 我们(类似地)加载了所有朋友的列表，然后为每个朋友加载了他们的日历活动。 如果您的朋友很少，而这些朋友的生活很少，就不会感觉到性能问题，但是受欢迎的朋友(超级用户)很快就会感觉到性能下降，因为他们会有很多朋友，并且很多要获取事件，因为它们也很忙。
In this scenario, you’ll pick up on this when profiling the app after getting reports that the power users are suffering bad UX and you try and replicate their data conditions.
在这种情况下，您将在获得有关高级用户遭受不良UX的报告并尝试复制其数据条件的报告后，在对应用程序进行性能分析时继续学习。
Once you spot this problem and realise how it scales badly (as you’re at the mercy of lots of network request), you have two main options;
一旦发现了这个问题并意识到它的扩展性很差(因为您受到许多网络请求的支配)，您将有两个主要选择：
Add realtime/push updates (e.g. WebSockets) to only fetch data (event) deltas 添加实时/推送更新(例如WebSockets)以仅获取数据(事件)增量 Batch the event fetching into one request, ideally decoupled from the loading of friends将事件抓取成批处理到一个请求中，最好与加载朋友分离In this decision making phase, I’d err towards the boring but reliable tech choice (2) as it’ll incur less risk and you’ll get the solution to users, faster. Here, you’ll be required to orchestrate a change at the mobile and backend which, if you are broken up into functional teams, will yield some friction but it’s a useful route to familiarising yourself with, given the macro benefits and tendency to reoccur.
在这个决策阶段，我会选择无聊但可靠的技术选择(2)，因为这样可以降低风险，并且可以更快地为用户提供解决方案。 在这里，您将需要在移动设备和后端进行协调，如果您分成职能团队，这会产生一些磨擦，但是鉴于宏的好处和发生的趋势，这是熟悉自己的有用途径。
Another example we’ve used was around requests that require a lot of processing and data transfer. In our case, this was CSV generation and download.
我们使用的另一个示例是关于需要大量处理和数据传输的请求。 在我们的案例中，这是CSV生成和下载。
We initially had a simple view & controller for generating CSV files but once new fields, filters and computation was added it became untenable in its current form, even with heavy use of memory-efficient generators.
最初，我们有一个简单的视图和控制器来生成CSV文件，但是一旦添加了新的字段，过滤器和计算，即使大量使用了内存高效的生成器，它也无法以其当前形式运行。
When addressing any requests where it’s likely to invoke a non-trivial amount of processing (you should be able to predict this), I’d advise moving to an asynchronous processing model which will typically be a task queue (e.g Celery) or something more exotic (Node.js/Twisted/AWS Lambda).
当处理任何可能调用少量处理的请求(您应该能够预测到这一点)时，我建议您转向异步处理模型，该模型通常是任务队列(例如Celery )或其他。异国情调(Node.js / Twisted / AWS Lambda)。
We’re using Zappa for a serverless infrastructure so we ported our view to an async task (Zappa task) which put the results on S3 which a frontend script would poll and download from S3 when ready. This saved a heap of issues with server load but required a fair amount of engineering to make it all work seamlessly and securely.
我们将Zappa用于无服务器基础架构，因此我们将视图移植到了一个异步任务( Zappa task )，该任务将结果放在S3上，当准备就绪时，前端脚本将从S3进行轮询和下载。 这样可以节省大量服务器负载问题，但需要大量的工程设计才能使其无缝，安全地工作。
No performance post would be complete without talking about caching but I tend to avoid caching data where possible despite the huge gains it can provide. This comes with the caveat that we operate at a modest scale in a regulated space, handling sensitive data and have a ~3:1 read-write ratio of data.
如果不谈论缓存，任何性能发布都将是不完整的，但是尽管它可以提供巨大的收益，但我倾向于避免在可能的情况下缓存数据。 需要注意的是，我们在有限的空间内以适度的规模运行，处理敏感数据，并且数据的读写比率约为3：1。
I take this stance as I feel there’s a much bigger cost of serving stale or incorrect data, managing cache invalidation and maintaining more subsystems vs. the gain you can get from tuning your queries. It turns out most databases can be blisteringly fast at getting data if you have the right indexes and queries.
我之所以采取这种立场，是因为我认为处理陈旧或不正确的数据，管理缓存无效化和维护更多的子系统的成本要比从调整查询中获得的收益高得多。 事实证明，如果您拥有正确的索引和查询，大多数数据库在获取数据方面的速度可能会非常快。
The only data caches we use tend to be ephemeral, low-risk lookups of things like internal metrics and deterministic, expensive calculations. I’ve found lru_cache to be super easy to drop into projects of this kind.
我们使用的唯一数据缓存往往是对内部指标和确定性，昂贵的计算等事物的临时性，低风险查找。 我发现将lru_cache放入此类项目非常容易。
Our static assets, on the other hand, are highly cached, making use of hashed filenames with long-lived cache expiry and ETags fronted by CloudFront CDN.
另一方面，我们的静态资产是高度缓存的，它使用具有长期缓存过期期限的哈希文件名和CloudFront CDN前面的ETag。
My preference is to avoid generators as I feel it leads to obtuse code at the benefit of the machine over the human, but I have found them a necessary evil to handle memory consuming operations like CSV generation. Here it’s a case of using the right tool for the job, even if you’re not fond of the tool.
我的偏好是避免使用生成器，因为我认为生成器会使机器受益于人类，从而使代码变得晦涩难懂，但是我发现它们对于处理诸如CSV生成之类的消耗内存的操作是必不可少的。 在这种情况下，即使您不喜欢该工具，也需要使用正确的工具来完成工作。
We spent a 2-week “sprint” fixing performance issues as we were spending a lot on our infrastructure (relative to the past) and customers were complaining. The main objective of this sprint was a faster experience for the end-user (measured in the speed of the slowest, most frequent requests) as we tend to do everything to optimises for the user. The infrastructure and cost issues should benefit from fixes applied to the user’s UX improvements.
我们花了2周的“冲刺”来解决性能问题，因为我们在基础架构上花了很多钱(相对于过去)，而客户也在抱怨。 此sprint的主要目标是为最终用户提供更快的体验(以最慢，最频繁的请求的速度衡量)，因为我们倾向于尽一切努力为用户进行优化。 基础结构和成本问题应从应用于用户UX改进的修补程序中受益。
The process followed largely what has been mentioned above, start with the metrics, fix the problems and repeat until you run out of time on the sprint.
该过程主要遵循上述内容，从指标开始，解决问题，然后重复进行，直到您在sprint上用完时间为止。
The results from our latest performance sprint were significant:
我们最新的性能冲刺结果非常显着：
We saw a big decrease in the number of complaints about performance. Yay! 我们发现，有关绩效的投诉数量大大减少了。 好极了！ We saw a net decrease in the amount of time it took our users to get the data they required. 我们发现用户获取所需数据所花费的时间有所减少。 We saw a significant reduction in required AWS capacity which led to a big cost saving. Here’s a graph that illustrates that: 我们发现所需的AWS容量显着减少，从而节省了大量成本。 以下图表说明了这一点：In summary, there’s a certain art to performance tuning but there’s no short supply of resources to get help. The Django docs themselves are a great further read https://docs.djangoproject.com/en/3.1/topics/db/optimization/.
总而言之，性能调整有一定的技巧，但没有短缺的资源可以寻求帮助。 Django文档本身对https://docs.djangoproject.com/zh-CN/3.1/topics/db/optimization/有很大的帮助。
A reminder that this post was written with the bias towards a service that values up-to-date data and security over absolute speed. If you’re a media publisher, I’d advise finding a post that goes deep on caching strategies.
提醒您，这篇文章的撰写偏向于一种服务，该服务重视绝对速度上的最新数据和安全性。 如果您是媒体发行商，建议您找一篇有关缓存策略的文章。
Say hello to me on Twitter @kulor
在Twitter @kulor向我问好
翻译自: https://medium.com/@kulor/pragmatic-performance-tuning-django-e5218abeb9e6
相关资源：django-stdimage：Django标准化图片字段-源码