云计算中的海量数据存储在哪
Are you experiencing slow transfer speeds between your GCE VM and a Cloud Storage bucket? Then read on to learn how to maximize your upload and download throughput for Linux and Windows VMs!
您在GCE VM和Cloud Storage存储桶之间的传输速度是否很慢？ 然后继续阅读以了解如何最大程度地提高Linux和Windows VM的上传和下载吞吐量！
I recently had a DoiT International customer ask why their data on a Google Compute Engine (henceforth referred to as GCE) Windows Server’s local SSD was uploading to a Cloud Storage bucket much slower than expected. At first, I began with what I thought would be a simple benchmarking of gsutil commands to demonstrate the effectiveness of using ideal gsutil arguments as recommended by the GCP documentation. Instead, my ‘quick ’ look into the issue turned into a full-blown investigation into data transfer performance between GCE and GCS, as my initial findings were unusual and very much unexpected.
我最近有一个DoiT International客户问为什么他们在Google Compute Engine(以下称为GCE)Windows Server的本地SSD上的数据上载到Cloud Storage存储桶的速度比预期的要慢得多。 最初，我以对gsutil命令的简单基准测试为gsutil以证明按照GCP文档的建议使用理想的gsutil参数的有效性。 取而代之的是，我的快速发现变成了对GCE和GCS之间数据传输性能的全面调查，因为我的最初发现并不寻常，而且非常出乎意料。
If you are simply interested in knowing what the ideal methods are for moving data between GCE and GCS for a Linux or Windows machine, go ahead and scroll all the way down to “Effective Transfer Tool Use Conclusions”.
如果您只是想了解在Linux或Windows计算机上的GCE和GCS之间移动数据的理想方法，请继续向下滚动至“有效的传输工具使用结论”。
If you instead want to scratch your head over the bizarre, often counter-intuitive throughput rates achievable through commonly used commands and arguments, stay with me as we dive into the details of what led to my complex recommendations summary at the end of this article.
如果您反而想通过通常使用的命令和参数来达到奇怪的，通常与直觉相反的吞吐率，那么请与我同在，因为我们将深入探讨导致本文复杂的建议摘要的细节。
Although the customer’s request involved data transfer on a Windows server, I first performed basic benchmarking where I felt the most comfortable:Linux, via the “Debian GNU/Linux 10 (buster)” GCE public image.
尽管客户的要求涉及在Windows服务器上进行数据传输，但我还是先进行了最舒适的基本基准测试：Linux，通过“ Debian GNU / Linux 10(破坏者)” GCE公共映像进行。
Since the customer was already attempting file transfers from local SSDs and I wanted to minimize the odds that networked disks would impact transfer speeds, I configured two VM sizes, n2-standard-4 and n2-standard-80, with each having one local SSD attached where we will perform benchmarking.
由于客户已经在尝试从本地SSD进行文件传输，并且我想最大程度地降低网络磁盘会影响传输速度的几率，因此我配置了两个VM大小，n2-standard-4和n2-standard-80，每个VM都有一个本地SSD附上我们将执行基准测试的位置。
The GCS bucket I will use, as well as all VMs described in this article, are created as regional resources located in us-central1.
我将使用的GCS存储桶以及本文中介绍的所有VM被创建为位于us-central1中的区域资源。
To simulate the customer’s large file upload experience, I created an empty file 30 GBs in size:
为了模拟客户的大文件上传体验，我创建了一个大小为30 GB的空文件：
fallocate -l 30G temp_30GB_fileFrom here, I tested two commonly recommended gsutil parameters:
从这里开始，我测试了两个常用的gsutil参数：
-m: Used to perform parallel, multi-threaded copy. Useful for transferring a large number of files in parallel, not the upload of individual files.
-m ：用于执行并行的多线程复制。 对于并行传输大量文件很有用，而不是单个文件的上传。
-o GSUtil:parallel_composite_upload_threshold=150M: Used to split large files exceeding the specified threshold into parts that are then uploaded in parallel and combined upon upload completion of all parts.
-o GSUtil:parallel_composite_upload_threshold=150M ：用于将超出指定阈值的大文件拆分为多个部分，然后并行上传并在所有部分完成上传后合并。
The estimated max performance for the local SSD on both VMs is as follows:
估计两个虚拟机上本地SSD的最大性能如下：
Local SSD Read/Write throughput limits 本地SSD读/写吞吐量限制We should therefore be able to achieve up to 660 MB/s read and 350 MB/s write throughput with gsutil. Let’s see what the upload benchmarks revealed:
因此，使用gsutil我们应该能够达到660 MB / s的读取速度和350 MB / s的写入吞吐量。 让我们看看上传基准测试显示了什么：
time gsutil cp temp_30GB_file gs://doit-speed-test-bucket/# n2-standard-4: 2m21.893s, 216.50 MB/s# n2-standard-80: 2m11.676s, 233.30 MB/stime gsutil -m cp temp_30GB_file gs://doit-speed-test-bucket/# n2-standard-4: 2m48.710s, 182.09 MB/s# n2-standard-80: 2m29.348s, 205.69 MB/stime gsutil -o GSUtil:parallel_composite_upload_threshold=150M cp temp_30GB_file gs://doit-speed-test-bucket/# n2-standard-4: 1m40.104s, 306.88 MB/s# n2-standard-80: 0m52.145s, 589.13 MB/stime gsutil -m -o GSUtil:parallel_composite_upload_threshold=150M cp temp_30GB_file gs://doit-speed-test-bucket/# n2-standard-4: 1m44.579s, 293.75 MB/s# n2-standard-80: 0m51.154s, 600.54 MB/sAs expected based on GCP’s gsutil documentation, large file uploads benefit from including -o GSUtil. When more vCPUs are made available to assist in the parallel upload of file parts, upload time is improved dramatically, to the point that with a consistent 600 MB/s upload speed on the n2-standard-80 we come close to achieving the SSD’s max throughput of 660 MB/s. Including -m for only one file decreases upload time by a few seconds. So far, we’ve seen nothing out of the ordinary.
正如基于GCP的gsutil文档所预期的那样，大文件上传受益于-o GSUtil 。 当提供更多vCPU来协助并行上传文件部分时，上传时间将大大缩短，以至于n2-standard-80上的一致600 MB / s的上传速度，我们几乎可以达到SSD的最大值吞吐量为660 MB / s。 仅对一个文件包含-m上传时间减少几秒钟。 到目前为止，我们没有发现任何异常。
Let’s check out the download benchmarks:
让我们检查一下下载基准：
time gsutil cp gs://doit-speed-test-bucket/temp_30GB_file .# n2-standard-4: 8m3.186s, 63.58 MB/s# n2-standard-80: 6m13.585, 82.23 MB/stime gsutil -m cp gs://doit-speed-test-bucket/temp_30GB_file .# n2-standard-4: 7m57.881s, 64.28 MB/s# n2-standard-80: 6m20.131s, 80.81 MB/s Hol up 霍尔Download performance on the 80 vCPU VM only achieved 23% of the maximum local SSD write throughput. Additionally, despite the enabling of multi-threading with -m not improving performance for this single file download, and despite both machines utilizing well under their maximum throughput (10 Gbps for n2-standard-4, 32 Gbps for n2-standard-80), evidently using a higher tier machine within the same family leads to a ~30% improvement in download speed. Weird, but not as weird as getting only 1/4th of local SSD write throughput with an absurdly expensive VM.
在80 vCPU VM上的下载性能仅达到最大本地SSD写入吞吐量的23％。 此外，尽管启用了带有-m的多线程功能并没有提高此单文件下载的性能，并且尽管两台机器在其最大吞吐量下都利用得很好(n2-standard-4为10 Gbps，n2-standard-80为32 Gbps) ，显然，在同一系列中使用更高级别的计算机可以使下载速度提高约30％。 奇怪，但并不奇怪，而价格昂贵的VM仅获得本地SSD写入吞吐量的1/4。
What is going on?
到底是怎么回事？
After much searching around on this issue, I found no answers but instead discovered s5cmd, a tool designed to dramatically improve uploads to and downloads from S3 buckets. It claims to run 12X faster than the equivalent AWS CLI commands (e.g. aws s3 cp) due in large part to being written in Go, a compiled language, versus the AWS CLI that is written in Python. It just so happens that gsutil is also written in Python. Perhaps gsutil is severely hampered by its language choice, or simply optimized poorly? Given that GCS buckets can be configured to have S3 API Interoperability, is it possible to speed up uploads and downloads with s5cmd by simply working with a compiled tool?
在对该问题进行了大量搜索之后，我没有找到任何答案，而是找到了s5cmd ，该工具旨在显着改善到S3存储桶的上传和下载。 它声称比等效的AWS CLI命令(例如aws s3 cp )运行速度快12倍，这主要是因为它是用Go(一种编译语言)编写的，而不是用Python编写的AWS CLI。 碰巧gsutil也用Python编写。 也许gsutil的语言选择受到严重阻碍，或者只是优化不佳？ 鉴于可以将GCS存储桶配置为具有S3 API互操作性，是否可以通过仅使用编译工具来使用s5cmd加快上载和下载？
It took a little bit to get s5cmd working, mostly because I had to discover the hard way that GCS Interoperability doesn’t support S3’s multipart upload API, and given that this tool is written only with AWS in mind it will fail on large file uploads in GCP. You must provide -p=1000000, an argument that forces multi-part upload to be avoided. See s5cmd issues #1 and #2 for more info.
s5cmd时间，主要是因为我不得不发现GCS互操作性不支持S3的分段上传API的困难方式，并且考虑到此工具仅针对AWS编写，因此在大文件上传时将失败在GCP中。 您必须提供-p=1000000 ，该参数强制避免分段上传。 有关更多信息，请参见s5cmd问题1和2 。
Note that s5cmd also offers a -c parameter for setting the number of concurrent parts/files transferred, with a default value of 5.
请注意， s5cmd还提供了-c参数，用于设置传输的并发部分/文件的数量，默认值为5。
With those two arguments in mind I performed the following Linux upload benchmarks:
考虑到这两个参数，我执行了以下Linux上传基准测试：
time s5cmd --endpoint-url https://storage.googleapis.com cp -c=1 -p=1000000 temp_30GB_file s3://doit-speed-test-bucket/# n2-standard-4: 6m7.459s, 83.60 MB/s# n2-standard-80: 6m50.272s, 74.88 MB/stime s5cmd --endpoint-url https://storage.googleapis.com cp -p=1000000 temp_30GB_file s3://doit-speed-test-bucket/# n2-standard-4: 7m18.682s, 70.03 MB/s# n2-standard-80: 6m48.380s, 75.22 MB/sAs expected, large file uploads perform considerably worse compared to gsutil given the lack of a multi-part upload strategy as an option. We are seeing 75–85 MB/s upload compared to gsutil’s 200–600 MB/s. Providing concurrency of 1 vs. the default 5 only has a small impact on improving performance. Thus, due to s5cmd’s treatment of AWS as a first-class citizen without consideration for GCP, we cannot improve uploads by using s5cmd.
如预期的那样，由于缺少gsutil上传策略作为选择，因此与gsutil相比，大文件上传的性能要差得多。 与gsutil的200-600 MB / s相比，我们看到75-85 MB / s的上传速度。 提供1与默认5的并发性只会对提高性能产生很小的影响。 因此，由于s5cmd将AWS视为头等公民而不考虑GCP，因此我们无法使用s5cmd来改善上传。
Below are the s5cmd download benchmarks:
以下是s5cmd下载基准：
time s5cmd --endpoint-url https://storage.googleapis.com cp -c=1 -p=1000000 s3://doit-speed-test-bucket/temp_30GB_file .# n2-standard-4: 1m56.170s, 264.44 MB/s# n2-standard-80: 1m46.196s, 289.28 MB/stime s5cmd --endpoint-url https://storage.googleapis.com cp -c=1 s3://doit-speed-test-bucket/temp_30GB_file .# n2-standard-4: 3m21.380s, 152.55 MB/s# n2-standard-80: 3m45.414s, 136.28 MB/stime s5cmd --endpoint-url https://storage.googleapis.com cp -p=1000000 s3://doit-speed-test-bucket/temp_30GB_file .# n2-standard-4: 2m33.148s, 200.59 MB/s# n2-standard-80: 2m48.071s, 182.78 MB/stime s5cmd --endpoint-url https://storage.googleapis.com cp s3://doit-speed-test-bucket/temp_30GB_file .# n2-standard-4: 1m46.378s, 288.78 MB/s# n2-standard-80: 2m1.116s, 253.64 MB/sWhat a dramatic improvement! While there is some variability in download time, it seems that by leaving out -c and -p, leaving them to their defaults, we achieve optimal speed. We are unable to reach the max write throughput of 350 MB/s, but ~289 MB/s on an n2-standard-4 is much closer to that than ~64 MB/s provided by gsutil on the same machine. That is a 4.5X increase in download speed simply by swapping out the data transfer tool used.
多么了不起的进步！ 尽管下载时间有所不同，但似乎通过省略-c和-p ，将它们保留为默认值，可以达到最佳速度。 我们无法达到350 MB / s的最大写入吞吐量，但是n2-standard-4上的〜289 MB / s与同一台计算机上gsutil提供的gsutil MB / s相比要近得多。 只需换出使用的数据传输工具，下载速度即可提高4.5倍。
Summarizing all of the above findings, for Linux:
总结以上针对Linux的所有发现：
Given that s5cmd cannot enable multi-part uploads when working with GCS, it makes sense to continue using gsutil for upload to GCS so long as you include -o GSUtil:parallel_composite_upload_threshold=150M.
鉴于s5cmd在使用GCS时无法启用s5cmd上传，因此只要您包含-o GSUtil:parallel_composite_upload_threshold=150M ，就可以继续使用gsutil上载到GCS。
s5cmd with its default parameters blows gsutil out of the water in download performance. Simply utilizing a data transfer tool written with a compiled language yields dramatic (4.5X) performance improvements.
带有默认参数的s5cmd gsutil的下载性能s5cmd 。 只需使用以编译语言编写的数据传输工具，即可显着提高(4.5X)性能。
If you thought the above wasn’t unusual enough, buckle in as we go off the deep end with Windows. Since the DoIT customer was dealing with Windows Server, after all, it was time to set out on benchmarking that OS. I began to suspect their problem was not going to be between the keyboard and the chair.
如果您认为上述情况还不够特殊，请在我们深入了解Windows的同时，尝试一下。 毕竟，由于DoIT客户正在与Windows Server打交道，因此该开始对该操作系统进行基准测试了。 我开始怀疑他们的问题不会在键盘和椅子之间。
Having confirmed that, for Linux, gsutil works great for upload when given the right parameters and s5cmd works great for download with default parameters, it was time to try these commands on Windows where I will once again feel humbled by my lack of experience with Powershell.
确认对于Linux， gsutil在提供正确参数的情况下非常适合上传，而s5cmd在使用默认参数的情况下也非常适合下载，是时候在Windows上尝试这些命令了，我由于对Powershell缺乏经验而再次感到沮丧。
I eventually managed to gather benchmarks from an n2-standard-4 machine with a local SSD attached running on the “Windows Server version 1809 Datacenter Core for Containers, built on 20200813” GCE VM image. Due to the per vCPU licensing fees that Windows server charges, I’ve opted to not gather metrics from an n2-standard-80 in this experiment.
我最终设法从一台n2-standard-4机器上收集了基准，该机器上装有本地SSD，该机器运行在“ Windows Server版本1809容器数据中心核心(建于20200813)” GCE VM映像上。 由于Windows服务器收取的每vCPU许可费用，因此我选择不从该实验的n2-standard-80收集指标。
An important side note before we dive into the metrics:The GCP documentation on attaching local SSDs recommends that for “All Windows Servers” you should use the SCSI driver to attach your local SSD rather than the NVMe driver you typically use for a Linux machine, as SCSI is better optimized for achieving maximum throughput performance. I went ahead and provisioned two VMs with a local SSD attached, one attached via NVMe and one via SCSI, determined to compare their performance alongside the various tools and parameters I’ve been investigating thus far.
在深入了解指标之前，请注意以下重要注意事项：GCP有关附加本地SSD的文档建议，对于“所有Windows服务器”，应使用SCSI驱动程序附加本地SSD，而不是通常用于Linux计算机的NVMe驱动程序， SCSI进行了更好的优化，以实现最大的吞吐量性能。 我继续进行操作，并为两台配备了本地SSD的VM进行了配置，其中一台通过NVMe进行了连接，另一台通过SCSI进行了安装，因此决定将它们的性能与我迄今为止研究的各种工具和参数进行比较。
Below are the upload speed benchmarks:
以下是上传速度基准：
Measure-Command {gsutil cp temp_30GB_file gs://doit-speed-test-bucket/}# NVMe: 3m50.064s, 133.53 MB/s# SCSI: 4m7.256s, 124.24 MB/sMeasure-Command {gsutil -m cp temp_30GB_file gs://doit-speed-test-bucket/}# NVMe: 3m59.462s, 128.29 MB/s# SCSI: 3m34.013s, 143.54 MB/sMeasure-Command {gsutil -o GSUtil:parallel_composite_upload_threshold=150M cp temp_30GB_file gs://doit-speed-test-bucket/}# NVMe: 5m54.046s, 86.77 MB/s# SCSI: 6m13.929s, 82.15 MB/sMeasure-Command {gsutil -m -o GSUtil:parallel_composite_upload_threshold=150M cp temp_30GB_file gs://doit-speed-test-bucket/}# NVMe: 5m55.751s, 86.40 MB/s# SCSI: 5m58.078s, 85.79 MB/s There are no words to convey my emotions right now 现在没有语言可以传达我的情感With no arguments provided to gsutil, upload throughput is ~60% of the throughput achieved on a Linux machine. Providing any combination of arguments degrades performance. When multi-part upload is enabled—which led to a 42% improvement in upload speed on Linux — the upload speed drops by 35%. You may also notice that when -m is not provided and gsutil is allowed to upload more optimally for a single large file, upload from the NVMe drive completes more quickly than on the SCSI drive, the latter of which supposedly has drivers more optimized for Windows Servers. What is going on?!
没有为gsutil提供任何参数，上传吞吐量约为Linux计算机上实现的吞吐量的60％。 提供参数的任何组合都会降低性能。 启用分段上传后，Linux上的上传速度提高了42％，上传速度下降了35％ 。 您可能还会注意到，当未提供-m且允许gsutil为单个大文件进行更优化的上传时，从NVMe驱动器上载的完成比在SCSI驱动器上完成的速度要快，后者应该具有针对Windows优化的驱动程序服务器。 到底是怎么回事？！
Low upload performance around 80–85 MB/s was the exact range that the DoiT customer was experiencing, so their problem was at least reproducible. By removing the GCP-recommended argument-o GSUtil:parallel_composite_upload_threshold=150M for large file uploads, the customer could remove a 35% performance penalty. 🤷
DoiT客户所经历的确切范围是大约80–85 MB / s的低上传性能，因此，他们的问题至少可以重现。 通过删除针对大型文件上传的GCP推荐参数-o GSUtil:parallel_composite_upload_threshold=150M ，客户可以消除35％的性能损失。 🤷
Download benchmarking tells a more harrowing tale:
下载基准测试讲述了一个更令人痛苦的故事：
Measure-Command {gsutil cp gs://doit-speed-test-bucket/temp_30GB_file .}# NVMe 1st attempt: 11m39.426s, 43.92 MB/s# NVMe 2nd attempt: 9m1.857s, 56.69 MB/s# SCSI 1st attempt: 8m54.462s, 57.48 MB/s# SCSI 2nd attempt: 10m1.023s, 51.05 MB/sMeasure-Command {gsutil -m cp gs://doit-speed-test-bucket/temp_30GB_file .}# NVMe 1st attempt: 8m52.537s, 57.69 MB/s# NVMe 2nd attempt: 22m4.824s, 23.19 MB/s# NVMe 3rd attempt: 8m50.202s, 57.94 MB/s# SCSI 1st attempt: 7m29.502s, 68.34 MB/s# SCSI 2nd attempt: 9m9.652s, 55.89 MB/sI could not yield consistent download benchmarks due to the following issue:
由于以下问题，我无法产生一致的下载基准：
Each download operation would hang for up to 2 minutes before initiating 每次下载操作最多会挂起2分钟，然后再启动 The download would begin and progress at about 68–70 MB/s, until…下载将开始并以大约68–70 MB / s的速度进行，直到…It sometimes paused again for an indeterminate amount of time 有时会暂停一段不确定的时间The process of hanging up and re-initiating the download would continue back-and-forth, causing same-VM same-disk download speed averages to range from 23 MB/s to 58 MB/s. It was madness trying to determine whether NVMe or SCSI were more optimal for downloads with these random, extended hang-ups in the download process. More on this verdict later.
挂断并重新启动下载的过程将反复进行，从而导致相同VM的相同磁盘下载速度平均范围为23 MB / s至58 MB / s。 试图通过下载过程中的这些随机扩展挂断来确定NVMe或SCSI对于下载是否更优化是疯狂的。 稍后将对此判决进行更多讨论。
Frustrated with wild and wacky gsutil download performance, I quickly moved on to s5cmd — perhaps it might solve or reduce the impact of hangups?
我对狂野古怪的gsutil下载性能感到沮丧，因此我Swift转到s5cmd ，也许它可以解决或减少挂断的影响？
Let’s cover the s5cmd upload benchmarks first:
让我们首先介绍s5cmd上传基准：
Measure-Command {s5cmd --endpoint-url https://storage.googleapis.com cp -c=1 -p=1000000 temp_30GB_file s3://doit-speed-test-bucket/}# NVMe: 6m21.780s, 80.46 MB/s# SCSI: 7m14.162s, 70.76 MB/sMeasure-Command {s5cmd --endpoint-url https://storage.googleapis.com cp -p=1000000 temp_30GB_file s3://doit-speed-test-bucket/}# NVMe: 12m56.066s, 39.58 MB/s# SCSI: 8m12.255s, 62.41 MB/sSimilar to s5cmd upload on Linux, it is hampered by the inability to utilize multi-part uploads. Upload performance with concurrency set to 1 is comparable to that achieved by the same tool on a Linux machine, but performance with concurrency left to its default value of 5 causes dramatic drops (and swings) in performance. Including concurrency is unusual in the severity of its impact, but since s5cmd upload performance continues to be markedly worse than gsutil upload performance (strange given that this is true when both are not using multi-part uploads) we don’t want to use s5cmd for uploads anyway; let’s just ignore s5cmd’s upload concurrency oddity.
与Linux上的s5cmd上载类似，它无法使用多部分上载受到阻碍。 并发设置为1的上载性能与Linux计算机上相同工具所达到的性能相当，但是将并发设置为默认值5的性能会导致性能急剧下降(或波动)。 并发对其影响的严重性是不寻常的，但是由于s5cmd上传性能仍然明显比gsutil上传性能差(奇怪的是，当两者都不使用s5cmd上传时，这是正确的)，因此我们不想使用s5cmd无论如何用于上传； 让我们忽略s5cmd的上传并发s5cmd 。
Moving on to the s5cmd download benchmarks:
继续进行s5cmd下载基准测试：
Measure-Command {s5cmd --endpoint-url https://storage.googleapis.com cp -c=1 -p=1000000 s3://doit-speed-test-bucket/temp_30GB_file .}# NVMe 1st attempt: 2m17.954s, 222.68 MB/s# NVMe 2nd attempt: 1m44.718s, 293.36 MB/s# SCSI 1st attempt: 3m9.581s, 162.04 MB/s# SCSI 2nd attempt: 1m52.500s, 273.07 MB/sMeasure-Command {s5cmd --endpoint-url https://storage.googleapis.com cp -c=1 s3://doit-speed-test-bucket/temp_30GB_file .}# NVMe 1st attempt: 3m18.006s, 155.15 MB/s# NVMe 2nd attempt: 4m2.792s, 126.53 MB/s# SCSI 1st attempt: 3m37.126s, 141.48 MB/s# SCSI 2nd attempt: 4m9.657s, 123.05 MB/sMeasure-Command {s5cmd --endpoint-url https://storage.googleapis.com cp -p=1000000 s3://doit-speed-test-bucket/temp_30GB_file .}# NVMe 1st attempt: 2m17.151s, 223.99 MB/s# NVMe 2nd attempt: 1m47.217s, 286.52 MB/s# SCSI 1st attempt: 4m39.120s, 110.06 MB/s# SCSI 2nd attempt: 1m42.159s, 300.71 MB/sMeasure-Command {s5cmd --endpoint-url https://storage.googleapis.com cp s3://doit-speed-test-bucket/temp_30GB_file .}# NVMe 1st attempt: 2m48.714s, 182.08 MB/s# NVMe 2nd attempt: 2m41.174s, 190.60 MB/s# SCSI 1st attempt: 2m35.480s, 197.58 MB/s# SCSI 2nd attempt: 2m40.483s, 191.42 MB/sWhile there are some hangups and variability with downloads as with gsutil, s5cmd is much more performant than gsutil at downloads once again. It also experienced lower duration and/or lower frequency hangups. These strange hangups still remain an occasional issue, though.
尽管与gsutil ，下载也有一些问题和变化， s5cmd性能比gsutil再次高得多。 它还经历了较短的持续时间和/或较低的频率挂断。 但是，这些奇怪的挂断仍然是一个偶然的问题。
In contrast to how I achieved maximum performance on a Linux VM by leaving out the -c and -p parameters, optimal performance seems to have been achieved by including both of these with -c=1 -p=1000000. It is difficult to declare inclusion of these as the most optimal configuration given the random hangups dogging my benchmarks, but it seems to run well enough with these arguments. As with gsutil, it is also challenging to determine whether NVMe or SCSI are better optimized due to the hangups.
与我如何通过省略-c和-p参数在Linux VM上实现最大性能相反，通过将这两个参数都包含-c=1 -p=1000000似乎可以实现最佳性能。 考虑到随机挂断困扰着我的基准，很难宣布将它们包括为最佳配置，但是使用这些参数似乎运行得很好。 与gsutil ，确定是否由于挂断而更好地优化了NVMe或SCSI也具有挑战性。
In an attempt to better understand download speeds on NVMe and SCSI with optimal s5cmd arguments, I wrote a function that reports the Avg, Min, and Max runtime from 20 repeated downloads, with the goal of averaging out the momentary hangups:
为了更好地了解具有最佳s5cmd参数的NVMe和SCSI上的下载速度，我编写了一个函数，该函数报告了20次重复下载的平均，最小和最大运行时间，目的是平均暂时的挂断：
Measure-CommandAvg {s5cmd --endpoint-url https://storage.googleapis.com cp -c=1 -p=1000000 s3://doit-speed-test-bucket/temp_30GB_file .}### With 20 sample downloads# NVMe:# Avg: 1m48.014s, 284.41 MB/s# Min: 1m23.411s, 368.30 MB/s# Max: 3m10.989s, 160.85 MB/s# SCSI: # Avg: 1m47.737s, 285.14 MB/s# Min: 1m24.784s, 362.33 MB/s# Max: 4m44.807s, 107.86 MB/sThere continues to be variability in how long the same download takes to complete, but it is evident that SCSI does not provide an advantage over NVMe in general for large file downloads, despite being the supposedly ideal driver to use with a local SSD on a Windows VM.
完成相同下载所需的时间仍然存在变化，但是很明显，SCSI在大文件下载方面通常不优于NVMe，尽管据称它是与Windows上本地SSD一起使用的理想驱动程序虚拟机。
Let’s also validate whether uploads are more performant via NVMe using the same averaging function running on 20 repeated uploads:
我们还使用在20次重复上传中运行的相同平均函数，验证通过NVMe上传的性能是否更高：
Measure-CommandAvg {gsutil cp temp_30GB_file gs://doit-speed-test-bucket/}# NVMe:# Avg: 3m23.216s, 151.17 MB/s# Min: 2m31.169s, 203.22 MB/s# Max: 4m13.943s, 121.42 MB/s# SCSI:# Avg: 5m1.570s, 101.87 MB/s# Min: 3m2.649s, 168.19 MB/s# Max: 35m3.276s, 14.61 MB/sWe see validation of our earlier individual runs that indicated NVMe might be more performant than SCSI for uploads. In this repeated twenty sample run case, NVMe is considerably more performant.
我们看到对我们较早的个人运行的验证表明，对于上传，NVMe可能比SCSI更有性能。 在这种重复的二十次样品运行的情况下，NVMe的性能要好得多。
Thus, with Windows VMs, in contrast to the GCP docs, not only should we avoid using -o GSUtil:parallel_composite_upload_threshold=150M with gsutil when uploading to GCS, we should also avoid using SCSI and prefer NVMe for our local SSD driver to improve uploads and maybe downloads. We also see that for downloads and uploads there are frequent, unpredictable pauses that range from 1–2 minutes to as much as 10–30 minutes.
因此，对于Windows VM，与GCP文档相比，我们不仅应避免在上载到GCS时避免将-o GSUtil:parallel_composite_upload_threshold=150M与gsutil一起使用，而且还应避免使用SCSI并首选NVMe作为本地SSD驱动程序来改善上载甚至下载。 我们还看到，对于下载和上传，会有频繁的，不可预测的暂停，范围从1-2分钟到长达10-30分钟。
At this point, I informed the customer that there are data transfer limitations inherent with using a Windows VM, however these could be partially mitigated by:
在这一点上，我告知客户，使用Windows VM存在固有的数据传输限制，但是可以通过以下方式部分缓解这些限制：
Leaving optional arguments to their defaults for gsutil cp large file uploads, in spite of the GCP documentation suggesting otherwise
尽管GCP文档建议采用其他方式，但将gsutil cp大文件上传的可选参数保留为默认值
By using s5cmd -c=1 -p=1000000 instead of gsutil for downloads
通过使用s5cmd -c=1 -p=1000000而不是gsutil进行下载
By using the NVMe instead of SCSI driver for local SSD storage to improve both upload and possibly download speeds, in spite of the GCP documentation suggesting otherwise尽管GCP文档表明存在其他问题，但通过使用NVMe而不是SCSI驱动程序进行本地SSD存储来提高上载和可能的下载速度However, I also informed the customer that uploads and downloads would be dramatically improved simply by avoiding Windows-based hangups entirely; move the data over to a Linux machine via disk snapshots, then perform data sync operations with GCS from a Linux-attached disk. That ultimately proved to be the quickest way to gain the expected throughput between a GCE VM and GCS, and led to a satisfied customer that was nonetheless left frustrated with nonsensical performance issues on their Windows Server.
但是，我还告知客户，仅通过完全避免基于Windows的挂断，即可极大地改善上载和下载。 通过磁盘快照将数据移到Linux机器上，然后从Linux附带的磁盘通过GCS执行数据同步操作。 最终证明这是在GCE VM和GCS之间获得预期吞吐量的最快方法，并导致了满意的客户，但他们的Windows Server上出现了毫无意义的性能问题，这使他们感到沮丧。
My takeaway from the experience was this: Not only is gsutil woefully unoptimized for operations on Windows servers, there appears to be an underlying issue with GCS’ ability to transfer data to and from Windows as delays and hangups exist within both gsutil and s5cmd for both download and upload operations.
从经验中我的收获是在这里：不仅是gsutil远远未优化的Windows服务器上操作，所以出现延误和挂断两个中存在与GCS的能力来传输数据和从Windows中的基本问题gsutil和s5cmd两个下载和上传操作。
The customer issue was solved…and yet my curiosity had not yet been sated. What bandwidth banditry might I find when trying to transfer a large number of small files instead of a small number of large files?
客户问题已解决……但我的好奇心尚未得到满足。 尝试传输大量小文件而不是少量大文件时，我会发现什么带宽匪徒？
Moving back to Linux, I split the large 30 GB file into 50K (well, 50,001) files:
回到Linux，我将30 GB的大文件拆分为50K(以及50,001)个文件：
mkdir partssplit -b 644245 temp_30GB_filemv x* parts/And proceeded to benchmark upload performance with gsutil:
并使用gsutil进行基准测试上传性能：
nohup bash -c 'time gsutil cp -r parts/* gs://doit-speed-test-bucket/smallparts/' &# n2-standard-4: 71m30.420s, 7.16 MB/s# n2-standard-80: 69m32.803s, 7.36 MB/snohup bash -c 'time gsutil -m cp -r parts/* gs://doit-speed-test-bucket/smallparts/' &# n2-standard-4: 9m7.045s, 56.16 MB/s# n2-standard-80: 3m41.081s, 138.95 MB/sAs expected, providing -m to engage in multi-threaded, parallel upload of files dramatically improves upload speed — do not attempt to upload a large folder of files without it. The more vCPUs your machine possesses, the more file uploads you can engage in simultaneously.
不出所料，提供-m来进行多线程并行文件上传会大大提高文件上传速度-不要尝试在没有文件的情况下上传大文件文件夹。 您的计算机拥有的vCPU越多，您可以同时进行的文件上传就越多。
Below are the download performance benchmarks with gsutil:
以下是gsutil的下载性能基准：
nohup bash -c 'time gsutil cp -r gs://doit-speed-test-bucket/smallparts/ parts/' &# n2-standard-4: 61m24.516s, 8.34 MB/s# n2-standard-80: 56m54.841s, 9.00 MB/snohup bash -c 'time gsutil -m cp -r gs://doit-speed-test-bucket/smallparts/ parts/' &# n2-standard-4: 7m42.249s, 66.46 MB/s# n2-standard-80: 3m38.421s, 140.65 MB/sAgain, providing -m is a must — do not attempt to download a large folder of files without it. As with uploads, gsutil performance is improved through parallel file uploads with -m and numerous available vCPUs.
同样，必须提供-m不要尝试在没有它的情况下下载大文件文件夹。 与上传一样，通过使用-m和大量可用vCPU进行并行文件上传，可以提高gsutil的性能。
I have found nothing out of the ordinary on Linux with gsutil-based mass small file downloads and uploads.
通过基于gsutil的大量小文件下载和上传，我在Linux上没有发现任何异常。
Having already established that s5cmd should not be used for file uploads to GCS, I will only report on the Linux download benchmarks below:
已经确定s5cmd不应用于文件上传到GCS，我将仅报告以下Linux下载基准：
nohup bash -c 'time s5cmd --endpoint-url https://storage.googleapis.com cp s3://doit-speed-test-bucket/smallparts/* parts/' &# n2-standard-4: 1m19.531s, 386.26 MB/s# n2-standard-80: 1m31.592s, 335.40 MB/snohup bash -c 'time s5cmd --endpoint-url https://storage.googleapis.com cp -c=80 s3://doit-speed-test-bucket/smallparts/* parts/' &# n2-standard-80: 1m29.837s, 341.95 MB/sOn the n2-standard-4 machine we see a 6.9X speedup in mass small file download speed when compared to gsutil. It makes sense then to use s5cmd for downloading many small files as well as for downloading larger files.
在n2-standard-4机器上，与gsutil相比，小型文件的下载速度提高了6.9倍。 然后使用s5cmd来下载许多小文件以及下载较大的文件就很有意义。
Nothing out of the ordinary was observed on Linux with s5cmd-based mass small file downloads.
在Linux中，基于s5cmd的大量小文件下载没有发现任何s5cmd 。
Given that s5cmd has significantly faster downloads than gsutil on any OS, I will only consider s5cmd for Windows download benchmarks with small files:
鉴于s5cmd在任何操作系统上的下载速度都比gsutil快得多，因此我将仅考虑s5cmd用于带有小文件的Windows下载基准测试：
Measure-CommandAvg {s5cmd --endpoint-url https://storage.googleapis.com cp s3://doit-speed-test-bucket/smallparts/* parts/}# NVMe:# Avg: 2m39.540s, 192.55 MB/s# Min: 2m35.323s, 197.78 MB/s# Max: 2m44.260s, 187.02 MB/s# SCSI: # Avg: 2m45.431s, 185.70 MB/s# Min: 2m40.785s, 191.06 MB/s# Max: 2m50.930s, 179.72 MB/sWe see that downloading 50K smaller files to a Windows VM performs better and more predictably than when downloading much larger files. NVMe outperforms SCSI only by a sliver.
我们看到，将5万个较小的文件下载到Windows VM的性能要比下载大得多的文件时更好，更可预测。 NVMe仅比SCSI优越。
There is an odd consistency and lack of long hang-ups with the data sync in this use case vs. the individual large file copy commands seen earlier. Just to fully verify that the hang-up trend is more likely to occur with large files, I ran the averaging function over 20 repeated downloads of the 30 GB file:
与之前看到的单个大文件复制命令相比，在这种用例中，数据同步存在奇特的一致性，并且没有长时间挂起。 只是为了完全验证大文件更容易出现挂机趋势，我对30 GB文件的20次重复下载运行了平均值函数：
Measure-CommandAvg {s5cmd --endpoint-url https://storage.googleapis.com cp -p=1000000 s3://doit-speed-test-bucket/temp_30GB_file .}### With 20 sample downloads# NVMe:# Avg: 3m3.770s, 167.17 MB/s# Min: 1m34.901s, 323.70 MB/s# Max: 10m34.575s, 48.41 MB/s# SCSI: # Avg: 2m20.131s, 219.22 MB/s# Min: 1m31.585s, 335.43 MB/s# Max: 3m43.215s, 137.63 MB/sWe see that the Windows download runtime on NVMe ranges from 1m37s to 10m35s, whereas with 50K small files on the same OS the download time only ranged from 2m35s to 2m44s. Thus, there appears to be a Windows or GCS-specific issue with large file transfers on a Windows VM.
我们看到，NVMe上的Windows下载运行时范围为1m37s至10m35s，而在同一OS上有50K小文件，下载时间仅为2m35s至2m44s。 因此，Windows VM上的大文件传输似乎存在Windows或GCS特定的问题。
Also note that the average NVMe download time appears to be about 73% longer (3m3s vs 1m46s) than when running s5cmd on Linux.
另请注意，与在Linux上运行s5cmd相比，平均NVMe下载时间似乎长了约73％(3m3s比1m46s)。
It is tempting to say that SCSI might be more advantageous for mass small file downloads than NVMe based on the results above, but with random hang-ups in the download process skewing the average, I’m going to stick with NVMe as the preferred driver given its proven effectiveness at uploading mass small files (see below) and its comparably equal performance at downloading large files as shown earlier.
诱人的是，根据以上结果，SCSI相对于NVMe而言，相比于NVMe而言，对于大量小文件下载而言可能更具优势，但由于下载过程中的随机挂起使平均值偏离，我将继续使用NVMe作为首选驱动程序鉴于它在上传大容量小文件(见下文)方面的行之有效的性能，以及在下载大文件方面的性能相当，如前所述。
Below are the metrics for uploading many small files from a Windows VM:
以下是从Windows VM上传许多小文件的指标：
Measure-CommandAvg {gsutil -q -m cp -r parts gs://doit-speed-test-bucket/smallparts/}# NVMe:# Avg: 16m36.562s, 30.83 MB/s# Min: 16m22.914s, 31.25 MB/s# Max: 17m0.299s, 30.11 MB/s# SCSI: # Avg: 17m29.591s, 29.27 MB/s# Min: 17m5.236s, 29.96 MB/s# Max: 18m3.469s, 28.35 MB/sWe see NVMe outperforms SCSI, and we continue to see that speeds are much slower than on a Linux machine. With Linux, mass small file uploads take about 9m7s, making the average Windows NVMe upload time of 16m36s about 82% slower than Linux.
我们看到NVMe胜过SCSI，并且我们继续看到速度比Linux计算机上慢得多。 使用Linux，小文件的大量上载大约需要9m7s，这使得Windows NVMe平均16m36s的上载时间比Linux慢82％。
If you are interested in performing your own benchmarks to replicate my findings, below are the shell and Powershell scripts I used along with comments summarizing the throughput I observed:
如果您有兴趣执行自己的基准测试以复制我的发现，则以下是我使用的Shell和Powershell脚本，以及概述我观察到的吞吐量的注释：
Linux benchmarking script
Linux基准测试脚本
Windows benchmarking script
Windows基准测试脚本
All in all, there is far more complexity than there should be in determining what the best method is for transferring data between GCE VMs — with data located on a local SSD — and GCS.
总而言之，要确定在GCE VM(具有位于本地SSD上的数据)和GCS之间传输数据的最佳方法，复杂度要比应有的复杂得多。
The performance differences between various GCE OSs and GCS are all related somehow, I just know it 我只是知道，各种GCE操作系统和GCS之间的性能差异都是相关的Windows servers experience drastic reductions in both download and upload speeds for reasons as-of-yet unknown when compared to the best equivalent commands on a Linux machine. These performance drops are substantial, typically 70–80% slower than the equivalent best command to run on Linux. Large file transfers are impacted more significantly than the transfer of many small files.
与Linux机器上的最佳等效命令相比，Windows Server的下载和上传速度都大大降低，原因尚不清楚。 这些性能下降是相当大的，通常比在Linux上运行的最佳命令要慢70-80％。 大文件传输比许多小文件传输受到的影响更大。
Thus, if you need to migrate TBs of data or particularly large files from Windows to GCS in a time-sensitive manner, you may want to bypass these performance issues by taking a disk snapshot, attaching a disk created from that snapshot to a Linux machine, and uploading the data via that OS.
因此，如果您需要以对时间敏感的方式将TB的数据或特别大的文件从Windows迁移到GCS，则可能希望通过制作磁盘快照并将从该快照创建的磁盘附加到Linux机器来绕过这些性能问题。 ，然后通过该OS上传数据。
Separate from Windows server issues, the default data transfer tool gsutil available on GCE VMs is inadequate for high-throughput downloads for any OS. By using s5cmd instead, you can achieve several-fold improvements in download speed.
与Windows服务器问题不同，GCE VM上可用的默认数据传输工具gsutil不适合任何操作系统的高吞吐量下载。 通过使用s5cmd ，您可以将下载速度提高几倍。
To help you sort through the myriad of tool and argument choices, below is a summary of my recommendations for maximizing throughput based on the benchmarks covered in this article:
为了帮助您整理各种工具和参数选择，以下是根据本文涵盖的基准我为使吞吐量最大化而建议的摘要：
Linux — Download
Linux —下载
Single large file: s5cmd --endpoint-url https://storage.googleapis.com cp s3://your_bucket/your_file .
单个大文件： s5cmd --endpoint-url https://storage.googleapis.com cp s3://your_bucket/your_file .
Multiple files, small or large: s5cmd --endpoint-url https://storage.googleapis.com cp s3://your_bucket/path* your_path/
大小不同的多个文件： s5cmd --endpoint-url https://storage.googleapis.com cp s3://your_bucket/path* your_path/
Linux — Upload
Linux —上传
Single large file: gsutil -o GSUtil:parallel_composite_upload_threshold=150M cp your_file gs://your_bucket/
单个大文件： gsutil -o GSUtil:parallel_composite_upload_threshold=150M cp your_file gs://your_bucket/
Multiple files, small or large: gsutil -m -o GSUtil:parallel_composite_upload_threshold=150M cp -r your_path/ gs://your_bucket/
大小不同的多个文件： gsutil -m -o GSUtil:parallel_composite_upload_threshold=150M cp -r your_path/ gs://your_bucket/
Windows Server— Download
Windows Server —下载
Use NVMe, not SCSI, for connecting a local SSD使用NVMe而非SCSI连接本地SSDSingle large file: s5cmd --endpoint-url https://storage.googleapis.com cp -c=1 -p=1000000 s3://your_bucket/your_file .
单个大文件： s5cmd --endpoint-url https://storage.googleapis.com cp -c=1 -p=1000000 s3://your_bucket/your_file .
Multiple files, small or large: s5cmd --endpoint-url https://storage.googleapis.com cp s3://your_bucket/path* your_path/
大小不同的多个文件： s5cmd --endpoint-url https://storage.googleapis.com cp s3://your_bucket/path* your_path/
Windows Server— Upload
Windows Server-上传
Use NVMe, not SCSI, for connecting a local SSD使用NVMe而非SCSI连接本地SSDSingle large file: gsutil cp your_file gs://your_bucket/
单个大文件： gsutil cp your_file gs://your_bucket/
Multiple files, small or large: gsutil -m cp -r your_path/ gs://your_bucket/your_path/
大小不同的多个文件： gsutil -m cp -r your_path/ gs://your_bucket/your_path/
翻译自: https://blog.doit-intl.com/optimize-data-transfer-between-compute-engine-and-cloud-storage-9a1ecd030e30
云计算中的海量数据存储在哪
