s3gof3r

try https://github.com/rlmcpherson/s3gof3r you can often get wire speed

In S3 tests on c3.8xlarge instances, I've seen 8 Gbps throughput on both uploads and downloads using parallelized requests. Testing with iperf between two of the same instances maxed out about 8 Gbps as well so the throughput limitation is likely EC2 networking rather than S3. These tests were done over a year ago so bandwidth limitations on EC2 may have changed since. This testing was with https://github.com/rlmcpherson/s3gof3r

Excellent https://github.com/rlmcpherson/s3gof3r is my tool of choice for "fast, parallelized, pipelined streaming access to Amazon S3." If you want to saturate network bandwidth with S3 that's the one tool I know that can do it.

To be fair, here is a tool that has features no other tool I know: https://github.com/rlmcpherson/s3gof3r It can saturate your bandwidth and you can pipe to/from it. It's great to deal with hundreds GB's archive on the fly. "Examples: $ tar -cf - /foo_dir/ | gof3r put -b my_s3_bucket -k bar_dir/s3_object $ gof3r get -b my_s3_bucket -k bar_dir/s3_object | tar -x It's written in golang. So it's a simple binary.

I haven't used this library, but this appears to be what you want. https://github.com/rlmcpherson/s3gof3r Nice features include: > Retry Everything: All http requests and every part is retried on both uploads and downloads. > Configurable conncurrency > Uses an io.Writer (you could actually start posting to S3 before all of the data gets in on your side.) etc.

I missed your question about build instructions, sorry. I'll add them to the readme, but it's just "go get github.com/rlmcpherson/s3gof3r/gof3r" for the CLI once you have installed go. Usage instructions are in the readme already.

You don't need the content length of what you are uploading for multipart uploads, see (http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadIniti...). For each part of a multipart upload that is sent, however, you do need to include the content-length header, so that may be what you are referring to. These have a minimum size set by amazon of 5 MB. With the https://github.com/rlmcpherson/s3gof3r the memory overhead of a streaming upload is approximately part size * concurrent uploads (plus a couple of buffers in the pool). So, for example, if you configure the part size to 20 MB and set concurrent uploads to 10, that would be about 220 MB of memory usage.

Because of the lack of fast and reliable s3 clients out there, I built https://github.com/rlmcpherson/s3gof3r, that can do over 1 Gbps with ease, both multipart uploads and parallelized downloads. The killer feature, though, is streaming that enables things like gof3r get -b <bucket> -k <key> | tar -x to extract directories tarred directories or any other streaming application. It also provides end-to-end md5 integrity checking. For objects over a few GB, I haven't found anything matching in speed or reliability.