Limiting bandwidth of the of Amazon aws s3 uploads

Question

I'm currently having daily backups of my files on uploaded to S3 bucket using aws s3 mv command, i.e.:

aws s3 mv $webroot/backups/db/ s3://my-backups/$date/db

Everything works, except that recently my host (MediaTemple) started automatically disabling my server, because, apparently, those uploads started hitting some traffic bytes per second limits that are in place on all their servers.

Is there some way to limit the bandwidth of the aws s3 mv using parameters or aws configuration options?

I understand there are some 3rd party Linux utilities that can do that. I.e. throttle or trickle. However, I'd like to avoid additional software, if there's a built-in way to do that with Amazon's own tools.

score 3 · Accepted Answer · answered Dec 09 '15 at 09:39

3

The mv command has no such option to throttle bandwidth. You can see the full options list here:

http://docs.aws.amazon.com/cli/latest/reference/s3/mv.html

There are some configuration values you can customize to control S3 transfers commands (cp, sync, mv, and rm).

For example, setting the value of max_concurrent_requests to a value lower than 10 (which is the default), will make it less resource intensive. Have a look at the link below and try to adjust the values and see if it helps.

See details here: AWS CLI S3 Configuration

answered Dec 09 '15 at 09:39

Diamond

9,131

Thanks for the tip. I'll try reducing max_concurrent_requests and see if that helps. I'm not uploading a lot of files, just a few really large ones. So I'm not sure if the S3's multipart chunked uploads are considered "concurrent requests" or not. – martynasma Dec 09 '15 at 19:53
1

Looks like reducing max_concurrent_requests worked. Thanks! – martynasma Dec 10 '15 at 06:02
Glad that it worked! – Diamond Dec 10 '15 at 09:54
3

Per that article there is now a "max_bandwidth" option available as well. – dragon788 Oct 15 '18 at 22:25

Limiting bandwidth of the of Amazon aws s3 uploads

1 Answers1