botocore

I’m personally thoroughly unimpressed by the bare metal S3 implementations I’ve tried. And there’s also an issue with client libraries and their compatibility with various implementations. I recently discovered this issue: https://github.com/boto/botocore/issues/3394 This is GCS implementing the S3 API incorrectly in a way that really ought not to break clients, but it’s still odd because the particular bug on GCS’s end seems like it took active effort to get wrong. But it’s also boto (the main library used in Python to access S3 and compatible services) doing something silly, tripping over GCS’s bug, and failing. And it’s AWS, who owns boto, freely admitting that they don’t intend to fix it, because boto isn’t actually intended to work with services that are merely compatible with S3. As icing on the cake, to report this bug to Google,

3 years ago, I implemented a library to do this for Python/boto3: https://github.com/pyauth/exile - because of assumptions that the sigv4 signers make in botocore, it actually has to do a bit of monkeypatching to get them to work. I tried to get the change merged upstream to make the botocore signers more flexible (https://github.com/boto/botocore/issues/1689) but my PR is still open.

In this case I just wanted to keep it simple and do it the "native" way. I've been hearing a lot about aioboto3 and I'll surely check it out. However, it seems to have some limitations like https://github.com/boto/botocore/issues/458

The source of truth for these partitions is not that code. It's the endpoints.json released in botocore (and replicated into every other sdk): https://github.com/boto/botocore/blob/a0e22400396eabd5a58e4b... To quickly list all of them, you can use a simple jq expression, such as the following: curl https://raw.githubusercontent.com/boto/botocore/a0e22400/botocore/data/endpoints.json | jq '.partitions[] | [.partitionName, .partition, .regions]'

The AWS SDKs I've dissected contain the information needed to talk to each (service, region) tuple as machine-readable data. They seem to converge at boto: https://github.com/boto/botocore/blob/master/botocore/data/e... Updates somehow land automatically, e.g. this commit adding the new region a few hours ago: https://github.com/boto/botocore/commit/f7dc4730ad34c6c3322d...

Maybe in a parallel universe where I have time for these kind of things... It shouldn't be too hard to reverse-engineer botocore, use the these JSONs here [1], and generate code based on them. [1] https://github.com/boto/botocore/tree/develop/services

Isn't the new boto though botocore (https://github.com/boto/botocore)? AWS CLI, for example, uses botocore, and it says it's the low-level foundation of boto 3.

This latest release marks a milestone in the transition from the old Java based tools, to the new Python ones. Mitch Garnaat[1] who built and maintained boto over the years was picked up by Amazon last year and since has been building out botocore[2] - which the aws-cli[3] tools use under the hood. [1] https://github.com/garnaat [2] https://github.com/boto/botocore [3] https://github.com/aws/aws-cli

Interestingly it's a wrapper around the community maintained botocore[1]. It's listed in the requirements file[2] 1: https://github.com/boto/botocore 2: https://github.com/aws/aws-cli/blob/develop/requirements.txt