BOBoorishBears·Apr 9, 2017
We were using Cloudformation templates via Serverless since the ES domain was tightly integrated with the Lambdas we were deploying. Today we'd probably use SAM with the same CF templates (https://github.com/awslabs/serverless-application-model/blob...) to keep things simple.
We're pretty write heavy/low read with a comparatively small data set (~50GB expected over 4 months). I know ES can be tuned for that use case to improve index throughput at the cost of slower searching but I never tried setting it up on AES (I'm guessing it'd go in the CF-Template).
One challenge with elastic.co is there's no equivalent to Amazons ElasticsearchDestinationFirehose. We were putting data into a Firehose which batched, backed up to S3 and accounted for failures via a Lambda but elastic.co is giving you "raw" ES instances, so you need to handle those use
Read the rest
cases.
You could probably write a Kinesis stream that replicated that functionality(and from what I understand Firehose came to be because that's exactly what people were doing), but we control the devices uploading to the Lambda, so we had the devices act as the buffer (increased send thresholds, and propagated failures to send to ES to our devices which retry automatically) and sent a duplicate record to an S3DestinationFirehose for backup and replaying data