quilt

> What other platforms are there for versioning data and recording data provenance? Quilt also versions data and data pipelines: https://medium.com/pytorch/how-to-iterate-faster-in-machine-... https://github.com/quiltdata/quilt (Python)

I think a Gitlab for data is a much better proposition . See https://quiltdata.com/ It is also open source and written in python. https://github.com/quiltdata/quilt

Can do. A lot of the magic happens in the es/indexer and search lambdas here: https://github.com/quiltdata/quilt/tree/master/lambdas. The short of what we do: we listen for bucket notifications in Lambda, open the object metadata and send it, along with a snippet of the file contents, to ElasticSearch for indexing. ElasticSearch mappings are a bit of a bear and we had to lock those down to get them to behave well. What are the big barriers you're bumping into on the data management and portability side of things?

Hi. R is really important to us and we want to add support (probably through SparklyR). If you email me I can add you to our Slack channel and we can talk through extending Quilt (aneesh at quiltdata dot io). There is a sliver in our docs but it's not complete: https://docs.quiltdata.com/basics.html The hooks for extending build targets are here: https://github.com/quiltdata/quilt/blob/4aa6897f9e33349b7778... Again it's a bit raw but we're here to help make R support easy.

There's a RESTful API that's used by the Python client, though it's not documented yet. If you're willing to go through the code, you can find /api/... calls in https://github.com/quiltdata/quilt/blob/master/quilt/tools/c... . We're planning to write proper documentation soon.

Author here. Questions about data engineering, versioning data, or why you shouldn't use git to store data? :) Ask away. You can also file feature requests or PRs against our GitHub repo https://github.com/quiltdata/quilt