getting-started

Thanks for chipping in. I’ve been leaning towards this direction. I think I/O is the biggest part that in the case of plain code steps still needs fixing. Input being data/stream and parameterization/config and output being some sort of typed data/stream. My “let’s not reinvent the wheel” alarm is going off when I write that though. Examples that come to mind are text based (Unix / https://scale.com/blog/text-universal-interface) but also the Singer tap protocol (https://github.com/singer-io/getting-started/blob/master/doc...). And config obviously having many standard forms like ini, yaml, json, environment key value pairs and more. At the same time, text feels horribly inefficient as encoding for some of the data objects being passed around in these flows. More specialized and optimized binary formats come to mind (Arrow, HDF5, Protobuf

I'm not aware of any. I did just open this issue[0] in the Meltano project to open discussion with the team/community. It could be an interesting iteration on the Singer Spec[1] if we find that users are interested in it and it helps solve some bottleneck challenges. [0] https://gitlab.com/meltano/meltano/-/issues/2616 [1] https://github.com/singer-io/getting-started/blob/master/doc...

http://Singer.io is an open source ETL project written in Python. The components are small, composable programs that you can run independently, so you should be able to walk before your run. A good beginner project is to build an adapter to a new data source (known as a "tap" in Singer). Most taps pull data out of business tools like Salesforce or Marketo, but people also build them to pull characters from the Marvel API (https://www.stitchdata.com/blog/tapping-marvel-api/) Check out the getting started guide (https://github.com/singer-io/getting-started), or jump into the Singer Slack and ask for help (linked from the guide)

What's the motivation for the schemas? Are they just verifying that the APIs haven't changed and are still returning what you'd expect them to? I read through https://github.com/singer-io/getting-started/blob/master/SPE..., but I'm trying to better understand why they're necessary.