datastation

There was a technique that existed already where you could use `go test -cover` and the `-o` flag to produce a binary from `go test` rather than actually running tests. So you could build a binary that had coverage enabled. Then when you exercised code paths by running that binary with whatever inputs, coverage would be collected and dumped into some local file. Here's an example of the build step: https://github.com/multiprocessio/datastation/blob/main/runn.... And here's how I'd analyze coverage after using the binary: https://github.com/multiprocessio/datastation/blob/main/runn.... This analysis actually combined both unit test coverage files with integration test coverage so it gave a pretty good overall indication of code paths covered by any tests. I can't remember where I found this technique but it's been around for a while. This

You can do this for small datasets and I built an app around exactly this use case [0]. As your data gets bigger (just over time even if individual days don't emit more data) you'll end up building out partitioning schemes yourself or you'll move to a system that does that for you like bigquery/snowflake/etc. [0] https://github.com/multiprocessio/datastation

I use Electron for DataStation [0]. Electron has builtin message passing between the UI layer and the main process. I built a small wrapper around this in TypeScript so I could have the messages be typechecked. Within the main process I actually have it call a Go process to handle all the heavy lifting that goes on in DataStation like database queries, HTTP requests, etc. Communication between the Nodejs main process and the Go process just happens in temp files. All long-term data is stored in SQLite. Happy to talk more about it if you want! My email is in my profile. [0] https://github.com/multiprocessio/datastation

I build a somewhat similar app, DataStation [0], that is in JavaScript and Go. It supports scripting in Python, Julia, R, JavaScript, Ruby, etc. The server version of it exists and I run it myself but that process is not documented yet. (Most people use it as a desktop app today.) [0] https://github.com/multiprocessio/datastation

My biggest issue with Pyodide (which is of course an awesome project/build on its own) is the long wait times. I haven't figured out a way around a ~5 second load time where the entire UI hangs every single time you load the page. My app (similar to Simon's, a lite mode of a data IDE): https://app.datastation.multiprocess.io. My code: https://github.com/multiprocessio/datastation/blob/main/shar....

I use Go heavily cross-platform developing DataStation [0] and dsq [1]. I am not an expert. And I don't have proof for it but on some rudimentary benchmarks the Linux-specific file idioms in the Go standard library definitely don't seem to translate well to even macOS let alone Windows. For example some good streaming techniques for reading large files on Linux that work really well there seemed to be pretty bad on macOS. I think Amos has presented more proof than I can on the topic of just how Linux-influenced Go is. And I think it is fine for the majority of Go users because the majority users of Go are building server apps or Linux CLIs. Amos has spent some time building cross-platform desktop systems with Go for itch.io and I think I'm seeing some of the same things they are in that scenario. I think this is a reasonable article. I di

Thanks for including dsq! Its focus is always going to be flexibility and UX over performance since it's just a thin wrapper over the internals of a GUI data integration tool [0]. For example many other tools operate on only one or two file types and have optimized for querying on those types. Some like octosql and spyql IIRC have implemented their own sql parsing layer and engine. So if full sql support is necessary you may not be able to use them. In contrast dsq supports excel, avro, parquet, json, csv/tsv, nginx logs, apache logs, json newline, concatenated json, etc. And this list keeps growing. Dsq stores data in sqlite under the hood so you have access to the full sqlite dialect. Textql and q also do it this way. That said, despite performance not being the major I goal I think it will get better over time. There's already an OSS c

I don't sign the build at all for Windows or Mac for DataStation [0]. I'm planning to sign the mac build because it's cheap and not insane. Windows signing is expensive and insane. [0] https://github.com/multiprocessio/datastation

This looks awesome! I've already played around with pyodide and coldbrew doing the same thing for CPython. I use it for an in-memory playground [0] of an open-source desktop app I build [1]. I've been waiting for Ruby, Julia, and R support to add them in too. That said, I am not seeing a link in here about how to actually use this code. Is there a good tutorial/example somewhere? [0] app.datastation.multiprocess.io [1] github.com/multiprocessio/datastation

I've got some good first projects if you're interested in OSS data tools and have some Go experience. Check out: https://github.com/multiprocessio/datastation/blob/main/GOOD...

I build a desktop data IDE that helps you query files, APIs, databases; script the results; and graph/export. It was important to me to be desktop-first so that you can try it out at work without needing to go through the rigmarole required of a SaaS data analysis product. It'd be as hard to get permission for as installing Sublime or DBeaver. If I got funding or this becomes sustainable I'd be interesting in building native versions. https://github.com/multiprocessio/datastation

When I worked as an employee I contributed mostly just with docs fixes or bug reports (and not that often). I think it's mostly a mindset of seeing an issue yourself and noticing that others probably also are confused or misled by something so you go and report it! Now that I'm working full-time on open-source and my own company I can contribute more easily to projects like upgrading SQLite source in a few bindings libraries [0], [1], [2] when 3.38 came out. If anyone is interested in contributing to open-source and wants a bit more guidance though I have a number of good "first timer" projects related to data tooling for [3]. Only expectation is that you have some experience with Go. Join discord.multiprocess.io, go to the #dev channel and say hi! [0] https://github.com/mattn/go-sqlite3/pull/1019 [1] https://github.com/mapbox/node-sqlite

In addition to jq (already mentioned) here are some other useful CLI tools for dealing with JSON data. Descriptions are directly from GitHub. Note: dsq also has a companion GUI app called DataStation[0] if you're looking for that. fx: Command-line JSON processing tool - https://github.com/antonmedv/fx dasel: JSON, YAML, TOML, XML, and CSV query and modification tool - https://github.com/TomWright/dasel dsq: CLI tool for running SQL queries against JSON, CSV, Excel, Parquet, and more - https://github.com/multiprocessio/dsq gron: Make JSON greppable - https://github.com/tomnomnom/gron jello: Filter JSON and JSON Lines data with Python syntax - https://github.com/kellyjonbrazil/jello jless: Command-line pager for JSON data - https://github.com/PaulJuliusMartinez/jless jid: Json incremental digger - https://github.com/simeji/jid jql: JSON que

I've got a lot of good first projects for github.com/multiprocessio/datastation and github.com/multiprocessio/dsq. If you have any experience in Go it will be easy to get you up and running. I've already had a few students and self-learners start contributing. Join #dev on discord.multiprocess.io and say hi and I'll show you some of the stuff you can help with.