bacalhau

When we were doing a bunch of customer examples for Bacalhau[1], we needed to create a fake NCSA compatible log generator[2]. We also containerized it to make it even easier[3]. E.g. docker run -v ./logs:/var/log/app ghcr.io/bacalhau-project/access-log-generator:latest Absolutely no business model behind this - just Apache2/MIT. If you like it, just use it! If you don't, happy to tweak it! [1] https://github.com/bacalhau-project/bacalhau [2] https://github.com/bacalhau-project/examples/tree/main/utili... [3] https://github.com/orgs/bacalhau-project/packages/container/...

hosting your blog on substack ... for those who thought "you know what? medium.com isn't nearly annoying enough. let's turn it up to 11" https://github.com/bacalhau-project/bacalhau#readme (Apache 2) "Compute over Data framework for public, transparent, and optionally verifiable computation"

This is really interesting - we’ve tried really hard to solve some of these with Bacalhau[1] - a much simpler distributed compute platform. Would love your feedback! [1] https://github.com/bacalhau-project/bacalhau Disclosure: I confounded Bacalhau

It's an interesting question! The problem is that the data is spread everywhere - no choice about that. So with that in mind, how do you query that data? Today, the idea is that you HAVE to put it into a central location. With tools like Bacalhau[1] and DuckDB [2], you no longer have to - a single query can be sharded amongst all your data - EFFECTIVELY giving you a lot of what you want from a data lake. It's not a replacement, but if you can do a few of these items WITHOUT moving the data, you will be able to see really significant cost and time savings. Disclosure: I co-founded Bacalhau [1] https://github.com/bacalhau-project/bacalhau [2] https://github.com/duckdb/duckdb

FULLY BIASED COMMENT: We (the bacalhau.org[0] project) are interested in helping with this - one of our philosophies has been that part of the problem is with that first step. By first moving to a lake of some kind, you end up giving up lots of optionality. Even basic things like aggregation, filtering, windowing, etc now need to be in the "locked-in" tool, which is exactly the wrong first step to take. SHOW HN: We have a solution that uses DuckDB to do some of this initial work[1] which can save you 70%+ or more on total data throughput. Further, it allows you to do interesting things like eventing, multi-homing observability data, etc. I'd be very interested to hear any/all thoughts! [0] https://github.com/bacalhau-project/bacalhau [1] https://blog.bacalhau.org/p/bacalhau-x-duckdb-deploying-appl... Disclosure: I co-founded the Bacalhau

Hey everyone! I'm David Aronchick - co-founder of Bacalhau[1] - and we are specifically building an open source platform to help with this. Basically, we allow you to do compute over data, making it much easier to schedule jobs (container, WASM, arbitrary binary) to where the data is being stored, even over irregular/unreliable networks. We were able to show 90%+ savings in things like log aggregations. Happy to answer any questions if you'd like! David Aronchick aronchick@expanso.io [1] https://github.com/bacalhau-project/bacalhau

Hi everyone! We're super proud to announce the 1.0 release of the OSS Bacalhau platform[1] Just a few of the features we released include: - Multi-architecture support - Intel, Apple Silicon (M1/M2), ARMv6 & ARMv7, AMD64 - Running 10k+ jobs simultaneously - Simplified private cluster setup - DAG execution through Project Amplify - Insulated Jobs & Federated Learning And lots more! Please let us know if you have any questions :) Disclosure: I am co-founder of Bacalhau [1] https://github.com/bacalhau-project/bacalhau

Very selfishly - distributed compute. Not decentralized, distributed. Compute and data are being created and run everywhere, we need platforms that understand how to use it and get insights without (or before) moving it. Our contribution: https://github.com/bacalhau-project/bacalhau (think Kubernetes but built for the distributed world). Disclosure: I co-founded the Bacalhau Project

Links off to the repo: Waterlily.ai: https://github.com/bacalhau-project/Waterlily Bacalhau: https://github.com/bacalhau-project/bacalhau FVM: https://github.com/filecoin-project/ref-fvm

Hi! I'm David Aronchick - co-founder for Bacalhau (the distributed compute framework that runs Waterlily.ai - https://github.com/bacalhau-project/bacalhau). We'd love your thoughts on our project - if you have any questions, please ask away!

Thank you so much for the feedback. Yeah, we REALLY do want to figure out a better naming/reference scheme. Do you have anything you've seen you really like? Disclosure: I work on Bacalhau[1] https://github.com/bacalhau-project/bacalhau

OH and we do do some streaming right now on Twitch - follow our Twitter account [1] and star our GitHub repo [2] to get notified when these go live! [1] https://twitter.com/BacalhauProject [2] https://github.com/bacalhau-project/bacalhau

I'm David Aronchick - first non-founding PM on Kubernetes, co-founder of Kubeflow [1], and co-founder of the SAME project [2] - and we've spent the past year working on Bacalhau [3], an open source project to bring compute to data. We've recently opened up a public-hosted cluster (all runnable from colab in our docs [4]) and would love your feedback - you can see our vision at the attached blog post. Thanks! Disclosure: I co-founded Bacalhau :) [1] https://kubeflow.org/ [2] https://sameproject.ml/ [3] https://github.com/bacalhau-project/bacalhau/ [4] https://docs.bacalhau.org/