TOtodsacerdoti·May 27, 2020Submission
Dat project renamed Hypercore, CLI looking for new maintainer
2 points · Open discussiondat-ecosystem
dat
JavaScript2013–2020
dat-ecosystem / dat
dat
:floppy_disk: peer-to-peer sharing & live syncronization of files via command line
19Distinct people
19Mentions
0Explicit recommenders
8.2KGitHub stars
What to know before exploring
What it is
:floppy_disk: peer-to-peer sharing & live syncronization of files via command line
GitHub
Where it is used
No documented product use was verified in the last source check.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 1:49:46 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year19 dated mentions
Observed mentions: 2013-06-29 21:13:00 UTC to 2020-05-27 13:10:24 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2013 | 2013-01-01 00:00:00 UTC | 2014-01-01 00:00:00 UTC | 4 | 3 | 1 |
| 2014 | 2014-01-01 00:00:00 UTC | 2015-01-01 00:00:00 UTC | 4 | 1 | 3 |
| 2015 | 2015-01-01 00:00:00 UTC | 2016-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2016 | 2016-01-01 00:00:00 UTC | 2017-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2017 | 2017-01-01 00:00:00 UTC | 2018-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2018 | 2018-01-01 00:00:00 UTC | 2019-01-01 00:00:00 UTC | 3 | 0 | 3 |
| 2019 | 2019-01-01 00:00:00 UTC | 2020-01-01 00:00:00 UTC | 2 | 0 | 2 |
| 2020 | 2020-01-01 00:00:00 UTC | 2021-01-01 00:00:00 UTC | 5 | 1 | 4 |
What Hacker News says
19 sourcesi swung and missed :-( https://github.com/datproject/dat/issues/1008
In Dat: A P2P hypermedia protocol with public-key-addressed file archives> we were part of a flock of 5 or 10 similar projects and I'm pretty sure the only one still around today is Kaggle.
Dat is still around, on version 14, last update in git was 5 days ago.
https://github.com/datproject/dat
https://dat.foundation/
In Dolt is Git for dataThere is actually an old git for data project too:
https://github.com/datproject/dat
It's ~5 years old and I really wanted it to be huge. Hoping this new project is a success. Especially since I notice I went to high school with one of the founders of Dolt (Hey Tim!)
In Dolt is Git for dataI haven't seen it mentioned yet so don't forget about the dat project: https://github.com/datproject/dat
In A group of ex-NSA and Amazon engineers are building a ‘GitHub for data’This seems very similar to Borg Backup [1], so I'm interested to hear from others who have used both on how it compares.
More generally, I've been looking for a solution that helps distribute backups in a peer-to-peer way. I have a few friends with their own home servers, and we want to replicate backups across each other's servers for geographical redundancy. Currently, I have a script that uses rsync to copy some tar archives over daily, but this doesn't scale well as more peers want to join our backup-sharing group, since it requires them granting me SSH access.
What I need is a decentralized network to share and retrieve backups from peers. I tried using dat [2] with a Borg Backup repository inside it, but ran into some nasty issues with dat which would cause it to regularly crash and one time even corrupt the data.
Does anyone have a
Read the rest
ny suggestions for such a situation?
[1] https://www.borgbackup.org/
[2] https://github.com/datproject/dat
Similar sharing, but not onion routed.
https://github.com/datproject/dat
In OnionShare 2 adds anonymous dropboxesAnyone have any views on if this is materially different to Dat for the same challenge? https://github.com/datproject/dat
In Dragonfly: an intelligent P2P based file distribution systemThere are so many tools that can transfer files between two computers. I really like ones like this because you don't have to have SSH access or forward any ports to send a file from A to B. Its similar vein to other peer-to-peer utilities like zget [1], sharedrop [2], instant.io (webtorrent) [3], filepizza (webtorrent) [4], magic-wormhole [5], toss [6], dat [7], and many many others.
During Hacktoberfest I also started my own, written in Go, so I could have my friends can use it without installing a Python ecosystem [8].
[1]: https://github.com/nils-werner/zget
[2]: https://github.com/cowbell/sharedrop
[3]: https://github.com/webtorrent/instant.io
[4]: https://github.com/kern/filepizza
[5]: https://github.com/warner/magic-wormhole
[6]: https://github.com/zerotier/toss
[7]: https://github.com/datproject/dat
[8]: https://github.com/schollz
Read the rest
It's been almost 5 years at this point somehow!
Check out the first commit: https://github.com/datproject/dat/tree/464679267049899eafa34...
Bit more on the funding history, etc. : https://blog.datproject.org/2017/09/15/dat-funding-history/
In How do dat:// sites interact with servers?I really want a content-addressed storage system with differential compression to solve this problem.
I was browsing through dat for awhile, but haven't caught up with it lately:
https://github.com/maxogden/dat
Basically disk is so cheap that you should just keep 2 or 3 copies of your data around. And then you can sync them really quickly and do the processing on any one of N machines.
In Scalability, but at what cost? (2015)Here's a good explanation of what the software tries to achieve, from one of the files in the Github repo. Much more informative than the Wired article:
Here's a concrete example: A police department in a city hosts an Excel spreadsheet on their web server called Crime-2013.xls. It contains all of the reported crime so far this year and gets updated every night at midnight with all of the new crimes that were reported each day.
Say you wanted to write a web application that showed all of the crime on a map. To download the new data every night you'd have to write a custom program that downloads the .xls file every night at midnight and imports it into your application's MySQL database.
To get the fresh data imported you can simply delete your entire local crime database and re-import all rows from the new .xls file, a process known as a '
Read the rest
kill and fill'.
But the kill and fill method isn't very robust, for a variety of messy reasons. For instance, what if you cleaned up some of the rows in the crime data in your local database after importing it last time? Your edits would get lost.
Another option is a manual merge, where you try and import each and every row of the incoming Excel file one at a time. If the data in the row already exists in the database, skip it. If the row already exists but the incoming data is a new version, overwrite that row. If the row doesn't exist yet, make a whole new row in the database.
The manual merge can be tricky to implement. In your import script you will have to write the logic for how to check if an incoming row already exists in your database. Does the Excel file have its own Crime IDs that you can use to look up existing records, or are you searching for the existing record by other method? Do you assume that the incoming row should completely overwrite the existing row, or do you try to do a full row merge?
At this point the import script is probably a few dozen lines and is very specific to both the police department's data as well as your application's database. If you decide to switch from MySQL to PostgreSQL in the future you will have to revisit this script and re-write major parts of it.
If you have to do things like clean up formatting errors in the Police data, re-project geographic coordinates, or change the data in other ways there is no straightforward way to share those changes publicly. The best case scenario is that you put your import script on GitHub and name it something like 'City-Police-Crime-MySQL-Import' so that other developers that want to consume the crime data in your city won't have to go through all the work that you just went through.
Sadly, this workflow is the state of the art. Open data tools are at a level comparable to source code management before version control.
https://github.com/maxogden/dat/blob/master/docs/what-is-dat...
Announcing the dat stable alpha
3 points · Open discussionYeah, they're hidden away in their git repo [1]. They have a good set of example usage commands [2], and a nice introductory document explaining what dat is [3]. Perhaps most important are the technical notes [4] that describe the supported formats.
[1]: https://github.com/maxogden/dat
[2]: https://github.com/maxogden/dat/blob/master/usage.md
[3]: https://github.com/maxogden/dat/blob/master/what-is-dat.md
[4]: https://github.com/maxogden/dat/blob/master/notes.md
In Dat – A git-like tool for large datasetsThere is another readme that is more informative:
https://github.com/maxogden/dat/blob/master/what-is-dat.md
In Dat – A git-like tool for large datasetsShowing the 15 most recent of 19 sources. Enable JavaScript to search and browse every indexed mention.