BAbanditelol·Dec 2, 2024
I think you want something aling the line of dvc (github.com/iterative/dvc)
In Demystifying Git Submodulestreeverse
dvc
Python2018–2024
treeverse / dvc
dvc
🦉 Data Versioning and ML Experiments
14Distinct people
15Mentions
0Explicit recommenders
16KGitHub stars
What to know before exploring
What it is
🦉 Data Versioning and ML Experiments
GitHub
Where it is used
No documented product use was verified in the last source check.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 1:49:46 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year15 dated mentions
Observed mentions: 2018-07-09 03:16:41 UTC to 2024-12-02 12:39:30 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2018 | 2018-01-01 00:00:00 UTC | 2019-01-01 00:00:00 UTC | 1 | 1 | 0 |
| 2019 | 2019-01-01 00:00:00 UTC | 2020-01-01 00:00:00 UTC | 4 | 2 | 2 |
| 2020 | 2020-01-01 00:00:00 UTC | 2021-01-01 00:00:00 UTC | 6 | 0 | 6 |
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 1 | 1 | 0 |
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 1 | 1 | 0 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 1 | 0 | 1 |
What Hacker News says
15 sourcesML Experiments Management with Git
112 points · Open discussionIt was definitely a bad choice. I wasn't there so I can only speculate. My guess is because it is sort of ubiquitous and thus a low-hanging fruit and devs didn't know better, or the related corollary, it's what S3 uses for ETags, so it probably seemed logical. Either way, seems like someone did it and didn't know better, no one agrees on a fix or whether it's even necessary to change, and thus it's stuck for now.
There's an ongoing discussion about replacing/configuring the hash function, and it looks like there might be some movement toward replacing the hash and other speedups in 3.0
https://github.com/iterative/dvc/issues/3069
> We not only want to switch to a different algorithm in 3.0, but to also provide better performance/ui/architecture/ecosystem for data management, and all of that while not seizing releases with new features (ex
Read the rest
periements, dvc machine, plots, etc) and bug fixes for 2.0, so we've been gradually rebuilding that and will likely be ready for 3.0 in the upcoming months. - https://github.com/iterative/dvc/issues/3069#issuecomment-93...
DVC: Git for Datasets and Models
1 points · Open discussionHey! From the top of my head.
1. Pick the project. Something that you like, but also something that is not huge or very mature (they tend to have longer turn around due to the volume of things they are getting). You can then check issue tracker for the project and see if there are tasks labeled "need help", "good first issue", etc. They are usually things that team would appreciate help with. On GH, it looks like this: https://github.com/iterative/dvc/labels/good%20first%20issue
If you don't know the project, try to search GH, may be here - https://github.com/explore ?
2. Check that project is "healthy" - check the recent history of PRs - that code is being reviewed and merged. It can be very demotivating to do stuff that won't be answered at all. Ask in the issue you like and see if there is a response.
3. Find the contributors guide. Th
Read the rest
ere should be a link in the project's README, something like https://github.com/iterative/dvc#contributing . The guide should help you understand how to do setup dev environment, how to create a PR, etc.
4. In a lot of cases these days, projects have chat or forum (or GH issue comments) to ask questions, ask help, etc. Check them our, see if they are "healthy" and don't hesitate to ping the team - usually people are responsive and are willing to help.
Good luck! and thank you for considering doing OSS contributions. That means a lot for them!
Congrats on the launch! This looks interesting, however I feel like this space is quite crowded. You mentioned that your most important feature is the fact that you are open-source, but off the top of my head I can think of several projects:
* Kubeflow: https://github.com/kubeflow/kubeflow
* MLFlow: https://github.com/mlflow/mlflow
* Pachyderm: https://github.com/pachyderm/pachyderm
* DVC: https://github.com/iterative/dvc
* Polyaxon: https://github.com/polyaxon/polyaxon
* Sacred: https://github.com/IDSIA/sacred
* pytorch-lightning + grid:
https://github.com/PyTorchLightning/pytorch-lightning
* DeterminedAI: https://github.com/determined-ai/determined
* Metaflow: https://github.com/Netflix/metaflow
* Aim: https://github.com/aimhubio/aim
* And so many more...
In addition to this list, several other hosted platform offer experiments trackin
Read the rest
g and model management. How do you compare to all of these tools, and why do you think users should move from one of them to use replicate, thank you.
Hey, one of the DVC maintainers here!
TL;DR: I think it should be compared with the upcoming DVC feature - https://github.com/iterative/dvc/wiki/Experiments . Stay tuned - it'll be released very soon but you can try it now in beta.
First of all, congrats on the launch! I do really like the aesthetics of the website, and the overall approach. It resonates with our vision and philosophy!
Good feedback on experiments feeling heavyweight! We've been focused on doing great foundation to manage data and pipelines in the previous DVC versions and were aware about this problem (https://github.com/iterative/dvc/issues/2799). As I mentioned - Experiments feature is already there in beta testing. It means that users don't have to do commits anymore until they are ready, still can share experiments (it's a long topic and we'll write a blog post at
Read the rest
some point since I really excited about the way it'll be implemented using custom Git refs), support for DL workflow (auto-checkpoints), and more. Would love to discuss and share any details, it would be great to compare the approaches.
Congratulations with the launch.
We've used https://github.com/iterative/dvc for a long time and quite happy. What's the main difference between replicate.ai and dvc?
In Launch HN: Replicate (YC W20) – Version control for machine learning"Ten Simple Rules for Reproducible Computational Research" http://www.ploscompbiol.org/article/info%3Adoi%2F10.1371%2Fj... :
> Rule 1: For Every Result, Keep Track of How It Was Produced
> Rule 2: Avoid Manual Data Manipulation Steps
> Rule 3: Archive the Exact Versions of All External Programs Used
> Rule 4: Version Control All Custom Scripts
> Rule 5: Record All Intermediate Results, When Possible in Standardized Formats
> Rule 6: For Analyses That Include Randomness, Note Underlying Random Seeds
> Rule 7: Always Store Raw Data behind Plots
> Rule 8: Generate Hierarchical Analysis Output, Allowing Layers of Increasing Detail to Be Inspected
> Rule 9: Connect Textual Statements to Underlying Results
> Rule 10: Provide Public Access to Scripts, Runs, and Results
... You can get a free DOI for and archive a tag of a Git repo with FigShare
Read the rest
or Zenodo.
... re: [Conda and] Docker container images https://news.ycombinator.com/item?id=24226604 :
> - repo2docker (and thus BinderHub) can build an up-to-date container from requirements.txt, environment.yml, install.R, postBuild and any of the other dependency specification formats supported by REES: Reproducible Execution Environment Standard; which may be helpful as Docker Hub images will soon be deleted if they're not retrieved at least once every 6 months (possibly with a GitHub Actions cron task)
BinderHub builds a container with the specified versions of software and installs a current version of Jupyter Notebook with repo2docker, and then launches an instance of that container in a cloud.
“Ten Simple Rules for Creating a Good Data Management Plan” http://journals.plos.org/ploscompbiol/article?id=10.1371/jou... :
> Rule 6: Present a Sound Data Storage and Preservation Strategy
> Rule 8: Describe How the Data Will Be Disseminated
... DVC: https://github.com/iterative/dvc
> Data Version Control or DVC is an open-source tool for data science and machine learning projects. Key features:
> - Simple command line Git-like experience. Does not require installing and maintaining any databases. Does not depend on any proprietary online services.
Management and versioning of datasets and machine learning models. Data is saved in S3, Google cloud, Azure, Alibaba cloud, SSH server, HDFS, or even local HDD RAID.
> - Makes projects reproducible and shareable; helping to answer questions about how a model was built.
There are a number of great solutions for storing and sharing datasets.
... "#LinkedReproducibility"
Sounds very similar to the OSS DVC project: https://github.com/iterative/dvc
In A group of ex-NSA and Amazon engineers are building a ‘GitHub for data’Thanks Tim! I definitely second your observation that there's room and reason for plenty of tools in this space. DVC probably belongs in your list too: https://github.com/iterative/dvc. Looking forward to checking out Open Images.
In Launch HN: Quilt (YC W16) – A versioned data portal for S3Data Version Control: Git for Data and Models
4 points · Open discussionDVC: Data and models versioning for ML projects
4 points · Open discussionGreat point! `dvc pull\push mydataset` actually exists in DVC. But `dvc tag` is missing and now we (DVC team) understand that it prevent DVC from a proper datasets and ML models tracking. It will be implemented soon.
There is an ongoing discussion in DVC GitHub about datasets tracking and tags https://github.com/iterative/dvc/issues/1487 and some discussions in DVC discord channel https://dvc.org/chat
In DVC – Open Source Machine Learning Version Control SystemData Version Control – Git for Data Scientists
2 points · Open discussion