HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
rom1504
img2dataset
Python2022–2025
rom1504 / img2dataset

img2dataset

Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.

10Distinct people
11Mentions
0Explicit recommenders
4.5KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 10:49:33 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year11 dated mentions

Observed mentions: 2022-10-03 00:23:54 UTC to 2025-08-30 01:35:54 UTC.

024682022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments20222023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 8 mentions; 2 submissions, 6 comments20232024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-01-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 2 mentions; 1 submissions, 1 comments2025

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC101
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC826
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC000
20252025-01-01 00:00:00 UTC2026-01-01 00:00:00 UTC211

What Hacker News says

8 sources
Both lawyers and the photographer are not looking great, but the legal questions seem much harder to answer than it might appear at first. Keep in mind that German copyright law is quite different from its British or American equivalent. Also, IANAL. If I understand it correctly: 0. Laion uses WAT-files from Common Crawl to generate a list of suitable image-text pairs, including a URL to the raw image. [0] 1. Laion downloads the raw images for further processing. 2. Based on step 2, Laion creates a curated list of image-text pairs. The images themselves are uncurated. 3. Using [1], anyone can download the images. Tags such as `noai` are respected. -- Question 1: Under German law, is Laion allowed to do what they are doing? Maybe, but far from clear. The lawyers [2] point to § 44b UrhG [3], which is rooted in [7], and § 66d UrhG [4]. These
Read the rest
norms state that you can use publicly available data for data mining, especially if you are doing it as part of scientific research. Copyright holders may prohibit such activities but must do it in machine readable form on the website. First, I am not positive that Laion's activities fall under the law's definition of "data mining". The law says data mining is an activity done in order to gain insight into "patterns, trends and correlations". But Laion does not even attempt to gain any insight from the data, it just wants to provide a data set for model building [0]. I don't think § 44b UrhG is applicable at all. Second, Laion cannot simply yell "it's for research". To use of § 66d UrhG, you (a) must qualify as a research organisation, and (b) there must not be any private organisation in the background directing the research. Whether or not Laion qualifies as such would have to be decided by the courts, but I think it's questionable. Overall I think the lawyers' position as outlined in [2] is not very strong. Perhaps they find better arguments when a court is involved. -- Question 2: Under German law, does the photographer have any claim against Laion? Difficult. The photographer asked Laion to remove his work from their dataset [5]. The lawyers replied that Laion does not store any photographies so there is nothing to remove. It seems that the photographer did not understand this correctly when writing the initial cease-and-desist letter. Does Laion infringe on the photographer's copyright if they mereley distribute links? The photographer argues that Laion admittedly does download raw images themselves, which pertains to Question 1. Aside from this, in German law there is a theory called "Störerhaftung" [6]. It means that if what you are doing enables copyright infringement you may be liable as well, even if you do not distribute the content yourself. If you get notified that you participate in any such infringement by a copyright holder you must cease and desist immediately. In some cases, distributors of hyperlinks to copyrighted material have been found to commit copyright infringement, though it has been rather the exception. I have no idea how a court would decide in the current case. One may argue that simply adding the equivalent of a `robots.txt` will suffice to end the infringement, but that is too simple. The photographer sells his work to a customer who may use it on their website, but does not want anyone else to use his work. However, the photographer cannot control whether or not the customer uses a `noai` on the website. [3] is opt-out, but for this to work, the photographer must have the opportunity to opt-out. It seems the law did not sufficiently appreciate this scenario. It also seems it did not foresee instances where content mining enables AI models replicating (parts of) the content they did mine. One plausible scenario is that the court denies the photographer's claims to Laion because he can contractually mandate clients to place a `noai` tag on their website. In any case, I think the photographer would have to demonstrate how specifically Laion cost him money as there are no punitive damages in German law. [0] https://laion.ai/blog/laion-5b/ [1] https://github.com/rom1504/img2dataset [2] https://www.alltageinesfotoproduzenten.de/2023/04/24/laion-e... [3] https://www.gesetze-im-internet.de/urhg/__44b.html [4] https://www.gesetze-im-internet.de/urhg/__60d.html [5] https://www.alltageinesfotoproduzenten.de/2023/02/20/laion-v... [6] https://de.wikipedia.org/wiki/St%C3%B6rerhaftung [7] https://en.wikipedia.org/wiki/Directive_on_Copyright_in_the_...
In AI used photographer’s photos for training, then slapped him with an invoice
Ethics are subjective, and debatable. I'm happy to discuss further, but "citation needed" makes it sound like you think I must prove to you with some facts of the validity of the position. That's not the conversation we need to have. I don't think you can take a general summary of this situation, round off edges and nuance, and then say that I made the argument that this DIFFERENT scenario is "inherently unethical". Ethics are nuanced and highly specific to the context, and the framing. A single word, phrasing, emphasis (or lack thereof) can change the scenario from an ethical to an unethical one, and vice versa. For example, here is how I would phrase the exact same content in a way that i believe WOULD be ethical: ``` //First sentence unchanged Websites can pass the http headers `X-Robots-Tag: noai`, `X-Robots-Tag: noindex` , `X-Robots-
Read the rest
Tag: noimageai` and `X-Robots-Tag: noimageindex` By default img2dataset will ignore images with such headers. //Then pull up the policy link first To understand why image creators and artists may choose to specify such headers for their images, and choose to actively not consent to their images being collected, see [AI use impact](https://github.com/rom1504/img2dataset#ai-use-impact). AI training would not be possible without the contribution of artists, and it is the recommendation of this tool's authors that you should respect their communicated wishes. However, if you have a legitimate reason to bypass these headers, to disable this behavior and download all images, you may pass --disallowed_header_directives '[]' ```
In Please make this tool “opt-in” by default
The established norm is that scrapers have to download robots.txt and support the standard robots.txt features, notably including `Crawl-Delay` which sets a rate limit. This is the established standard by which websites tell scrapers what the rules are for scraping them. This tool is scraping sites, it has webmasters reporting actual disruption, it doesn't have robots.txt support. When people complained (eg in https://github.com/rom1504/img2dataset/issues/48), the author's stance was basically "PRs welcome". It looks like a third party recently contributed a PR to make it respect robots.txt (https://github.com/rom1504/img2dataset/pull/302), albeit without `Crawl-Delay` support, which is not merged yet. I have seen the same thing with other recent AI tools (eg https://github.com/m1guelpf/browser-agent/issues/2) and I think it's important t
Read the rest
o defend the robots.txt convention and nip this in the bud. If a bot doesn't make a reasonable effort to respect robots.txt and it causes disruption, it's a denial-of-service attack and should be treated as such. No excuses.
In An AI Scraping Tool Is Overwhelming Websites with Traffic
While this may seem quaint, let's review the github package's README: ``` Opt-out directives Websites can pass the http headers `X-Robots-Tag: noai`, `X-Robots-Tag: noindex` , `X-Robots-Tag: noimageai` and `X-Robots-Tag: noimageindex` By default img2dataset will ignore images with such headers. To disable this behavior and download all images, you may pass --disallowed_header_directives '[]' See [AI use impact](https://github.com/rom1504/img2dataset#ai-use-impact) to understand better why you may decide to enable or disable this feature. ``` This does not pass the minimum thresholds of ethics for me. It may be legal, but it's not ethical. It's good that the default behaviour respects the headers. It's fine that there is an opt-out (if you didn't offer one, someone would just clone your code and offer it). But if the most important informa
Read the rest
tion you provide is "here's how to use my tool unethically" rather than "here's why the default ethical use is important, though if you really have a good use case, here you go", then you're complicit in unethical use of this tool.
In Please make this tool “opt-in” by default

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.