HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
franzenzenhofer
robotstxt
HISTORICAL RECORD2011–2020
franzenzenhofer / robotstxt · Not publicly available

robotstxt

Discover this repository through its Hacker News conversations.

Languages & CompilersRobotics & IoT
Explore on GitHubRead Hacker News discussions
2Distinct people
3Mentions
0Explicit recommenders
REPOSITORY AT A GLANCE

What to know before exploring

What it is

A public description has not been verified.

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year3 dated mentions

Observed mentions: 2011-05-26 10:25:20 UTC to 2020-05-15 12:00:37 UTC.

0122011-01-01 00:00:00 UTC to 2012-01-01 00:00:00 UTC (exclusive): 2 mentions; 1 submissions, 1 comments20112012-01-01 00:00:00 UTC to 2013-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2013-01-01 00:00:00 UTC to 2014-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2014-01-01 00:00:00 UTC to 2015-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2015-01-01 00:00:00 UTC to 2016-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments20152016-01-01 00:00:00 UTC to 2017-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2017-01-01 00:00:00 UTC to 2018-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2018-01-01 00:00:00 UTC to 2019-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-01-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2020

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20112011-01-01 00:00:00 UTC2012-01-01 00:00:00 UTC211
20122012-01-01 00:00:00 UTC2013-01-01 00:00:00 UTC000
20132013-01-01 00:00:00 UTC2014-01-01 00:00:00 UTC000
20142014-01-01 00:00:00 UTC2015-01-01 00:00:00 UTC000
20152015-01-01 00:00:00 UTC2016-01-01 00:00:00 UTC000
20162016-01-01 00:00:00 UTC2017-01-01 00:00:00 UTC000
20172017-01-01 00:00:00 UTC2018-01-01 00:00:00 UTC000
20182018-01-01 00:00:00 UTC2019-01-01 00:00:00 UTC000
20192019-01-01 00:00:00 UTC2020-01-01 00:00:00 UTC000
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC101

What Hacker News says

3 sources
franze·May 15, 2020
I coded a few robots.txt parsers i.e.: (these are just the ones i cared to publish) https://github.com/franzenzenhofer/simple-functional-robots-... https://github.com/franzenzenhofer/robotstxt and even though robots.txt seems like a very, very simple text based protocoll there are unanswered mysteries the biggest mystery is user agent groups and comments i.e.: User-agent: googlebot User-agent: bing User-agent: yandex Disallow: / so i am disallowing everything for google, bing, yandex; easy enough, but: User-agent: googlebot User-agent: yandex Disallow: / means that googlebot has no instructions, but yandex is disallowed all. if a whole line is commented out User-agent: googlebot #User-agent: bing User-agent: yandex Disallow: / is a commented out line a blank line or a non existing line? if it is interpreted a
Read the rest
s a blank linke, the disallow only counts for yandex, if the line is non existant, it counts for googlebot and yandex. i like simple things, but sometimes the complexity is between the lines.
In So you want to write your own CSV code (2014)
bauchidgw·Jul 15, 2011
see here for a spec how the robots.txt is parsed http://code.google.com/web/controlcrawlindex/docs/robots_txt... the robots.txt of http://picolisp.com (found at http://picolisp.com/robots.txt) allows the indexing of http://picolisp.com/21000 and all follow up pages. why? see the spec: The disallow directive specifies paths that must not be accessed by the designated crawlers. When no path is specified, the directive is ignored. see here https://github.com/franzenzenhofer/robotstxt for coffeescript implementation of a robots.txt parser
In How to make fun of Google Bot (PicoLisp Wiki)

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.