smallweb

Why does Kagi's Smallweb have a 'no submissions' label? I was under the impression you just PR:ed here: https://github.com/kagisearch/smallweb

(Disclosure: I work at Kagi) Thanks for checking it out. There's currently over 30K blogs in the feed and we try to diversify it as much as possible. We welcome anyone to contribute more blogs, video channels and comics here: https://github.com/kagisearch/smallweb#%EF%B8%8F-guidelines-...

Personally, I use https://github.com/kagisearch/smallweb/ to feed my search engine / crawler. It contains 30k + rss/atom feeds of indie web sites. Thx for sharing this and the other directories.

For being supposedly human curated, it's shocking how much garbage has infiltrated the smallweb.txt registry. I have an agent with browser access running right now checking it and it's close to ~10%... Here's a sample of 10 examples that were found in literally the last two minutes... - "vermontdailychronicle.com" : https://github.com/kagisearch/smallweb/blob/5aa8a7740149bd67... - "ziggit.dev" : https://github.com/kagisearch/smallweb/blob/5aa8a7740149bd67... - "fastfrwrd.info" : https://github.com/kagisearch/smallweb/blob/5aa8a7740149bd67... - "evchargingstations.com" : https://github.com/kagisearch/smallweb/blob/5aa8a7740149bd67... - "escapecollective.com" : https://github.com/kagisearch/smallweb/blob/5aa8a7740149bd67... - "findingada.com" : https://github.com/kagisearch/smallweb/blob/5aa8a7740149bd67... - "fontsinuse.com" : https://gith

You may want to clarify that on https://github.com/kagisearch/smallweb because the README there says: > Blog has recent posts (<7 days old) This may be different than inclusion criteria for websites in general, but on first read it looks like it has to be very active. I might have missed something while skimming it, but would assume others would miss it as well.

The criteria are here: https://github.com/kagisearch/smallweb#small-web

Great idea. Index is based on submitted sites: https://github.com/kagisearch/smallweb/blob/main/smallweb.tx... There is also Small Comic: https://kagi.com/smallweb/?comic https://github.com/kagisearch/smallweb/blob/main/smallcomic.... And Small YouTube: https://kagi.com/smallweb/?yt https://github.com/kagisearch/smallweb/blob/main/smallyt.txt

You can check: <https://github.com/kagisearch/smallweb/blob/main/smallweb.tx...>. I can see that your RSS URL is listed there. But it currently does not appear in the search results here: <https://kagi.com/smallweb/?search=zahlman>. The reason appears to be this: "If the blog is included in small web feed list (which means it has content in English, it is informational/educational by nature and it is not trying to sell anything) we check for these two things to show it on the site: • Blog has recent posts (<7 days old) [...]" (Source: https://github.com/kagisearch/smallweb#criteria-for-posts-to...)

But per the instructions, it seems like that if one wants to add your own website, then one needs to add 2 other small websites (that are not on the list already)...so technically it does open things up to those who are not aware of the repo...assuming their site is pulled in when someone wants to add their own website. Obviously this scale is slow...but i think that's kinda the point, eh? Nevertheless, for every 1 person wanting to add their stuff, 2 others would technically get added i guess. See: https://github.com/kagisearch/smallweb?tab=readme-ov-file#%E...

Relevant https://github.com/kagisearch/smallweb/pull/425

There is a practical limit that we can't cache results for too long; Search engine users are particularly sensitive to stale data, especially around current events. Without a holistic and realiable way to know when the cache ought to be invalidated, our caching is mostly focused on mitigating "abuse", e.g., someone / bunch of people spamming the same search in a short timespan; no sense in repeating all those upstream calls. Most "cost saving engineering" is involved in finding cases/hueristics where we only need to use a subset of sources and omitting calls in the first place, without compromising quality. For example, we probably don't need to fire all of our sources to service a query like "youtube" or "facebook". Marginalia data is physically consolidated into the same infra that we use for small web results in our SERP, but also amon

I was surprised to find my blog there. I did a git blame on https://github.com/kagisearch/smallweb and it was in the initial commit, so I guess I'll never find out...

Btw this makes me think about kagi's smallweb initiative: https://github.com/kagisearch/smallweb