open-source-search-engine

Any new open-source search option is good, but I also wish more attention was given to prior open projects like GigaBlast[0]/KBlast[1] crawlers, etc. It hasn't escaped the wider world that quality open-source search is desirable, and it's hard to think what this new EU project brings to the table that isn't already available if others want to contribute to existing efforts. I wish the EU project the best of luck of course! [0] https://github.com/gigablast/open-source-search-engine [1] https://github.com/fossabot/kblast

there's some stuff here : https://github.com/gigablast/open-source-search-engine

A snapshot of the source code seems available here: https://github.com/gigablast/open-source-search-engine They offer "stripped" which removes many tracking parts of the website. Additionally it could be useful for individuals that are on a degraded network connection (different parts of the world). It also has an API which I haven't interacted with but seems neat. I just figured I would share it.

Just found a little gem myself. I am working on another open source search engine[0], and needed a way to make bad behaving document filters timeout. Unfortunately the document filter in questioning dose spawn child processes, so the normal way of using fork() and a monitoring process was not working. However using ulimit like this should work: https://github.com/gigablast/open-source-search-engine/blob/... . Hadn’t thought about spanning a new shell and let it have control like that :) 0: https://github.com/searchdaimon/enterprise-search

I love how the code is such a mess. You can really tell one guy just wrote this whole thing over the span of a decade... It's just one patch on top of another and the comments are pretty amusing. Also funny to see hardcoded algorithms for pre-defined site paths and whole domains such as facebook/myspace/vimeo. This is truly a makeshift search engine on a massive scale. EDIT: Gotta say, this has some very useful pieces of code. I'm working on a niche-specific crawler and am battling the url stripping/cleanup part of it. This is very useful: https://github.com/gigablast/open-source-search-engine/blob/...

The code does not seem to be neatly written: have randomly checked a few files and found that the const methods and exceptions are not properly used. Here is a sample function: const char *CountryCode::getAbbr(int index) { if(index < 0 || index > s_numCountryCodes) index = 0; return(s_countryCode[index]); } https://github.com/gigablast/open-source-search-engine/blob/...

Gigablast (founded in 2000 by Matt Wells) announced [0] about open-sourcing their engine under the Apache version 2 license at July 30. The engine is written in mixture of C and C++ and counts more than 500,000 lines of code, see the Github page [1]. Some facts about the engine: The code compiles into a single executable file which can scale on thousands servers. It is easily configurable and has a nice documentation [2]. The code is very stable, it works in production since 2002. Document processing is done using plugins, so you can write a plugin for any type of documents. --- I would like to see a search engine based on this in the dark-nets, particularly in I2P. [0] - http://www.prnewswire.com/news-releases/gigablast-now-an-ope... [1] - https://github.com/gigablast/open-source-search-engine [2] - https://www.gigablast.com/admin.html