colly

Interesting. Can you compare it to colly? [0] Last time I looked it was the most popular choice for scraping in Go and I have some projects using it. Is it similar? Does it have more/less features or is it more suited for a different use case? (Which one?) [0] https://github.com/gocolly/colly

Sounds cool, but how is this different from Colly: https://github.com/gocolly/colly?

I've heard https://github.com/gocolly/colly#readme mentioned fondly, but I've never used it

The whole thing is written in Go on my end. Ingesting new headlines is handled in a goroutine that spawns within the process every 30 mins using a combo of the wonderful gofeed (https://github.com/mmcdole/gofeed) and colly (https://github.com/gocolly/colly) libraries. When loading the front page, you're loading a 1-minute-cached HTML page of it that was constructed out of headlines already in my PostgreSQL database that were put there by the ingestion goroutine. I like the idea of word clouds actually, I think you're on to something there. I think you just need to pre-generate them rather than doing it adhoc (if that's what you're doing here) for speed. Additionally, perhaps consider using sentiment in a way that orients stories based on positive and negative sentiment. Right now I am not seeing how I as a visitor/user can act on the sent

If performance, Especially concurrency matters then we should include Go based libraries in the discussion as well. Colly[1] is a all batteries included scrapping library, But could be bit intimidating for someone new to HTML scrapping. rod [2] is a browser automation library based on DevTools Protocol which adheres to Go'ish way of doing things and so it's very intuitive but comes with overhead of running chromium browser even if its headless. [1] https://github.com/gocolly/colly [2] https://github.com/go-rod/rod

Look into Go, specifically the colly pkg. Learnt Go just so I could use this library after wrestling with highly parralelized Python web scraping. The 1k/requests/sec/core figure is true and frankly deadly, and Go borrows a lot of useful stuff from Python like array slicing. It was pretty remarkable to go from 32 cores being maxed for 100 concurrent workers to... A 10-15% usage spike on a 4 core XPS :) https://github.com/gocolly/colly

I've considered that but didn't really know anything about the organization and how long they'll be around. We're gonna give Colly a try: https://github.com/gocolly/colly.