feedparser

stupid question: why did you not use feedparser? https://github.com/kurtmckee/feedparser/

i am using the feedparser library in python https://github.com/kurtmckee/feedparser/ which basically takes an RSS url and standardizes it to a reasonable extent. But I have noticed that different websites still get parsed slightly differently. For example look at how https://beincrypto.com/feed/ has a long description (containing actual HTML) inside but this website https://www.coindesk.com/arc/outboundfeeds/rss/ completely cuts the description out. I have about 50 such websites and they all have slight variations. So you are saying that in addition to storing parsed data (title, summary, content, author, pubdate, link, guid) that I currently store, I should also add an xml column and store the raw <item></item> from each url till I get a good hang of how each site differs?

Many people have given their "why," but I'll give another "how". castget¹ is a simple tool that does exactly what you want, just add a feed URL and it pulls all the episodes. It has a nice catchup mode so you can add feeds that you've listened to elsewhere without pulling all the old episodes too. hpodder² is another tool I've used in the past, but I can't remember why I switched. Depending on how much you like cobbling together your own solution html-xml-utils³ makes it incredibly easy to script a solution. For example, "curl feed | hxselect -c -s '\n' 'enclosure::attr(url)'" would list just the URLs. feedparser⁴ is a battle tested solution for processing feeds if you'd prefer a proper parsing solution over a hacky shell script(it will correctly handle different formats without any extra work for example). ¹ https://castget.johndal.c

There is JSON Feed¹ already. One of the spec writers is behind https://micro.blog, which is the first place I saw it(and also one of the few places I've seen it). I don't think it is a bad idea, and it doesn't take all that long to implement it given the sane and quite readable specification. I have long hoped it would pick up steam with the JSON-ify everything crowd, just so I'd never see a non-Atom feed again. We perhaps wouldn't need sooo much of the magic that is wrapped up in packages like feedparser² to deal with all the brokeness of RSS in the wild then. ¹ https://www.jsonfeed.org/ ² https://github.com/kurtmckee/feedparser

But RSS's date & time format was practically messy (due to being not properly specified and enforced---take a look at [1]) and it had no way to specify that an article has been updated so it should not show up again to the timeline. When I had some dozens of feeds it had been quite annoying to mentally ignore updated articles. For the obvious reason (cough timely standardization cough) Atom is technically and often practically superior to the RSS family of formats (somehow not distinguished to each other). And after more than 10 years of usage, I'm pretty sure that every feed reader in the continued existence now has Atom support. [1] https://github.com/kurtmckee/feedparser/tree/develop/feedpar...

The selfoss reader [0] has a nice concept of extending the format of the incoming data feeds, would be also a very nice feature for your code. Even nicer it would be if new incoming data formats could be configured in yaml files without having to recompile. the selfoss author mentions these kind of data: * RSS Feeds * Images from a RSS Feed * Images from deviantArt Users * Images from tumblr * Your twitter timeline * Tweets of a twitter user * heise News with full content * golem News with full content * MMOSpy News with full content * RSS Feeds with readability certainly not the most important sources, but it gives an idea about how diverse data sources are on the net today, so limiting to rss and atom is not a good idea. maybe some people would like to add facebook streams or other "social" sources. Also please take a look at the extens