facebook-page-post-scraper

I love the way you format your py code. clean and super easy to read. https://github.com/minimaxir/facebook-page-post-scraper/blob...

Interestingly, Facebook is surprisingly liberal with their public data endpoints, with no apparent rate limiting for API retrieval from public Facebook Pages and Groups (source: I develop a scraper and have not been yelled at by FB, yet: https://github.com/minimaxir/facebook-page-post-scraper ) Twitter, meanwhile, still has that stupid 3200 most-recent tweet limit on historical data, which is apparently a "technical limitation."

> (Disclosure: I have developed a Facebook Page Post Scraper [https://github.com/minimaxir/facebook-page-post-scraper] which explicitly follows the permissions set by the Facebook API.) I played with the idea of creating some social aggregation type service with some friends (as a business). The more I read about FB's past behavior with regard to this, and how essential they are to any sort of service like, that, I canned the project. Regardless of what their TOS say, if you get on their radar and they send you a cease-and-desist, it's game over. Facebook is not in the business of subverting their revenue stream, so if you are making money off them and it's preventing them from capitalizing on their users, don't expect to last long if you exist by their grace. Really, there's an interesting space between so small nobody cares and large

Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API following rate limits or b) if there is no API, and the data has the explicit purpose of being shared publically (e.g blogs), I follow robots.txt. Of course, most companies have a do-not-scrape clause in their ToS anyways, to my personal frustration. (Disclosure: I have developed a Facebook Page Post Scraper [https://github.com/minimaxir/facebook-page-post-scraper] which explicitly follows the permissions set by the Facebook API.)

Nice! I was considering using the tool you built [0] to do similar analysis of the posts from the leading presidential candidates FB pages. [0] https://github.com/minimaxir/facebook-page-post-scraper

As someone who owns a repository with 11,529 stars (https://github.com/minimaxir/big-list-of-naughty-strings), I notice a few problems with the logic presented. I happen to have a scraper handy for tracking stars over time, and profiling the users who made those stars (https://github.com/minimaxir/get-profile-data-of-repo-starga...). Here is a chart of the daily number of Star events on the big-list-of-naughty-strings repository as of today: http://i.imgur.com/NzzjuKK.png The paper assumes that popular repositories are popular on their own merit. But as you can see from my chart, there are clear spikes, one when the repository was first released, one a few days afterward, and one in February for literally no reason. The cause of those spikes were Hacker News, /r/programming, and /r/webdev respectively. Here's another one of my repositorie

Jupyter notebooks are for running ad hoc blocks of code. The biggest advantage of doing so is being able to check the output of each block to make sure the output is expected. This feature in particular makes for a great tool for tutorials. (Example of mine: https://github.com/minimaxir/facebook-page-post-scraper/blob...) It is definitely not a tool to replace a typical Python workflow.

Fun Graph API fact: while it's incredibly easy for any user to scrape data off of public Facebook Pages (https://github.com/minimaxir/facebook-page-post-scraper), it is very hard to scrape personal data from people on Facebook, even if you are Friends with them or their Facebook statuses are public. Personal data is much more valuable. I've tried multiple endpoints to do so (for data analytics purposes) and they have all failed. The only way to get data on a person is for them to opt into granting the necessary permissions for the app.