flockdb

In 2010, Twitter was using a graph database, FlockDB, instead of a relational database to implement their social graph where relationships like following and followers are represented as edges between nodes that represent people. It's since been deprecated, and I'm not sure what type of database they're using today. Anyway, they have a nice write-up blog post [1] and FlockDB is open source [2]. [1]: https://blog.twitter.com/engineering/en_us/a/2010/introducin... [2]: https://github.com/twitter-archive/flockdb

Some software that Twitter has put out: [1] Heron, a realtime, distributed, fault-tolerant stream processing engine - https://github.com/twitter/heron [2] Finagle, a fault tolerant, protocol-agnostic RPC system - https://github.com/twitter/finagle/ [3] FlockDB, a distributed, fault-tolerant graph database - https://github.com/twitter/flockdb [4] Gizzard, a flexible sharding framework for creating eventually-consistent distributed datastores - https://github.com/twitter/gizzard [5] Twemcache, a Twitter fork of Memcached - https://github.com/twitter/twemcache [6] Twemproxy, a fast, light-weight proxy for memcached and redis - https://github.com/twitter/twemproxy

Twitter puts out a LOT of software, and open-sources a good portion. I don't know if this is smart in the sense of being overstaffed relative to their revenue, but they do it regardless. Some examples: [1] Fabric, an SDK for mobile apps - https://docs.fabric.io/android/fabric/overview.html [2] Heron, a realtime, distributed, fault-tolerant stream processing engine - https://github.com/twitter/heron [3] Finagle, a fault tolerant, protocol-agnostic RPC system - https://github.com/twitter/finagle/ [4] FlockDB, a distributed, fault-tolerant graph database - https://github.com/twitter/flockdb [5] Ruby implementation of the ICU (International Components for Unicode - https://github.com/twitter/twitter-cldr-rb [6] Clockwork Raven, Human-Powered Data Analysis with Mechanical Turk - https://github.com/twitter/clockworkraven [7] Gizzard, a flexible

Yeah, FlockDB (https://github.com/twitter/flockdb) comes to mind. I think Titan (https://github.com/thinkaurelius/titan) should / will be able to handle this too.

Twitter does this. Flockdb is built ontop of an RDBMS http://github.com/twitter/flockdb the downside is that unlike a proper graph database (such as neo4j) you can only query a single hop per SQL query (which is fine for twitter, since there is no requirement for 'followers of followers' etc.)

Twitter, LinkedIn and Facebook all make some bits and pieces of their system available as open-source. You might particularly want to look at Twitter's FlockDB[1], LinkedIn's Kafka[2], and/or Facebook's Scribe[3]. [1]: https://github.com/twitter/flockdb [2]: http://sna-projects.com/kafka/ [3]: https://github.com/facebook/scribe FlockDB is designed to deal with the "social graph" connection stuff (who follows who, etc.) and is optimized for that sort of thing. Kafka and Scribe are more about delivering message widely. Poke around, look at their code, chase down and associated papers, and you should be able to sort out an approach... even if you don't adopt any of those codebases directly. FWIW, I'm looking at integrating FlockDB for some stuff I'm working on, but I haven't gotten very far down that path yet, so I don't have a lot mor

It's worth noting that Twitter has built a lot on top of MySQL to get to the scale they're at. Take FlockDB for example, https://github.com/twitter/flockdb So I guess that statement should be written as "you can't scale with just those". :)

I'll be working mostly on Quoddy ( https://github.com/fogbeam/Quoddy ), but may also work on Neddick (https://github.com/fogbeam/Neddick ) a little. Both are part of something called ScrewPile: http://code.google.com/p/screwpile/ I'll also be spending some time with the Programming Scala book, and probably fiddling around with Twitter's FlockDB ( https://github.com/twitter/flockdb/ ) a little.

They're also using their own FlockDB (http://github.com/twitter/flockdb) to store their social graph.