snowflake

There is source code available for it (although the repo's retired): https://github.com/twitter-archive/snowflake/blob/b3f6a3c6ca... Basically one big server had a number of threads each requesting the next value in the sequence on that machine. The sequence only had to be unique to that machine and was just a counter of how many tweets since <last whole millisecond>. Tweets per millisecond per server was probably never a huge number, so they were able to share the 10 bits between datacenter id and that counter.

If you're not distributed, use an incrementing integer. If you're distributed, look into vector clocks[1] or snowflake[2][3] [1] https://en.wikipedia.org/wiki/Vector_clock [2] https://github.com/twitter-archive/snowflake/tree/snowflake-... [3] https://en.wikipedia.org/wiki/Snowflake_ID

The Data Center ID has been in Snowflake IDs for a while. It's present in the original version of Snowflake that Twitter open sourced back in 2010 [0]. Looks like it was added in this commit [1] from August 2010. [0] https://github.com/twitter-archive/snowflake/blob/snowflake-... [1] https://github.com/twitter-archive/snowflake/commit/ba2e67ea...

Snowflake has been retired in 2014 [1]. Are there any similar services around that are still maintained? [1]: https://github.com/twitter/snowflake

"Small Twitter clone" can be taken two ways, and this one, I think, is the less interesting of the two. (Not to denigrate the project for what it is; it's a good idea.) On a one-machine scale, Twitter can be a single Postgres table, or a single ElasticSearch index. At the scale they actually are, it's really a whole lot more. I'd really love a tutorial—or possibly a whole book—that instead took you through setting up the sort of distributed system that is required to make a Twitter clone run at a Twitter-like scale. Either through IaaS APIs, or on your own with something like OpenStack DevStack. It could probably start where this tutorial ends—with a one-node system running a Twitter-backend-alike monolith. And then each chapter would increase the scale, introduce a problem the scale causes, and then walk you through adding in an addition

For internal purposes, a 64 bit GUID will be sufficient. Look how Twitter uses Snowflake - https://github.com/twitter/snowflake - that's a scalable way to generate unique IDs which won't ever be revealed to the user. For default user names, I quite like how gfycat does it - http://gfycat.com/about - AdjectiveAdjectiveAnimal - so a user is given a default name like "ReluctantCandidWombat". Fun, unique, hard to iterate, and hopefully provides an incentive to the user to change it later.

Twitter's Snowflake, though it's gone now. The principle is the same. https://github.com/twitter/snowflake.

Twitter came up with a different scheme called Snowflake (https://github.com/twitter/snowflake).

Neat! I made one of these in Java a while back after reading the snowflake Twitter post (which has since been removed). Looks like Twitter has retired Snowflake: https://github.com/twitter/snowflake https://github.com/mumrah/flake-java

He could have just Googled. I think Twitter's Snowflake came to some very similar conclusions. https://github.com/twitter/snowflake/ I even wrote something in Ruby as an idea based off it (although it hasn't been used in production yet) https://github.com/pmarreck/ruby-snippets/blob/master/star_i...

A topic I rarely see (at least in depth, or in discussion) in stories/blogs about APIs is ID generation, i.e. best practices for creating access_token, resource ID and especially handling collisions. The Flickr blog post [1], which featured a simple implementation of a ticket server, is so far one of the easiest and most secure to use. Personally I don't feel comfortable using "hash of i++", "hash of ...", "i++", as they all "fall apart" when you need ids of different specifications (e.g. a 8-char ID, a 24-char token ID). Anyone have two cents on this? I have a difficult time imagining that larger APIs just "live with" the chance of collision (even though it can be very low) - they must mitigate it somehow, right? combined probability (such as the nonce check and tokens on oauth1 requests)? Actual ticket servers (e.g. [2], [3]) either int

I imagine many things use Unix time or a similarly flat calendar internally (Twitter's Snowflake[0] uses a custom scale), it's the translation into the Gregorian (/Jewish/Mayan/etc) calendar and time scale where the bugs come from. Can't count on 31540000 seconds being a year, time zones, etc. At the same time, the human brain doesn't seem particularly well-suited to breaking down Unix time into meaningful and reliably consistent scales, so I don't see time getting much easier barring some revolutionary timekeeping system. [0] https://github.com/twitter/snowflake

Twitter played with writing a UUID service a while back called Snowflake. https://github.com/twitter/snowflake

Maybe you have deep enough experience with Node to avoid the thin ice, but my experience has been one of sparsely-featured and buggy libraries when compared to their counterparts in Python, Java, etc. All languages need time to grow up, but for now we use Node as a pinch-hitter in our stack (I created a Snowflake server with great results) Honestly, Facebook has a lot of frontend code. I don't think Rails is a bad choice for that. My experience with large systems is that they're usually not homogenous. There's no reason they couldn't implement the Rest API components outside of Ruby if that was their biggest obstacle to success. Though I'm not sure it is, because even though you've obviously given this issue some thought, you didn't really articulate who was put off by their use of Rails? Edit: To avoid confusion, Snowflake as in https://

Can you explain what it means to "check the data"? The point here is that, for the last year, Twitter has NOT been generating sequential IDs. Just look at the source code[1]. They save a few low bits for the worker ID, data center, and worker-specific sequence. The highest bits are the timestamp bits. Maybe you can correct me if my assumptions based on the code are wrong, but my guess is you'll magically find that Twitter's highest ID grows linearly with time. Then you're just randomly sampling within this maximum timestamp. At this point you're assuming that Twitter sees a constant rate of signups wrt time, which I highly doubt. [1] https://github.com/twitter/snowflake/blob/master/src/main/sc...