Github repo: https://github.com/nathanmarz/storm
Strange Loop presentation: http://www.infoq.com/presentations/Storm
In ØMQ: Mission Accomplishednathanmarz
storm
Java2011–2014
nathanmarz / storm
storm
Distributed and fault-tolerant realtime computation: stream processing, continuous computation, distributed RPC, and more
22Distinct people
31Mentions
1Explicit recommenders
8.8KGitHub stars
What to know before exploring
What it is
Distributed and fault-tolerant realtime computation: stream processing, continuous computation, distributed RPC, and more
GitHub
Where it is used
No documented product use was verified in the last source check.
What people are saying
The master node coordinates the Storm cluster. It distributed code around the cluster, monitors topologies, and reassigns any tasks that have failed. Check out the wiki for…
nathanmarz used a recommendation phrase. One indexed comment matches. This is a phrase match, not a verdict.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
- nathanmarz/storm-starter · linked in 2 of the same discussions
Mentioned in the same Hacker News threads. That is not proof they are substitutes.
This week on Hacker News
Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 12:45:45 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year31 dated mentions
Observed mentions: 2011-09-19 16:01:59 UTC to 2014-06-01 03:03:00 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2011 | 2011-01-01 00:00:00 UTC | 2012-01-01 00:00:00 UTC | 10 | 1 | 9 |
| 2012 | 2012-01-01 00:00:00 UTC | 2013-01-01 00:00:00 UTC | 16 | 2 | 14 |
| 2013 | 2013-01-01 00:00:00 UTC | 2014-01-01 00:00:00 UTC | 4 | 0 | 4 |
| 2014 | 2014-01-01 00:00:00 UTC | 2015-01-01 00:00:00 UTC | 1 | 1 | 0 |
From the project README
An excerpt from the repository, fetched Nov 13, 2026.
Read the project’s introduction
IMPORTANT NOTE!!!
Storm has Moved to Apache. The official Storm git repository is now hosted by Apache, and is mirrored on github here:
https://github.com/apache/incubator-storm
Contributing
Source code contributions can be submitted either by sumitting a pull request or by creating an issue in JIRA and attaching patches.
Migrating Git Repos from nathanmarz/storm to apache/incubator-storm
If you have an existing fork/clone of nathanmarz/storm, you can migrate to apache/incubator-storm by doing the following:
Create a new fork of apache/incubator-storm
Point your existing clone to the new fork:
git remote remove origin
git remote add origin git@github.com:username/incubator-storm.git
Issue Tracking
The official issue tracker for Storm is Apache JIRA:
https://issues.apache.org/jira/browse/STORM
User Mailing List
Storm users should send messages and subscribe to user@storm.incubator.apache.org.
You can subscribe to this list by sending an email to user-subscribe@storm.incubator.apache.org. Likewise, you can cancel a subscription by sending an email to user-unsubscribe@storm.incubator.apache.org.
You can view the archives of the mailing list here.
Developer Mailing List
Storm developers should send messages and subscribe to dev@storm.incubator.apache.org.
You can subscribe to this list by sending an email to dev-subscribe@storm.incubator.apache.org. Likewise, you can cancel a subscription by sending an email to dev-unsubscribe@storm.incubator.apache.org.
You can view the archives of the mailing list here.
Which list should I send/subscribe to?
If you are using a pre-built binary distribution of Storm, then chances are you should send questions, comments, storm-related announcements, etc. to user@storm.apache.incubator.org.
If you are building storm from source, developing new features, or otherwise hacking storm source code, then dev@storm.incubator.apache.org is more appropriate.
What will happen with storm-user@googlegroups.com?
All existing messages will remain archived there, and can be accessed/searched here.
New messages sent to storm-user@googlegroups.com will either be rejected/bounced or replied to with a message to direct the email to the appropriate Apache-hosted group.
Read the full README on GitHubWhat Hacker News says
10 sourcesThe master node coordinates the Storm cluster. It distributed code around the cluster, monitors topologies, and reassigns any tasks that have failed. Check out the wiki for more information about how Storm works and how to use it: https://github.com/nathanmarz/storm/wiki
Some of the things that Storm does are non-trivial, especially guaranteeing data processing. I recommend thinking more about whether you want to re-implement these kinds of things on other technologies.
In Storm: distributed and fault-tolerant realtime computation (slides)Storm does not have 100's of jars of dependencies, nor does it require much memory to run. In fact, it's quite straightforward to spin up a Storm cluster (see https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-... ) and Storm has a local mode where you can develop/test topologies on your local mode completely in-process.
In Twitter's Storm (complex event processing system) is now open sourceStorm can be used with any programming language. I still need to finish the documentation on this, but I have notes here: https://github.com/nathanmarz/storm/wiki/Using-non-JVM-langu...
The gist is: Storm topologies are just Thrift structures, and Storm can execute processing components in any language by communicating with subprocesses over a simple protocol based on JSON messages over stdin/stdout. Storm has adapters implementing this protocol for Ruby and Python, and it's easy to make a new adapter for any other language. The adapter libraries are ~100 lines of code and have no dependencies other than JSON.
In Twitter's Storm (complex event processing system) is now open sourceYou generally feed a Storm topology with a queueing system at the source. For example, we use Kestrel to feed Storm topologies. There's a setting called "TOPOLOGY_MAX_SPOUT_PENDING" which controls how many tuples can be pending on any given spout task at a time. This is more than sufficient for controlling bursts of data. If things back up, they will queue on the queueing server (s). In the future (not for a few months, at least), Storm will have auto-scaling where it will automatically scale the topology to the data.
If a tuple fails to be processed, the tuple(s) that triggered that tuple are replayed from the spout. See https://github.com/nathanmarz/storm/wiki/Guaranteeing-messag... for more info on that. In that sense, Storm is an at-least-once delivery system (but messages are sent more than once only in failure scenarios). It is up t
Read the rest
o you to architect your systems to handle this. The approach I take is to build systems using a hybrid of batch processing (Hadoop) and realtime processing (Storm). With batch processing, you can run idempotent functions even with duplication, which lets you correct what's happening at the realtime layer. In that sense, Hadoop and Storm are extremely complementary. Here are some slides from a presentation I gave about this technique: http://www.slideshare.net/nathanmarz/the-secrets-of-building...
Fields grouping uses consistent hashing underneath. So if you redeploy with more parallelism, it scales naturally and easily.
If a supervisor dies, it starts back up like nothing happened. Most notably, nothing happens to the worker processes. All state is kept either in disk or in Zookeeper. All daemons in Storm are fail-fast, and the Supervisor uses kill -9's to kill workers, so that makes things extremely robust.
I hope that answered your questions!
The Storm deploy is awesome. It's built using an awesome tool called Pallet: https://github.com/pallet/pallet
I'm going to try to get more documentation on using Storm from Clojure in the next few weeks. I write all my topologies in Clojure using a small Clojure DSL that ships with Storm:
https://github.com/nathanmarz/storm/blob/master/src/clj/back...
In Twitter's Storm (complex event processing system) is now open sourceHey all, I'm the author of Storm. Just wanted to point you to a few resources:
I've written a lot of documentation on the wiki, which you can find here: https://github.com/nathanmarz/storm/wiki
There's a few companion projects to Storm. These are:
One-click deploy for Storm on EC2: https://github.com/nathanmarz/storm-deploy
Adapter to use Kestrel as a Spout within Storm: https://github.com/nathanmarz/storm-kestrel
Starter project with example topologies that you can run in local mode: https://github.com/nathanmarz/storm-starter
Feel free to ask me questions here or on Storm's mailing list ( http://groups.google.com/group/storm-user ), and I'll answer as best I can!
In Twitter's Storm (complex event processing system) is now open sourceIt is a little buried in the github wiki, but this 'Rationale' page is a good overview of the project - https://github.com/nathanmarz/storm/wiki/Rationale
In Twitter's Storm (complex event processing system) is now open sourceTwitter's Storm (complex event processing system) is now open source
461 points · Open discussion