HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
Yelp
mrjob
Python2011–2016
Yelp / mrjob

mrjob

Run MapReduce jobs on Hadoop or Amazon Web Services

6Distinct people
8Mentions
0Explicit recommenders
2.6KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Run MapReduce jobs on Hadoop or Amazon Web Services

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 9–Nov 16 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/16/2026, 3:59:04 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year8 dated mentions

Observed mentions: 2011-01-10 14:06:51 UTC to 2016-10-19 05:19:58 UTC.

01232011-01-01 00:00:00 UTC to 2012-01-01 00:00:00 UTC (exclusive): 3 mentions; 1 submissions, 2 comments20112012-01-01 00:00:00 UTC to 2013-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2013-01-01 00:00:00 UTC to 2014-01-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments20132014-01-01 00:00:00 UTC to 2015-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2015-01-01 00:00:00 UTC to 2016-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2016-01-01 00:00:00 UTC to 2017-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2016

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20112011-01-01 00:00:00 UTC2012-01-01 00:00:00 UTC312
20122012-01-01 00:00:00 UTC2013-01-01 00:00:00 UTC000
20132013-01-01 00:00:00 UTC2014-01-01 00:00:00 UTC303
20142014-01-01 00:00:00 UTC2015-01-01 00:00:00 UTC000
20152015-01-01 00:00:00 UTC2016-01-01 00:00:00 UTC000
20162016-01-01 00:00:00 UTC2017-01-01 00:00:00 UTC202

What Hacker News says

3 sources
Here's the code that's doing the actual line parsing: https://github.com/Yelp/mrjob/blob/master/mrjob/protocol.py#... It's just splitting on tab for input and re-joining on tab for output. Some extra logic for lines without tabs. EDIT: The example in the post is using JSON for communication between intermediate steps, while the Hadoop Streaming example is using a custom delimiter format. So this isn't really a fair comparison; the mrjob example could just as easily use the same efficient intermediate format.
In A Guide to Python Frameworks for Hadoop
Former[1] primary mrjob maintainer here, thanks for the shout-out! I'd like to make a couple of notes and corrections. In particular, I thank you for recommending mrjob for EMR usage, as it's something we've made a point of trying to be the best at. First of all, all of these frameworks use Hadoop Streaming. As mentioned in the mrjob 0.4-dev docs [2] (pardon the run-on sentence): "Although Hadoop is primarly designed to work with JVM code, it supports other languages via Hadoop Streaming, a special jar which calls an arbitrary program as a subprocess, passing input via stdin and gathering results via stdout." mrjob's role is to give you a structured way to write Hadoop Streaming jobs in Python (or, recently, any language). When your task runs, it's taking input and output the same way as your raw Python example does, except mrjob is passi
Read the rest
ng it through to the methods you've defined after running each line through a deserialization function. It picks the code to run based on command line arguments such as --mapper, --reducer, etc. The output of the code is again serialized. The methods of [de]serialization are defined declaritively in your class, as you showed. So why did you find mrjob to be slower than bare Hadoop Streaming? I don't know! In theory, you're running approximately the same code between the mrjob and the bare Python versions of your script. If anyone has time to dig into this and find out where that time is being spent, I would be grateful. Results should be sent to the issue tracker on the Github page [3]. Feel free to ask clarifying questions. I realize I may not be explaining this effectively to people unfamiliar with the ins and outs of Python MapReduce frameworks. I'm thinking of organizing the 2nd "mrjob hackathon" in the near future, so please ping me if you're interested in contributing to an easy-to-handle lots-of-low-hanging-fruit OSS project. (Particularly if you're a Rubyist, because we have an experimental way to use Ruby with it.) [1] mrjob is maintained by Yelp, where I worked until recently. It's still under active development, though it's slowed somewhat since Dave and I moved on. [2] http://mrjob.readthedocs.org/en/latest/guides/concepts.html#... [3] http://www.github.com/yelp/mrjob/issues/new
In A Guide to Python Frameworks for Hadoop

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.