spark

First I'd ask what's the skill you want to gain? Do you want to learn how to setup Hadoop clusters and Zookeeper etc. on bare metal and understand all the maintenance of such a system. Or do you want to learn how to use the tool for enabling data science projects? If it's the former, companies like Databricks are becoming popular because they abstract a lot of that complexity away: https://databricks.com/try-databricks Because you're coming at it from scratch I'd strongly advise you look at the future trend and start straight with Spark. It is also made by Apache and is the next generation solution. https://spark.apache.org/ To give an idea of the difference, Hadoop uses Map (transform) and Reduce (action) higher order functions to solve distributed data queries and aggregations. Spark can do the same but it has access to so many addition

Thanks for pointing out the switch away from RDD, looking at the master branch for MLlib in github it looks like ALS is still using RDDs? https://github.com/apache/spark/blob/master/mllib/src/main/s... Admittedly I'm not a spark expert though, thanks for bringing this up!

What's wrong with exporting to csv? Seems like it's supported: https://github.com/apache/spark/blob/master/sql/core/src/mai...

You're right about Python and R having to pass data back and forth to the JVM for certain operations, but also keep in mind that native code still runs in the native interpreter. That means you have access to the full ecosystem of the native language. For example, if I want to convert an RDD of JSON strings into Python dictionaries: import json rdd_dict = rdd.map(lambda x: json.loads(x)) Same goes for any external Python libraries I install on the cluster and want to use in my Spark job. You can even run your Python code on PyPy [4]! For me, working in Python generally feels like a first class experience on Spark. There are areas -- like GraphX [0], certain niche features [1] -- where Scala is definitely easier to work with, but with time that is becoming less [2] and less [3] true thanks to the DataFrame API. [0] https://spark.a

OP points to SPARK-12795 [0] and is all open source. They generate Java source code. You can read more at the prototype pull request: https://github.com/apache/spark/pull/10735 (I couldn't find a spec doc). If I understand correctly they insert a `WholeStageCodegen`[1] operator into the plan: /** * WholeStageCodegen compile a subtree of plans that support codegen together into single Java * function. * * Here is the call graph of to generate Java source (plan A support codegen, but plan B does not): * * WholeStageCodegen Plan A FakeInput Plan B * ========================================================================= * * -> execute() * | * doExecute() ---------> inputRDDs() -------> inputRDDs() ------> execute() * | * +------

You don't indent after a package e.g. https://github.com/apache/spark/blob/master/core/src/main/ja... And it's fair enough to indent with a main since it is so infrequent. Ideally you only have one main entry point.