HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
apache
spark
Scala2014–2026
apache / spark

spark

Apache Spark - A unified analytics engine for large-scale data processing

Data & Databases
Explore on GitHubRead Hacker News discussions
28Distinct people
31Mentions
0Explicit recommenders
44.5KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Apache Spark - A unified analytics engine for large-scale data processing

GitHub

Where it is used

No documented product use was verified in the last source check.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 7:56:55 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year31 dated mentions

Observed mentions: 2014-09-12 21:21:57 UTC to 2026-01-28 20:16:34 UTC.

024682014-01-01 00:00:00 UTC to 2015-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments20142015-01-01 00:00:00 UTC to 2016-01-01 00:00:00 UTC (exclusive): 8 mentions; 0 submissions, 8 comments2016-01-01 00:00:00 UTC to 2017-01-01 00:00:00 UTC (exclusive): 6 mentions; 0 submissions, 6 comments2017-01-01 00:00:00 UTC to 2018-01-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments2018-01-01 00:00:00 UTC to 2019-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-01-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 4 mentions; 0 submissions, 4 comments20202021-01-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments2022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 2 mentions; 1 submissions, 1 comments2025-01-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-01-01 00:00:00 UTC to 2027-01-01 00:00:00 UTC (exclusive): 2 mentions; 1 submissions, 1 comments. Incomplete calendar period.2026*

* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20142014-01-01 00:00:00 UTC2015-01-01 00:00:00 UTC101
20152015-01-01 00:00:00 UTC2016-01-01 00:00:00 UTC808
20162016-01-01 00:00:00 UTC2017-01-01 00:00:00 UTC606
20172017-01-01 00:00:00 UTC2018-01-01 00:00:00 UTC303
20182018-01-01 00:00:00 UTC2019-01-01 00:00:00 UTC000
20192019-01-01 00:00:00 UTC2020-01-01 00:00:00 UTC202
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC404
20212021-01-01 00:00:00 UTC2022-01-01 00:00:00 UTC303
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC000
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC000
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC211
20252025-01-01 00:00:00 UTC2026-01-01 00:00:00 UTC000
2026 (partial)2026-01-01 00:00:00 UTC2027-01-01 00:00:00 UTC211

What Hacker News says

8 sources
mbesto·Sep 10, 2015
> And it’s not just Node.js. Maybe you haven’t looked at what IBM is doing with open source lately. I was surprised when I dug in. For example, did you know they are leading contributors to: Linux OpenStack Cloud Foundry Docker plus many Apache projects like Spark, Cordova and Hadoop and of course, Node They are? (keyword => "leading") https://github.com/docker/docker/graphs/contributors https://github.com/apache/spark/graphs/contributors I find this claim dubious. Maybe I'm not digging hard enough.
In What the IBM Acquisition of StrongLoop Means for the Node.js Community
virtualwhys·Jul 27, 2015
Good points, as always. > None of those features (except sealed ADTs, which really aren't necessary) is missing from Kotlin. The set of Scala features Spark is using is pretty much identical to Kotlin. I didn't mention implicit conversions, but that's one area where Scala has taken the high risk high reward path. If you look a bit further down on the linked Spark page you'll see the conversions defined [1]. I'd say Spark's style guide is more geared toward Scala consumers and not necessarily library authors (i.e. if you know what you're doing have at it; otherwise keep to a strict subset). Basically, in aggregate Scala's feature set is incredibly powerful; that power (despite the complexity, tooling challenges, and slow builds) is what draws dsl authors to Scala. When IBM throws down an absolute boatload of cash on a project written in Sc
Read the rest
ala, by proxy Scala benefits. Right off the bat there are de facto 3,500 new Scala developers (since the initial IBM Spark team will of course be learning Scala). Furthermore, it's not like competitors aren't aware of this, Spark automically becomes more of a "big deal" as a result, which cannot help but be a boon for Scala. As for other JVM languages, agreed, Clojure has its own supply of awesome, though the lack of (built-in) static types are a no-op here. Groovy, meh, seems to be on a downtrend, I suspect non-immutable-by-default dynamic languages (except of course, javascript) will continue to fall out of favor. Now, Kotlin, it's a nice language. If they can deliver on near Java build times, might just be the Java killer that Scala will never be, or at least not in its current design (where, as you rightly say, there is often more than one way to do the same thing). Let's see how things play out over the next 3-5 years, JVM alternative language landscape is diverse and evolving quickly, good times now and great times ahead... [1] https://github.com/apache/spark/blob/master/sql/catalyst/src...
In Ask HN: Has anyone here programmed in Kotlin? What do you think about it?
virtualwhys·Jul 25, 2015
Interesting analogy, incredible how Spark drives the Scala helicopter like a car; they seem to have nailed parallel parking ;-) Seriously though, I doubt Kotlin would ever be the first choice for a project like Spark since Scala excels in the DSL department (really without rival on the JVM, and off, only Haskell comes close). Look through Spark source code [1] and you'll see: operator overloading, (sealed) ADTs, exhaustive pattern matching, etc. functionality that you can't even think the thought in in a language like Kotlin. This has nothing to do with monad transformers and other FPisms, just straight DSL power. Finally, to put things in perspective in terms of the mindhsare that Scala and its ecosystem are drawing, IBM is investing over 100 million dollars per year in Spark research and development [2]. Sorry, but that's absolutely mas
Read the rest
sive, maybe there are no dragons? Or if there are, the big players don't seem overly bothered. [1] https://github.com/apache/spark/blob/master/sql/catalyst/src... [2] https://www-03.ibm.com/press/us/en/pressrelease/47107.wss
In Ask HN: Has anyone here programmed in Kotlin? What do you think about it?
choppaface·Jun 12, 2015
They've integrated Tungsten / native sorting into shuffle and observed some decent speedups: * https://issues.apache.org/jira/browse/SPARK-7081 * https://github.com/apache/spark/pull/5868#issuecomment-10183... However, I guess reduceByKey (and friends) don't benefit yet. Their SGD implementation still uses TreeAggregate ( https://github.com/apache/spark/blob/e3e9c70384028cc0c322cce... ) so I wonder when they're planning to add some of the "Parameter Server" stuff (e.g. perhaps butterfly mixing or Kylix http://www.cs.berkeley.edu/~jfc/papers/14/Kylix.pdf )
In Announcing Apache Spark 1.4
super_sloth·Feb 17, 2015
Has anyone had some good experiences with Spark? I put several weeks in to moving our machine learning pipeline over to Spark only to find I kept hitting a race condition in their scheduler. After doing a bit of searching, it seems this is actually a known issue https://issues.apache.org/jira/browse/SPARK-4454 and there's been a fix on their github for a while: https://github.com/apache/spark/pull/3345 and yet in that time two releases have swung by bringing a tonne of features. I ended up having to drop Spark ultimately because I wasn't confident about putting it in to production (the random OOMs and NPEs during development weren't great either). Does anyone have any positive experiences?
In Introducing DataFrames in Spark for Large Scale Data Science

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.