datavec

d handling things like computer vision workloads. In general, I always like seeing new takes on the columnar format processing approach but I'm just not seeing anything novel here. Clarification of intent would be great! [1]: https://github.com/deeplearning4j/DataVec [2]: https://github.com/deeplearning4j/nd4j

ow. I compete with databricks, google cloud,tensorflow, mxnet and a lot of the spark ecosystem The other problem is..pyspark itself is just bad for linear algebra. Period. You need specialized communications for tensors because the data often lives elsewhere and is in a blob type format. To get benefits from simd and the like you shouldn't be adding networked communications on top of this. That's why we made jumpy for our library. We interop directly at the pointer level and then use our parameter server and gpu stack. Using arrow I think mapd is attempting to do this as well. I don't see spark ever becoming good at linear algebra anytime soon despite what the folks at berkeley are pushing. As for "reasoning about data frames" spark underneath basically turns your dataframe declarations into a DAG. It does this in a very different way than tensorflow's XLA though. XLA's code gen is focused on math ops. Schemas are a whole other different animal. The great thing about spark is it already has "typed rdds" like dataframes and datasets it uses internally. It's really to rely on the JVM's type system and JIT for this. Whereas with math, if you constrain the domain and can make assumptions you can do codegen assuming that all you're doing are "for loops" which generalizes well for cuda and cpu with simd/AVX etc. You can't make the assumption that everything is a "for loop" as much with spark. We write and maintain datavec: https://github.com/deeplearning4j/DataVec and nd4j: https://github.com/deeplearning4j/nd4j which are integrated together and have a "tensor data type" which uses nd4j directly. This is very similar to what arrow does. We define a "transform process" which you could say: do ffts with. That kind of ETL isn't supported with dataframes directly without a lot of hacks. TLDR: TF will mainly stick to linear algebra

e. If you see something that's open source that's close to what you want try engaging the authors to see what they have to say. Maybe they will guide you. We've done that recently for our lapack integration with cpu and gpu as well as various neural net implementations. No offense but it kills me to see comments like this. I see tons of people complaining about features yet doing nothing to add them let alone engaging open source authors. It's kinda funny - every time someone has actually did that I've hired them. The developers that actually take action when engaging open source are amazing people.I have a feeling it's because they take the time to learn and get their feet wet even if it's initimidating. Other neat community initiatives include flink: https://issues.apache.org/jira/browse/FLINK-5782 Nasa (Apache Tika): https://github.com/apache/tika/pull/165 A language for our ETL library DataVec (supports binary vectorization AND sql like transformations!): https://github.com/deeplearning4j/DataVec/issues/224 A scala lib like tensorflow built on top of nd4j: https://github.com/ThoughtWorksInc/DeepLearning.scala Our spark ml integration: https://github.com/deeplearning4j/deeplearning4j/tree/master... The community is very active. We have 4200 people in a gitter room alone: http://gitter.im/deeplearning4j/deeplearning4j