hadoofus

Personal plug, because the article mentions snakebite; hadoofus[1] is a C library that natively speaks to HDFS (v1), with a Python wrapper. No Java involved. Pros (vs Snakebite): * C, not Python. More portable to non-Python languages; probably faster * Speaks v1 of the protocol (AFAIK, nothing else does this except via JNI to hadoop's Java client) * Kerberos authentication supported (but not datanode tokens or the digest auth they employ) Cons: * Sort of abandonware. I don't personally use HDFS anymore, so it has stagnated. * As an extension of the above, doesn't support the v2 (Protobuf-based) protocol * C, not Python. All the usual potential issues with C that Python shields you from. [1]: https://github.com/cemeyer/hadoofus

Slightly tangential, but I'd like to shamelessly plug my HDFS client library (with nice Python bindings)[0]. If you want to access files on HDFS from your Python tasks outside of your typical map / shuffle inputs from the streaming API, it might be handy? It doesn't go through the JVM (the library is in C), so it might save a little latency for short Python tasks. [0]: https://github.com/cemeyer/hadoofus Also, I'm pretty new to publishing my own open source libraries. If people would be so kind, I'd love some constructive criticism. Thanks HN!