wdt

Interesting that nobody has mentioned: Warp speed Data Transfer (WDT)[1]. From the readme: - Warp speed Data Transfer (WDT) is an embeddedable library (and command line tool) aiming to transfer data between 2 systems as fast as possible over multiple TCP paths. - Goal: Lowest possible total transfer time - to be only hardware limited (disc or network bandwidth not latency) and as efficient as possible (low CPU/memory/resources utilization) 1. https://github.com/facebook/wdt

Look forward to it. I'd be interested to hear how your tool compares with Facebook WDT[1], as that would be my go-to right now if someone asked me for a fast point-to-point data transfer solution. [1] https://github.com/facebook/wdt

That's great stuff, I wonder how it compares to wdt[0] when there's only one TCP path. This has been my go-to tool to transfer files on the network. 0: https://github.com/facebook/wdt

WDT from Facebook - https://github.com/facebook/wdt/ We use it for copying hundreds of TiB's regularly, occasionally over a PiB.

Facebook open sourced WDT, which seems to have the same goals: https://github.com/facebook/wdt

So are they talking about software or hardware? They talk about LHC, but the innovation there was using a different file system: GPFS (http://iopscience.iop.org/1742-6596/219/7/072030) which meant that data is sharded, managed by age and transparently cached intelligently Are they instead talking about using replacing TCP with something more designed for bulk data transfer? Or are they talking about lighting up fibre with different transmitter pairs? (think dwdm x 10 http://www.webopedia.com/TERM/D/DWDM.html) for 5 million, I'd assume its software, if that the case, its pretty much just copy and paste what everyone else has been doing: http://filecatalyst.com/ http://asperasoft.com/ for opensource there is: http://uftp-multicast.sourceforge.net/ http://tsunami-udp.sourceforge.net/ https://github.com/facebook/wdt And a myriad of others. Mu