gcsfuse

It uses FUSE and there's three types of Kernel cache you could use with FUSE (although, it seems like gcsfuse is exposing only one): 1. Cache of file attributes in the Kernel (this is controlled by "stat-cache-ttl" value - https://github.com/GoogleCloudPlatform/gcsfuse/blob/7dc5c7ff...) 2. Cache of directory listings 3. Cache of file contents It should be possible to use (2) and (3) for a better performance but might need changes to the underlying fuse library they use to expose those options.

Is this the same gcsfuse that's been around for years, only now with official Google support? https://github.com/GoogleCloudPlatform/gcsfuse

https://github.com/GoogleCloudPlatform/gcsfuse?

Currently being tracked here: https://github.com/GoogleCloudPlatform/gcsfuse/issues/961

The only advantage of Google Cloud is the TPUs--if you're not running massive machine learning workloads, AWS is almost always the better choice on features, service, and reliability. The *link between compute and storage* is not even officially a production product: "Please treat gcsfuse as beta-quality software. Use it for whatever you like, but be aware that bugs may lurk, and that we reserve the right to make small backwards-incompatible changes." https://github.com/GoogleCloudPlatform/gcsfuse/ If a supposed cloud platform can't even produce a reliable way to access your data, then they have no basis being used in any halfway serious setting.

Disclosure: I work on Google Cloud, and helped out with this effort. We're hoping to provide some more "How Fermilab did it" later (maybe at NEXT?) but our blog post had a little more information [1]. End-to-end from "Let's do this!" to Supercomputing was about 3 weeks (including some bugfixes to the GCE support in HTCondor). For people asking, Fermilab put up about 500T into a GCS Regional bucket (into us-central1). We have great peering at Google, so I think this was streaming at about 100 Gbps from Fermilab to us. As surmised, the science that we ran was lots of individual tasks, each one needing about 2 GB of RAM per vCPU, so during Supercomputing we right-sized the VMs using Custom Machine Types [2]. IIRC, several tasks needed up to 20 GB per task, so we sized PD-HDD to 20 GB per vCPU. All of this was read from GCS in parallel via gc