python-preloaded

We considered to use sth like this to cache some Python program state to speed up the startup time, as the startup time was quite long for some of our scripts (due to slow NFS, but also importing lots of libs, like PyTorch or TensorFlow). We wanted to store the program state right after importing the modules and loading some static stuff, before executing the actual script or doing other dynamic stuff. So updating the script is still possible while keeping the same state. Back then, CRIU turned out to not be an option for us. E.g. one of the problems was that it was not possible to be used as non-root (https://github.com/checkpoint-restore/criu/pull/1930). I see that this PR was merged now, so maybe this works now? Not sure if there are other issues. We also considered DMTCP (https://github.com/dmtcp/dmtcp/) as another alternative to CRIU

At the other extreme, checkpointing the whole process once all imports have been resolved and restoring it for every execution can be used for frequently-run tools: https://github.com/albertz/python-preloaded

I would love to make more use of CRIU. E.g. I considered to use CRIU for my Python preloaded logic (https://github.com/albertz/python-preloaded). Unfortunately, at that point in time, CRIU must be used with root access, which was not an option. However, I see that the PR was merged now, so maybe it works now? (https://github.com/checkpoint-restore/criu/pull/1930) There is also DMTCP (https://github.com/dmtcp/dmtcp/) but this might have other problems for my use case. My solution was to use a fork server instead, which works almost equally well. There are not really much downsides with this approach. And this is actually quite simple, and also quite cross-platform (except Windows).