sacred

ters': n_conv_filters, 'n_output_hus': n_output_hus, 'activation': activation, 'batch_norm': batch_norm, 'output_bn': output_bn, 'generation': generation, 'data_spec': { 'uniform_frac': uniform_frac, 'include_augment': True, 'batch_size': batch_size, 'bulk_chunk_size' : bulk_chunk_size, 'max_bulk_chunk_size': max_bulk_chunk_size, 'loss_weighter': loss_weight }, 'train_spec': { 'early_stopping_patience': early_stopping_patience, 'lr_plateau_patience': lr_plateau_patience, 'learning_rate': init_lr, 'clip_grads': clip_grads, 'partial_weight': partial_weight } } I've wanted to give Sacred a try https://github.com/IDSIA/sacred - looks promising but haven't tried yet so can't comment. I still tend to keep track of model performance by hand though. But I have always have the notebooks I can go back to for reference. This is something sacred could help a lot with. Another very non-trivial aspect of this kind of work is the compute/storage infrastructure you need to scale beyond a single workstation. We have a nice system here where $HOME lives on NFS and gets mounted when you log into any machine on the network - I can hardcode paths in my code and count on every worker having the same filesystem. I can't imagine how we would do distributed jobs without NFS. That's not a very realistic solution for homegamers though - you need a very fast network and expensive commodity hardware. And sys admins. Does anyone have a solution for that half of the problem? I've seen a number of merkle-tree based data version control solutions recently...