WeightWatcher

Neat result. The symmetry exploitation here reminds me of recent work connecting neural network training dynamics to renormalization group theory. Charles Martin's SETOL paper https://arxiv.org/abs/2507.17912 shows that well-trained layers converge to something like an RG fixed point—the eigenvalue spectrum of the weight matrix develops power-law tails with exponent α ≈ 2, which is the signature of scale invariance. At this fixed point, the "effective correlation space" is low-dimensional: you can truncate the SVD aggressively and recover nearly identical test accuracy. I wonder if there's a connection to your Taylor truncation order. In RG terms, higher-order polynomial interactions are "irrelevant operators"—they get suppressed as you flow toward the fixed point. If trained attention heads are sitting near this fixed point, that might e

These models are pretty much always extrapolating [0] Whether the extrapolation is crude/low-rank or astute/high-rank is a question of memorization vs generalization. That gets into the question of whether or not the model is over-fitted or under-fitted. There are certain heuristics borrowed from high dimensional statistical physics that can be used to guess how good the test performance of a model will be on a typical task without even knowing what the test data is [1]. Originality for me means finding better answers to sub-tasks, and then combining those answers together in a better way. This is the nirvana of cross-entropy minimization - the emergence of capability results from gaining the ability to amass a wider range of skills, improving upon them, and percolating those improvements towards multiplying the leverage of other skills.

Here's an alternative approach, that actually provides real world results https://calculatedcontent.com/2019/12/03/towards-a-new-theor... Using techniques from statistical mechanics and strongly correlated systems, we can compute the average-case-behavior of a real world DNN We believe we can reproduce some of the results of the NTK by using a Gaussian Random Matrix. But if we use a more realistic, heavy tailed matrix, we get more practical results A early fork of the theory has been published in JMLR https://arxiv.org/abs/1810.01075 and the empirical results in Nature Communications https://www.nature.com/articles/s41467-021-24025-8 and we have an open source tool , weightwatcher, which can be used in production pip install weightwatcher https://github.com/CalculatedContent/WeightWatcher Please give it a try and let me know if it is us

I'm developing the weightwatcher tool for deep neural networks into a full fledged product http://github.com/CalculatedContent/WeightWatcher The weightwatcher lets you detect potential problems in a trained neural network https://calculatedcontent.com/2020/02/16/weightwatcher-empir...