SIsillysaurusx·Oct 21, 2021
> Take your example about StyleGAN vs BigGan, I assume once it became clear that the latter was superior to the former that likely resulted in changes to existing architectures that then found additional improvements. This change in consensus is what enables that and is a good thing.
Well, no. :) But I think the "no" is because of the uniqueness of ML rather than a "no" to your point in general. You might be right about other fields; I don't have experience there.
In ML, there an enormous number of techniques. Style mixing was presented as a core feature of StyleGAN (https://arxiv.org/abs/1812.04948) and was enabled by default in the codebase (https://github.com/NVlabs/stylegan/blob/03563d18a0cf8d67d897...).
So there's a lot of "inertia" -- for example, when StyleGAN 2 came out, style mixing was still the default (https://github.com/NVlab
Read the rest
s/stylegan2/blob/f2f751cdc7f996e3138...).
I haven't had time to dig into StyleGAN 3, but I suspect that style mixing might still be enabled by default.
It wasn't until we did a detailed, methodical analysis side-by-side with BigGAN, specifically to answer the question "Why is BigGAN so much better for diverse datasets?" that, on a whim, I turned off style mixing and was astonished to see BigGAN type quality pop out of a StyleGAN type arch.
Discoveries like that usually go unnoticed, frankly because it's a lot of effort to write a paper specifically to say "Hey, if you're training StyleGAN, definitely turn off style mixing. It only seems to work well on faces."
However, if such a paper were to be written, and accepted into a peer-reviewed journal, then your original point would probably be valid. So I don't even know if it's worth writing all of this -- I just thought it'd be interesting to point out the "Well, not really" in this case. The knowledge ends up floating around on Twitter and Discord rather than being transmitted via scientific papers...
But, this all does tie in to your final point:
> Consensus is easily changed with the introduction of new data, faith hangs on no matter how much evidence is put forward that it's horseshit.
It's remarkably easy for old, accepted ideas to hang around. You'd think it'd just be a matter of "Run the experiment; experiment proves thing; thing becomes accepted." But in practice it's felt quite different...
The thing is, everything you're saying is true in general. As t approaches infinity, there tends to be more and more consensus about older ideas, like the existence of black holes, or the validity of laws like F=ma. So we should probably pay attention when there is 99% consensus on a particular topic.
But, for example, one reason I wouldn't want to publish a paper claiming style mixing was bad, is because it would contradict the results of Karras, who is famous. I'd better be very certain about my claim! So there's sometimes a reluctance to contradict the consensus, too, which ends up equivalent to "faith" in your example -- we have faith that famous scientists are correct. (They usually are.)
As a cherry on top, I'll just leave a link to Feynman's messenger lectures: https://www.youtube.com/watch?v=-kFOXP026eE&ab_channel=TalkR... ... the history of science is fascinating. I'd dreamt for years of becoming a scientist, but the actual experience turned out to be surprisingly different than what I thought it'd be. I love it though -- all these weird corner cases are the spice of life.