Internal parametrization, Analog VLSI, and the art of nonlinear search

• why noise? It allows for the most perfect matched filter in order to synchronously detect the gradient. Any other signal would do, but with varying degrees of success because they might lose information (if they lacked a certain spectral component) or not dominate dy/dt (if they didn't have sharp enough slope at all points). Matched filtering with noise is difficult unless you use the noise right then and there, and perform analogous transforms on both components.
  
• what does the analog hardware do? The important thing to remember is that although the math works out perfectly, there is no such thing as a perfect integrator/differentiator/averager, and the analog vlsi components must simply be good approximations of these components. It is nice that the system works, but not much can be said beyond that.
  
  

• The conclusion. We knew something like this all along, from Bishop. If the first moment of the Fourier transform is bounded = Cf, then n functions with internally varying parameters can approximate a function to within Cf^2/n, whereas a linear superposition can only approach Cf/d (1/n)^1/d. This has profound implications for why neural networks are so good. Key points: this works for *all* functions in a convex set in function space, and is not a mere average-case scenario (despite the use of the WLLN in the proof), and has to do with the 'smearing-out' that the internal parameters can do. Keep in mind the nonlinear nature of diodes and other physical devices when considering such theorems: *how could I actually build something that does this?* The first example, the gradient descent using noise, and the Barron paper, are not entirely unlinked.
• We resolved that sinusoids, sigmoids, etc. are equally powerful in the Cf-L2 sense that Barron uses. Bernd wisely observed that this does not always hold for real-life problems. Sigmoids and RBFs are better than mere sinusoids in almost all NN problems. This is because the probability measure defined over the set of 'smooth functions' is not very well-defined; using the trivial distribution is probably not a very useful thing to do.
• We observed that the term 'exponential expressiveness,' which everyone is much fond of tossing around, is useless unless we do some real work explaining what it really is.
• We observed that there is an exponential constraint on the smoothness. Say that my deviation along any one dimension is k. Then Cf ~ k^d, e.g. my one-D deviation must go down exponentially as my dimension increases. We note, however, that under the same circumstances the linear case fares far worse. We also discussed the search process -- e.g., the linear case is essentially a series of projections which can be done in linear time (or, for the matrix case, n^3).
• We noted the definition of Hilbert space and asked the Dirac question.
• Quantum mechanics offers a square root speedup, although internal parametrization offers an exponential decrease in the number of functions needed. How does this affect the best computer on earth.
• We discussed Kirkpatricks 'hard problems'. We discusses how reparametrizing the variables and the measures could make hard problems seem easy. Discussed 'natural measures.'
• Bernd: 'in practice...'
• Quick discussion of EM algorithm, error surfaces, and other stuff. Mentioned Tad Hogg, structured search. Mentioned spin glasses.

• How can one link entropy with the moment of the Fourier distribution?
• Answer: not obvious. We must decide on some probability measure that suggests a good way to do this.
• Hint: uniform continuity and generalizations thereof.
• Is the nonlinear basis more powerful than the linear basis in the infinite limit?
• Can we operate in a stochastic fashion?
• Should we discard dynamics and proceed after this fashion.