what regularized auto-encoders learn from the data-stronggeneratingstrong.pdf

Journal of Machine Learning Research 15 (2014) 3743-3773 Submitted 6/13; Published 11/14What Regularized Auto-Encoders Learn from theData-Generating DistributionGuillaume Alainguillaume.******@ Bengioyoshua.******@ puter Science and Operations ResearchUniversity of MontrealMontreal, H3C 3J7, Quebec, CanadaEditors:Aaron Courville, Rob Fergus, and Christopher ManningAbstractWhat do auto-encoders learn about the underlying data-generating distribution? Recentwork suggests that some auto-encoder variants do a good job of capturing the local manifoldstructure of data. This paper clari?es some of these previous observations by showing thatminimizing a particular form of regularized reconstruction error yields a reconstructionfunction that locally characterizes the shape of the data-generating density. We show thatthe auto-encoder captures the score (derivative of the log-density with respect to the input).It contradicts previous interpretations of reconstruction error as an energy function. Unlikeprevious results, the theorems provided here pletely generic and do not depend onthe parameterization of the auto-encoder: they show what the auto-encoder would tendto if given enough capacity and examples. These results are for a contractive trainingcriterion we show to be similar to the denoising auto-encoder training criterion with smallcorruption noise, but with contraction applied on the whole reconstruction function ratherthan just encoder. Similarly to score matching, one can consider the proposed trainingcriterion as a convenient alternative to maximum likelihood because it does not involvea partition function. Finally, we show how an approximate Metropolis-Hastings an be setup to recover samples from the estimated distribution, and this is con?rmed insampling :auto-encoders, denoising auto-encoders, score matching, unsupervised repre-sentation learning, manifold learning, Markov chains, generative models1. Introd