AutoEncoder
Table of Contents
An encoder-decoder architecture trained to reconstruct its input (hence the word auto (self)) passing via an intermediate embedding stage before the decoder maps it to the reconstruction.
- A direct, non-reducing pipe (no characteristic hourglass) will result in no important condensed representation of the input being learned and is not of much use.
- a bottleneck layer in the middle at the end of the encoder is important
- note that this bottleneck layer should still be complex enough to be able to capture the semantic differences in the input distribution completely
- for instance, 10 units for encoding the MNIST dataset (0-9 handwritten digits) is sufficient but going lower will result in unecessary semantic overlaps
The loss for the case of:
- regression : mean squared error between input and reconstruction
- classification : binary cross entropy loss between input and output binary vector
1. Denoising Autoencoder
A denoising autoencoder jitters the input with guassian noise and tries constructing the clean (denoised) variant of the noisy input by trying to map it to the original clean input.