The Effect of Pre-ReLU Input Distribution on Deep Neural Net's Performance
This research seeks to understand, through a more statistical formulation, the underlying idea behind pre-ReLU batch-normalization (BN) in deep neural net (DNN). While BN has been found to be able to boost DNN training by correcting internal covariate shift, we attempt to further simulate (and even optimize) its effect through different approaches. Our experiments went from easier learnable transformations to some much more complex techniques--- such as moment-matching formulation and Copula transformations, which showed promising results. In this presentation, we will introduce the methods we employed and review our some of our results from experiments.