Data-dependent Initializations of Convolutional Neural Networks

TitleData-dependent Initializations of Convolutional Neural Networks
Publication TypeJournal Article
Year of Publication2015
AuthorsKrahenbuhl, P., Doersch C., Donahue J., & Darrell T.
Published inCoRR
Volumeabs/1511.06856
Date Published11/2015
Abstract

Convolutional Neural Networks spread through computer vision like a wildfire, impacting almost all visual tasks imaginable. Despite this, few researchers dare to train their models from scratch. Most work builds on one of a handful of ImageNet pre-trained models, and fine-tunes or adapts these for specific tasks. This is in large part due to the difficulty of properly initializing these networks from scratch. A small miscalibration of the initial weights leads to vanishing or exploding gradients, as well as poor convergence properties. In this work we present a fast and simple data-dependent initialization procedure, that sets the weights of a network such that all units in the network train at roughly the same rate, avoiding vanishing or exploding gradients. Our initialization matches the current state-of-the-art unsupervised or self-supervised pre-training methods on standard computer vision tasks, such as image classification and object detection, while being roughly three orders of magnitude faster. When combined with pre-training methods, our initialization significantly outperforms prior work, narrowing the gap between supervised and unsupervised pre-training.

URLhttp://www.icsi.berkeley.edu/pubs/vision/datadependentinitializations15.pdf
ICSI Research Group

Vision