Differences

This shows you the differences between two versions of the page.

Link to this comparison view

in_layer_regularization [2016/11/23 19:45] (current)
Line 1: Line 1:
 +**Name** ​ Hidden Layer Regularization ​
  
 +**Intent**
 +
 +Add regularization terms that involve activations from inner layers.
 +
 +**Motivation**
 +
 +How can we influence the structure of the model patterns of the inner hidden layers?
 +
 +**References**
 +
 +
 +http://​arxiv.org/​abs/​1507.02672 Semi-Supervised Learning with Ladder Networks
 +
 +We combine supervised learning with unsupervised learning in deep neural networks. The proposed model is trained to simultaneously minimize the sum of supervised and unsupervised cost functions by backpropagation,​ avoiding the need for layer-wise pre-training. Our work builds on the Ladder network proposed by Valpola (2015), which we extend by combining the model with supervision.
 +
 +{{https://​ai2-s2-public.s3.amazonaws.com/​figures/​2016-11-01/​64fd6b7139be4b8b104a9aed768deaf62f71f5c0/​2-Figure1-1.png}}
 +
 +Note:  A regularization term is added to minimize the output of a layer with its corresponding decoder.  ​
 +
 +http://​arxiv.org/​pdf/​1607.00485v1.pdf ​ Group Sparse Regularization for
 +Deep Neural Networks
 +
 +We show that
 +a sparse version of the group Lasso penalty is able to achieve
 +competitive performances,​ while at the same time resulting in
 +extremely compact networks with a smaller number of input
 +features.
 +
 +http://​arxiv.org/​abs/​1607.02397v1 ​ Enlightening Deep Neural Networks
 +with Knowledge of Confounding Factors
 +
 +We incorporate information on
 +prominent auxiliary explanatory factors of the data
 +population into existing architectures as secondary
 +objective/​loss blocks that take inputs from hidden
 +layers during training. ​
 +
 +http://​www.ee.cuhk.edu.hk/​~xgwang/​papers/​ouyangWiccv13.pdf ​ Joint Deep Learning for Pedestrian Detection
 +
 +This paper proposes that they should be jointly
 +learned in order to maximize their strengths through cooperation.
 +We formulate these four components into a joint
 +deep learning framework and propose a new deep network
 +architecture.
 +
 +http://​arxiv.org/​pdf/​1412.7854v2.pdf Joint Deep Learning for Car Detection ​
 +
 +https://​arxiv.org/​abs/​1611.05134 Cost-Sensitive Deep Learning with Layer-Wise Cost Estimation
 +
 +we propose a novel framework that can be applied to deep neural networks with any structure to facilitate their learning of meaningful representations for cost-sensitive classification problems. Furthermore,​ the framework allows end- to-end training of deeper networks directly. The framework is designed by augmenting auxiliary neurons to the output of each hidden layer for layer-wise cost estimation, and including the total estimation loss within the optimization objective.
 +
 +CSDNN starts with a regular DNN with fully-connected layers, but replaces the softmax layer at the end of the DNN by a cost-estimation layer. Each of the K neurons in the cost-estimation layer provides per-class cost estimation with regression instead of per-class probability estimation.