Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
invariant_representation [2018/05/20 11:04]
admin
invariant_representation [2018/12/21 18:19] (current)
admin
Line 397: Line 397:
 propose a covariance-insured screening methodology to identify predictors that are propose a covariance-insured screening methodology to identify predictors that are
 jointly informative but only marginally weakly associated with outcomes. jointly informative but only marginally weakly associated with outcomes.
 +
 +http://​www.uvm.edu/​~cdanfort/​courses/​237/​schmidt-lipson-2009.pdf Distilling Free-Form Natural Laws
 +from Experimental Data
 +
 +https://​arxiv.org/​abs/​1805.12491 Structure from noise: Mental errors yield abstract representations of events
 +
 +https://​arxiv.org/​abs/​1807.04689 Explorations in Homeomorphic Variational Auto-Encoding
 +
 +In this paper we investigate the use of manifold-valued latent variables. Specifically,​ we focus on the important case of continuously differentiable symmetry groups (Lie groups), such as the group of 3D rotations SO(3). We show how a VAE with SO(3)-valued latent variables can be constructed,​ by extending the reparameterization trick to compact connected Lie groups. Our experiments show that choosing manifold-valued latent variables that match the topology of the latent data manifold, is crucial to preserve the topological structure and learn a well-behaved latent space.
 +
 +https://​arxiv.org/​abs/​1808.05563 Learning Invariances using the Marginal Likelihood
 +
 +We argue that invariances should instead be incorporated in the model structure, and learned using the marginal likelihood, which correctly rewards the reduced complexity of invariant models.
 +
 +https://​arxiv.org/​abs/​1706.01350 Emergence of Invariance and Disentanglement in Deep Representations
 +
 +We propose regularizing the loss by bounding such a term in two equivalent ways: One with a Kullbach-Leibler term, which relates to a PAC-Bayes perspective;​ the other using the information in the weights as a measure of complexity of a learned model, yielding a novel Information Bottleneck for the weights. Finally, we show that invariance and independence of the components of the representation learned by the network are bounded above and below by the information in the weights, and therefore are implicitly optimized during training. The theory enables us to quantify and predict sharp phase transitions between underfitting and overfitting of random labels when using our regularized loss, which we verify in experiments,​ and sheds light on the relation between the geometry of the loss function, invariance properties of the learned representation,​ and generalization error.
 +
 +https://​arxiv.org/​pdf/​1809.02601v1.pdf Accelerating Deep Neural Networks with Spatial Bottleneck Modules
 +
 +This paper presents an efficient module named spatial
 +bottleneck for accelerating the convolutional layers in deep
 +neural networks. The core idea is to decompose convolution
 +into two stages, which first reduce the spatial resolution
 +of the feature map, and then restore it to the desired size.
 +This operation decreases the sampling density in the spatial
 +domain, which is independent yet complementary to
 +network acceleration approaches in the channel domain.
 +Using different sampling rates, we can tradeoff between
 +recognition accuracy and model complexity
 +
 +https://​arxiv.org/​abs/​1809.02591v1 Learning Invariances for Policy Generalization
 +
 +While recent progress has spawned very powerful machine learning systems, those agents remain extremely specialized and fail to transfer the knowledge they gain to similar yet unseen tasks. In this paper, we study a simple reinforcement learning problem and focus on learning policies that encode the proper invariances for generalization to different settings. We evaluate three potential methods for policy generalization:​ data augmentation,​ meta-learning and adversarial training. We find our data augmentation method to be effective, and study the potential of meta-learning and adversarial learning as alternative task-agnostic approaches. ​
 +
 +https://​openreview.net/​forum?​id=Ske25sC9FQ Robustness and Equivariance of Neural Networks
 +
 +Robustness to rotations comes at the cost of robustness of pixel-wise adversarial perturbations.
 +
 +https://​arxiv.org/​abs/​1809.10083v1 Unsupervised Adversarial Invariance
 +
 +We present a novel unsupervised invariance induction framework for neural networks that learns a split representation of data through competitive training between the prediction task and a reconstruction task coupled with disentanglement,​ without needing any labeled information about nuisance factors or domain knowledge. We describe an adversarial instantiation of this framework and provide analysis of its working. Our unsupervised model outperforms state-of-the-art methods, which are supervised, at inducing invariance to inherent nuisance factors, effectively using synthetic data augmentation to learn invariance, and domain adaptation. Our method can be applied to any prediction task, eg., binary/​multi-class classification or regression, without loss of generality.
 +
 +disentanglement is achieved between e1 and e2 in a novel way through two adversarial disentanglers
 +— one that aims to predict e2 from e1 and another that does the inverse.
 +
 +https://​openreview.net/​forum?​id=BklHpjCqKm Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning ​