research
∙
06/07/2023
Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning
In this paper, we first present an explanation regarding the common occu...
share
research
∙
06/05/2023
Aiming towards the minimizers: fast convergence of SGD for overparametrized problems
Modern machine learning paradigms, such as deep learning, occur in or cl...
research
∙
06/05/2023
On Emergence of Clean-Priority Learning in Early Stopped Neural Networks
When random label noise is added to a training dataset, the prediction e...
research
∙
05/15/2023
ReLU soothes the NTK condition number and accelerates optimization for wide neural networks
Rectified linear unit (ReLU), as a non-linear activation function, is we...
research
∙
05/24/2022
Quadratic models for understanding neural network dynamics
In this work, we propose using a quadratic model as a tool for understan...
research
∙
05/24/2022
Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecture
In this paper we show that feedforward neural networks corresponding to ...
research
∙
03/10/2022
Transition to Linearity of Wide Neural Networks is an Emerging Property of Assembling Weak Models
Wide neural networks with linear output layer have been shown to be near...
research
∙
12/08/2021
Hyper-parameter optimization based on soft actor critic and hierarchical mixture regularization
Hyper-parameter optimization is a crucial problem in machine learning as...
research
∙
10/02/2020
On the linearity of large non-linear models: when and why the tangent kernel is constant
The goal of this work is to shed light on the remarkable phenomenon of t...
research
∙
02/29/2020
Toward a theory of optimization for over-parameterized systems of non-linear equations: the lessons of deep learning
The success of deep learning is due, to a great extent, to the remarkabl...
research
∙
10/31/2018
MaSS: an Accelerated Stochastic Method for Over-parametrized Learning
In this paper we introduce MaSS (Momentum-added Stochastic Solver), an a...
research
∙
02/28/2018
