research
∙
05/28/2022
ByteComp: Revisiting Gradient Compression in Distributed Training
Gradient compression (GC) is a promising approach to addressing the comm...
share
research
∙
05/05/2022
dPRO: A Generic Profiling and Optimization System for Expediting Distributed DNN Training
Distributed training using multiple devices (e.g., GPUs) has been widely...
research
∙
05/17/2021
Compressed Communication for Distributed Training: Adaptive Methods and System
Communication overhead severely hinders the scalability of distributed m...
research
∙
07/26/2020
CSER: Communication-efficient SGD with Error Reset
The scalability of Distributed Stochastic Gradient Descent (SGD) is toda...
research
∙
06/24/2020
Accelerated Large Batch Optimization of BERT Pretraining in 54 minutes
BERT has recently attracted a lot of attention in natural language under...
research
∙
06/17/2020
Is Network the Bottleneck of Distributed Training?
Recently there has been a surge of research on improving the communicati...
research
∙
04/19/2020
ResNeSt: Split-Attention Networks
While image classification models have recently continued to advance, mo...
research
∙
11/20/2019
Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates
Recent years have witnessed the growth of large-scale distributed machin...
research
∙
07/09/2019
GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing
We present GluonCV and GluonNLP, the deep learning toolkits for computer...
research
∙
04/26/2019
Dynamic Mini-batch SGD for Elastic Distributed Training: Learning in the Limbo of Resources
With an increasing demand for training powers for deep learning algorith...
research
∙
04/16/2019
