research
∙
09/03/2023
FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs
The rapid growth of memory and computation requirements of large languag...
share
research
∙
08/07/2023
LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning
The low-rank adaptation (LoRA) method can largely reduce the amount of t...
research
∙
08/04/2023
Eva: A General Vectorized Approximation Framework for Second-order Optimization
Second-order optimization algorithms exhibit excellent convergence prope...
research
∙
07/04/2023
A Generic Multi-Player Transformation Algorithm for Solving Large-Scale Zero-Sum Extensive-Form Adversarial Team Games
Many recent practical and theoretical breakthroughs focus on adversarial...
research
∙
06/15/2023
Evaluation and Optimization of Gradient Compression for Distributed Deep Learning
To accelerate distributed training, many gradient compression methods ha...
research
∙
03/03/2023
FedML Parrot: A Scalable Federated Learning System via Heterogeneity-aware Scheduling on Sequential and Hierarchical Training
Federated Learning (FL) enables collaborations among clients for train m...
research
∙
02/24/2023
Decoupling the All-Reduce Primitive for Accelerating Distributed Deep Learning
Communication scheduling has been shown to be effective in accelerating ...
research
∙
11/30/2022
An Efficient Split Fine-tuning Framework for Edge and Cloud Collaborative Learning
To enable the pre-trained models to be fine-tuned with local data on edg...
research
∙
07/20/2022
EASNet: Searching Elastic and Accurate Network Architecture for Stereo Matching
Recent advanced studies have spent considerable human efforts on optimiz...
research
∙
06/30/2022
Scalable K-FAC Training for Deep Neural Networks with Distributed Preconditioning
The second-order optimization methods, notably the D-KFAC (Distributed K...
research
∙
06/06/2022
Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning
In federated learning (FL), model performance typically suffers from cli...
research
∙
05/19/2022
Nebula-I: A General Framework for Collaboratively Training Deep Learning Models on Low-Bandwidth Cloud Clusters
The ever-growing model size and scale of compute have attracted increasi...
research
∙
10/06/2021
FADNet++: Real-Time and Accurate Disparity Estimation with Configurable Networks
Deep neural networks (DNNs) have achieved great success in the area of c...
research
∙
07/14/2021
Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Tasks
Distributed training with synchronous stochastic gradient descent (SGD) ...
research
∙
01/14/2021
Automated Model Design and Benchmarking of 3D Deep Learning Models for COVID-19 Detection with Chest CT Scans
The COVID-19 pandemic has spread globally for several months. Because it...
research
∙
10/20/2020
Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters
Distributed training techniques have been widely deployed in large-scale...
research
∙
05/29/2020
Efficient Sparse-Dense Matrix-Matrix Multiplication on GPUs Using the Customized Sparse Storage Format
Multiplication of a sparse matrix to a dense matrix (SpDM) is widely use...
research
∙
05/27/2020
Communication-Efficient Distributed Deep Learning: Survey, Evaluation, and Challenges
In recent years, distributed deep learning techniques are widely deploye...
research
∙
03/24/2020
FADNet: A Fast and Accurate Network for Disparity Estimation
Deep neural networks (DNNs) have achieved great success in the area of c...
research
∙
03/10/2020
Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
Distributed deep learning becomes very common to reduce the overall trai...
research
∙
02/24/2020
Communication Contention Aware Scheduling of Multiple Deep Learning Training Jobs
Distributed Deep Learning (DDL) has rapidly grown its popularity since i...
research
∙
02/22/2020
Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selection
Distributed learning techniques such as federated learning have enabled ...
research
∙
12/18/2019
MG-WFBP: Merging Gradients Wisely for Efficient Communication in Distributed Deep Learning
Distributed synchronous stochastic gradient descent has been widely used...
research
∙
11/20/2019
Understanding Top-k Sparsification in Distributed Deep Learning
Distributed stochastic gradient descent (SGD) algorithms are widely depl...
research
∙
11/20/2019
Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees
To reduce the long training time of large deep neural network (DNN) mode...
research
∙
11/20/2019
Computer-Aided Clinical Skin Disease Diagnosis Using CNN and Object Detection Models
Skin disease is one of the most common types of human diseases, which ma...
research
∙
09/15/2019
Performance and Power Evaluation of AI Accelerators for Training Deep Learning Models
Deep neural networks (DNNs) have become widely used in many AI applicati...
research
∙
01/14/2019
A Distributed Synchronous SGD Algorithm with Global Top-k Sparsification for Low Bandwidth Networks
Distributed synchronous stochastic gradient descent (S-SGD) with data pa...
research
∙
11/27/2018
MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithms
Distributed synchronous stochastic gradient descent has been widely used...
research
∙
07/30/2018
Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
Synchronized stochastic gradient descent (SGD) optimizers with data para...
research
∙
05/10/2018
Modeling and Evaluation of Synchronous Stochastic Gradient Descent in Distributed Deep Learning on Multiple GPUs
With huge amounts of training data, deep learning has made great breakth...
research
∙
11/16/2017
Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs
Deep learning frameworks have been widely deployed on GPU servers for de...
research
∙
11/09/2017
