A study on communication optimization of distributed gradient descent algorithms based on large-scale machine learning

Abstract

In recent years, the rapid development of new generation information technology has resulted in an unprecedented expansion
of information capacity. Machine learning algorithms are also increasingly used to compute information sets and build information systems
to solve problems whose complexity makes algorithmic solutions infeasible. Examples include autonomous vehicles, speech recognition
or user determination (recommendation systems). The complexity of the machine learning model, combined with the larger amount of data
collected, makes it much more expensive to use the model on a single machine, or even impossible to train. Using the computing power of
distributed systems is a straightforward, simple solution to the problem. Today, powerful computer clusters are used to train complex deep
neural networks on large data sets. However, in large-scale clustered environments, the commonly used distributed synchronous stochastic
gradient descent algorithms require frequent node communication to ensure consistency of the gradients (parameters). This has led to the
communication bandwidth being a key constraint for distributed machine learning systems.