Parallax is a tool that automatically parallelizes training of a single-GPU deep learning model correctly and efficiently in distributed multi-GPU environments. Parallax correctly handles complicated auto-parallelization issues; in addition, it also leverages various optimizations to minimize communication overhead incurred by distributed training. If you are interested, you can find the technical details of Parallax in our arxiv paper.
Parallax is currently implemented on TensorFlow v1.6 and TensorFlow v1.11. In case that Parallax uses Message Passing Interface (MPI), Parallax requires AllReduce, AllGather operations implemented in Horovod v0.11.2. We plan to support multiple TensorFlow versions.
Parallax makes it easier for users to do distributed training of a deep learning model developed in a single device (e.g., GPU or CPU). A Parallax user simply specifies a single-device model graph, resource specification for distributed training and Parallax does the rest! For distributed training, Parallax supports two major communication styles: Parameter Server (PS) and Message Passing Interface (MPI). Users can choose which communication method to train their models, or Parallax can choose a communication method that works better automatically.
When a client initiates a deep learning job with a single-device computation graph, resource information, and optionally a flag that indicates either synchronous or asynchronous training, Parallax transforms the computation graph by analyzing its characteristics. Then, Parallax executes the transformed graph with its optimized communication layer in the distributed environment.
To give you an idea on how well Parallax performs, we present the following chart that shows the result of experiments done in a cluster of eight machines that are connected via Mellanox ConnectX-4 cards with 100Gbps InfiniBand. Each machine has six NVIDIA GeForce TITAN Xp GPU cards.
See the Troubleshooting page and submit a new issue or contact us if you cannot find an answer.
To contact us, send an email to [email protected].