Distributed Deep Learning with ChainerMN

ChainerMN enables multi-node distributed deep learning with the following features:

• Scalable — it makes full use of the latest technologies such as NVIDIA NCCL and CUDA-Aware MPI,

• Flexible — even dynamic neural networks can be trained in parallel thanks to Chainer’s flexibility, and

• Easy — minimal changes to existing user code are required.

This blog post provides our benchmark results using up to 128 GPUs.

ChainerMN can be used for both inner-node (i.e., multiple GPUs inside a node) and inter-node settings. For inter-node settings, we highly recommend to use high-speed interconnects such as InfiniBand.

ChainerMN examples are available on GitHub. These examples are based on the examples of Chainer and the differences are highlighted.

• Installation
  • Installation Guide
  • Step-by-Step Troubleshooting
• Tutorial
  • Overview
  • Step 1: Communicators and Optimizers
  • Step 2: Datasets and Evaluators
  • Tips and FAQs
• Model Parallel
  • Overview
  • Model Parallel on ChainerMN
  • Example 1: Simple MLP
  • Example 2: seq2seq
  • Example 3: Channel-wise Parallel Convolution
  • Example 4: Ensemble
• API Reference
  • Communicators
  • Optimizers and Evaluators
  • Dataset Utilities
  • Links
  • Functions
  • Iterators
  • Trainer extensions
  • Configurations