| **distributed_data_parallel.py** and **run.sh** show an example using Amp with | |
| [apex.parallel.DistributedDataParallel](https://nvidia.github.io/apex/parallel.html) or | |
| [torch.nn.parallel.DistributedDataParallel](https://pytorch.org/docs/stable/nn.html#distributeddataparallel) | |
| and the Pytorch multiprocess launcher script, | |
| [torch.distributed.launch](https://pytorch.org/docs/master/distributed.html#launch-utility). | |
| The use of `Amp` with DistributedDataParallel does not need to change from ordinary | |
| single-process use. The only gotcha is that wrapping your model with `DistributedDataParallel` must | |
| come after the call to `amp.initialize`. Test via | |
| ```bash | |
| bash run.sh | |
| ``` | |
| **This is intended purely as an instructional example, not a performance showcase.** | |