MS-AMP 0.1.0 Release Notes

MS-AMP package

• Support the new FP8 feature that is introduced by latest accelerators (e.g. H100).
• Speed up math-intensive operations, such as linear layers, by using Tensor Cores.
• Speed up memory-limited operations by accessing one byte compared to half or single-precision.
• Reduce memory requirements for training models, enabling larger models or larger minibatches.
• Speed up communication for distributed model by transmitting lower precision gradients.
• Support two optimization levels: O1 and O2.
• Support two optimizers: Adam and AdamW.

Examples using MS-AMP

• MNIST model
• DeiT model
• Swin-Transformer model