Changelog

v0.1.0 (22/03/2022)

New Features:

• Support Data Parallelism

• Support 1D Tensor Parallelism

• Support Pipeline Parallelism

• Unified distributed Layers for both single-GPU and multi-GPU training

• LazyConfig system for more flexible syntax and no predefined structures

• Easy-to-use trainer and engine

• Support both CV and NLP data processing

• Mixed Precision Training

• Activation Checkpointing

• Gradient Accumulation

• Gradient Clipping

• Zero Redundancy Optimizer (ZeRO)

Supported Models:

• Support 3D parallel BERT model

• Support 3D parallel GPT-2 model

• Support 3D parallel T5 model

• Support 3D parallel Vision Transformer model

• Support Data parallel Swin Transformer model

• Support finetune task in QQP project

• Support text classification task in text classification project