LiBai Model Zoo

To date, LiBai implements the following models:

• Vision Transformer

• Swin Transformer

• BERT

• T5

• GPT-2

Parallelism Mode in LiBai

A collection of parallel training strategies is supported in LiBai:

• Data Parallel Training

• Tensor Parallel Training

• Pipeline Parallel Training

You can refer to oneflow official tutorial for a better understanding of the basic conception of parallelization techniques.

Supported Model in LiBai

For more details about the supported parallelism training on different models, please refer the following tables:

Model	Data Parallel	Tensor Parallel	Pipeline Parallel
Vision Transformer	✔	✔	✔
Swin Transformer	✔	-	-
BERT	✔	✔	✔
T5	✔	✔	✔
GPT-2	✔	✔	✔

Additions: ✔ means you can train this model under specific parallelism techniques or combine two or three of them with ✔ for 2D or 3D paralleism training.

Benchmarks (coming soon)

Here we provided our benchmark test results of the supported models in LiBai.