Wan Xinyi
commited on
Commit
·
f8e95f6
1
Parent(s):
377fb85
Add links to README
Browse files
README.md
CHANGED
|
@@ -10,4 +10,15 @@ pinned: false
|
|
| 10 |
license: apache-2.0
|
| 11 |
---
|
| 12 |
|
| 13 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
license: apache-2.0
|
| 11 |
---
|
| 12 |
|
| 13 |
+
|
| 14 |
+
# Zero Bubble Pipeline Parallelism
|
| 15 |
+
|
| 16 |
+
Zero Bubble Pipeline Parallelism is a novel pipeline parallelism algorithm able to reduce the bubble of pipeline parallelism to almost zero while preserving synchronous semantics.
|
| 17 |
+
|
| 18 |
+
Check out our paper at:
|
| 19 |
+
* [Arxiv Version with ZBV](https://arxiv.org/abs/2401.10241)
|
| 20 |
+
* [ICLR Accepted version with ZB1P and ZB2P](https://openreview.net/pdf?id=tuzTN0eIO5)
|
| 21 |
+
|
| 22 |
+
Try out our implementation based on Megatron on [https://github.com/sail-sg/zero-bubble-pipeline-parallelism](https://github.com/sail-sg/zero-bubble-pipeline-parallelism)
|
| 23 |
+
|
| 24 |
+
Experiments shows zero bubble pipeline parallelism can accelerate training up to 30% with a similar memory comsumption. A detailed table of experiments is coming soon.
|