Fraction of available stage-time slots without useful work.
Device timeline
Rows are pipeline stages. Colored blocks are microbatch work; empty cells expose bubbles caused by warmup, drain, or a slow stage.
Ready to schedule eight microbatches.
Completed microbatches per normalized time slot.
Slots from first forward work to final backward completion.
Illustrative saved activation memory across in-flight microbatches.
More devices do not help when the schedule leaves them waiting.
Model layers become pipeline stages.
Each device owns a contiguous partition of the model. A microbatch moves forward through every stage, then gradients travel backward in reverse order.
stage 1 -> stage 2 -> ... -> stage PMicrobatches amortize the bubble.
One batch leaves later stages idle during warmup and earlier stages idle during drain. Splitting it into more microbatches creates overlap and raises utilization.
ideal bubble fraction roughly (P - 1) / (M + P - 1)GPipe stores more activations.
All forward passes complete before backward starts, so many microbatch activations remain live simultaneously.
1F1B releases memory earlier.
After warmup, each stage alternates one forward and one backward task, reducing the number of stored activations.
The slowest stage sets the clock.
A partition with extra compute stretches its slots and forces neighboring stages to wait. Balanced partitions matter as much as the schedule.