Skip to content

Add a subchannel FP8/INT8 quantized training example - #1024

Draft
PKUWZP wants to merge 1 commit into
masterfrom
subchannel-quantization-example
Draft

PKUWZP wants to merge 1 commit into
masterfrom
subchannel-quantization-example

Conversation

@PKUWZP

@PKUWZP PKUWZP commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Adds training/subchannel_quantization/, the companion example for deepspeedai/DeepSpeed#9131 (proposal: deepspeedai/DeepSpeed#9132). It should not merge before that PR does.

What it adds

  • train.py: continues training a pretrained causal LM on WikiText-103 with DeepSpeed, either as a dense model (data parallel) or as an MoE model with AutoEP. --recipe selects the quantization.subchannel settings: bf16 (baseline), e4m3_row, e4m3, int8, int8_h128. Every recipe sees the same data in the same order, and the script reports validation loss, median step time and peak memory.
  • README.md: prerequisites, the recipes, launch commands for a dense and an MoE model, and reference results.
  • requirements.txt: installs DeepSpeed from the tip of Add subchannel FP8/INT8 quantized training for nn.Linear and AutoEP experts DeepSpeed#9131.

Validation

Hardware: 4x NVIDIA H200, driver 595.112.07, torch 2.8.0+cu129, Triton 3.4.0, transformers 5.23.0.

  • The script as committed was run for 6 steps with e4m3_row on Qwen1.5-MoE-A2.7B (AutoEP) and on Qwen2.5-14B (--dense --grad_checkpointing --bf16_states); both completed and wrote their result files.
  • The reference results in the README were produced with an earlier version of the same script. The differences are cosmetic for those runs: model paths were hard-coded, gradient checkpointing was implied by --dense rather than a separate flag, and memory statistics came from torch.cuda instead of the accelerator interface.
  • Models were loaded from local copies. Loading by Hugging Face id, as the README commands do, was not run.
  • requirements.txt was not installed into a clean environment.

AI assistance

This example was written with AI assistance (Claude Code) and needs human review before it leaves draft.

Companion to deepspeedai/DeepSpeed#9131: continues training a causal LM (dense or AutoEP MoE) on WikiText-103 under BF16 and several quantization recipes, and reports validation loss, step time and peak memory.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Zhipeng Wang <7927074+PKUWZP@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant