Repository navigation
Conversation
Companion to deepspeedai/DeepSpeed#9131: continues training a causal LM (dense or AutoEP MoE) on WikiText-103 under BF16 and several quantization recipes, and reports validation loss, step time and peak memory. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: Zhipeng Wang <7927074+PKUWZP@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
training/subchannel_quantization/, the companion example for deepspeedai/DeepSpeed#9131 (proposal: deepspeedai/DeepSpeed#9132). It should not merge before that PR does.What it adds
train.py: continues training a pretrained causal LM on WikiText-103 with DeepSpeed, either as a dense model (data parallel) or as an MoE model with AutoEP.--recipeselects thequantization.subchannelsettings:bf16(baseline),e4m3_row,e4m3,int8,int8_h128. Every recipe sees the same data in the same order, and the script reports validation loss, median step time and peak memory.README.md: prerequisites, the recipes, launch commands for a dense and an MoE model, and reference results.requirements.txt: installs DeepSpeed from the tip of Add subchannel FP8/INT8 quantized training for nn.Linear and AutoEP experts DeepSpeed#9131.Validation
Hardware: 4x NVIDIA H200, driver 595.112.07, torch 2.8.0+cu129, Triton 3.4.0, transformers 5.23.0.
e4m3_rowon Qwen1.5-MoE-A2.7B (AutoEP) and on Qwen2.5-14B (--dense --grad_checkpointing --bf16_states); both completed and wrote their result files.--denserather than a separate flag, and memory statistics came fromtorch.cudainstead of the accelerator interface.requirements.txtwas not installed into a clean environment.AI assistance
This example was written with AI assistance (Claude Code) and needs human review before it leaves draft.