You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
perf(cuda): enable Q8 MLP fusion in FLUX.3 - #2373
Split FLUX.3's packed mlp_in Q8_0 weight into gate and up views (ggml_view_2d) feeding ggml_swiglu_split, so the CUDA backend fuses both projections with SwiGLU via the existing Q8 MMQ path. Pass the active graph to get_context(gf) and call expand_graph on both views beforehand so the projections stay adjacent for fusion matching. Unsupported shapes, biases, weight adapters, and other backends keep the existing path.
Related Issue / Discussion
N/A
Additional Information
Tested on RTX 5070 Ti Laptop (sm_120, CUDA 13.4, Vulkan 1.4, Windows Release) against 532d2c1. FLUX.3 Image Klein Q8_0 (16 Euler steps, CFG 1, seeds 41/42, two interleaved runs per version) produces pixel-identical images across all runs on CUDA (1024x1024) and on the unmodified Vulkan fallback path (512x512).
benchmark
baseline
fused
diff
CUDA (1024x1024, 16 steps, 2nd-image sampling)
12.68 s
12.14 s
-4.3%
On Vulkan (512x512, 16 steps), where the fallback path keeps the original graph unchanged, outputs are pixel-identical with no regression (13.04 s mean vs 13.49 s baseline, within run-to-run variance).
LoRA, SD_USE_UPSTREAM_GGML, Metal, and ROCm were not benchmarked.
If this PR adds or changes model support, I have added or updated its usage guide and relevant model index/README entries following the model documentation requirements.
No actionable comments were generated in the recent review.
Walkthrough
Linear adds a conditional SwiGLU projection path with a fallback. Flux3 uses this method for its MLP input projection and passes the graph to get_context.
Changes: Flux3 SwiGLU projection
Layer / File(s)
Summary
Add Linear SwiGLU path src/model/common/ggml_block.hpp
Linear::forward_swiglu uses separate weight views and projections when its specified graph, backend, weight, and input conditions match. Otherwise, it applies SiLU to the existing linear output.
Use SwiGLU in Flux3 src/model/diffusion/flux3.hpp
Flux3::Block::forward calls forward_swiglu for mlp_in. Flux3Runner::compute passes gf to get_context.
FLUX.3 gains a CUDA Q8 MLP fusion path that falls back to the existing behavior for unsupported cases. No concrete defect was established. The reported outputs are pixel-identical, and the speedup is about 4%. Row-split multi-GPU setups and LoRA were not tested.
Pre-merge checks: 4
Check name
Status
Explanation
Title check
The title names the main change: enabling Q8 MLP fusion for FLUX.3 on CUDA through a SwiGLU split of the packed mlp_in weight. It follows the conventional commit style and is specific. The title is sh…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Description check
The description is mostly complete and follows the template. It has a Summary that explains the change (splitting the packed mlp_in Q8_0 weight into gate and up views, feeding ggml_swiglu_split, passi…
Full details: Title check
Explanation
The title names the main change: enabling Q8 MLP fusion for FLUX.3 on CUDA through a SwiGLU split of the packed mlp_in weight. It follows the conventional commit style and is specific. The title is short and fits in one sentence, so it meets the criteria.
Full details: Description check
Explanation
The description is mostly complete and follows the template. It has a Summary that explains the change (splitting the packed mlp_in Q8_0 weight into gate and up views, feeding ggml_swiglu_split, passing the graph to get_context(gf), and keeping the existing path for unsupported cases). It includes Additional Information with hardware, build reference, seeds, and benchmark tables for CUDA and Vulkan, plus stated untested configurations. The Related Issue / Discussion section is filled with N/A, which is acceptable. The checklist is ticked. The description does not link the PR objective's reference to PR #2373, but that is non-critical. The description is specific and technical, not vague.
Finishing Touches
Generate unit tests (beta)
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Repository UI
Review profile: CHILL
Plan: Advanced
Run ID: 638f67b8-d03e-4e01-8ff9-1b7e50a2ca37
Commits
Reviewing files that changed from the base of the PR and between 532d2c1 and 1418d2c.
Files selected for processing: 2
src/model/common/ggml_block.hpp
src/model/diffusion/flux3.hpp
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Split FLUX.3's packed
mlp_inQ8_0 weight into gate and up views (ggml_view_2d) feedingggml_swiglu_split, so the CUDA backend fuses both projections with SwiGLU via the existing Q8 MMQ path. Pass the active graph toget_context(gf)and callexpand_graphon both views beforehand so the projections stay adjacent for fusion matching. Unsupported shapes, biases, weight adapters, and other backends keep the existing path.Related Issue / Discussion
N/A
Additional Information
Tested on RTX 5070 Ti Laptop (sm_120, CUDA 13.4, Vulkan 1.4, Windows Release) against
532d2c1. FLUX.3 Image KleinQ8_0(16 Euler steps, CFG 1, seeds 41/42, two interleaved runs per version) produces pixel-identical images across all runs on CUDA (1024x1024) and on the unmodified Vulkan fallback path (512x512).1024x1024, 16 steps, 2nd-image sampling)On Vulkan (
512x512, 16 steps), where the fallback path keeps the original graph unchanged, outputs are pixel-identical with no regression (13.04 smean vs13.49 sbaseline, within run-to-run variance).LoRA,
SD_USE_UPSTREAM_GGML, Metal, and ROCm were not benchmarked.Checklist