Skip to content

perf(cuda): enable Q8 MLP fusion in FLUX.3 - #2373

Open
losewayy wants to merge 1 commit into
leejet:masterfrom
losewayy:perf/packed-mlp-glu
Open

losewayy wants to merge 1 commit into
leejet:masterfrom
losewayy:perf/packed-mlp-glu

Conversation

@losewayy

@losewayy losewayy commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Split FLUX.3's packed mlp_in Q8_0 weight into gate and up views (ggml_view_2d) feeding ggml_swiglu_split, so the CUDA backend fuses both projections with SwiGLU via the existing Q8 MMQ path. Pass the active graph to get_context(gf) and call expand_graph on both views beforehand so the projections stay adjacent for fusion matching. Unsupported shapes, biases, weight adapters, and other backends keep the existing path.

Related Issue / Discussion

N/A

Additional Information

Tested on RTX 5070 Ti Laptop (sm_120, CUDA 13.4, Vulkan 1.4, Windows Release) against 532d2c1. FLUX.3 Image Klein Q8_0 (16 Euler steps, CFG 1, seeds 41/42, two interleaved runs per version) produces pixel-identical images across all runs on CUDA (1024x1024) and on the unmodified Vulkan fallback path (512x512).

benchmark baseline fused diff
CUDA (1024x1024, 16 steps, 2nd-image sampling) 12.68 s 12.14 s -4.3%

On Vulkan (512x512, 16 steps), where the fallback path keeps the original graph unchanged, outputs are pixel-identical with no regression (13.04 s mean vs 13.49 s baseline, within run-to-run variance).

LoRA, SD_USE_UPSTREAM_GGML, Metal, and ROCm were not benchmarked.

Checklist

@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

Linear adds a conditional SwiGLU projection path with a fallback. Flux3 uses this method for its MLP input projection and passes the graph to get_context.

Changes: Flux3 SwiGLU projection

Layer / File(s) Summary
Add Linear SwiGLU path
src/model/common/ggml_block.hpp
Linear::forward_swiglu uses separate weight views and projections when its specified graph, backend, weight, and input conditions match. Otherwise, it applies SiLU to the existing linear output.
Use SwiGLU in Flux3
src/model/diffusion/flux3.hpp
Flux3::Block::forward calls forward_swiglu for mlp_in. Flux3Runner::compute passes gf to get_context.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Refactor

Possibly related PRs

Suggested reviewers: leejet

Merge Risk: Minimal · up to 1418d

FLUX.3 gains a CUDA Q8 MLP fusion path that falls back to the existing behavior for unsupported cases. No concrete defect was established. The reported outputs are pixel-identical, and the speedup is about 4%. Row-split multi-GPU setups and LoRA were not tested.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: enabling Q8 MLP fusion for FLUX.3 on CUDA through a SwiGLU split of the packed mlp_in weight. It follows the conventional commit style and is specific. The title is sh…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description is mostly complete and follows the template. It has a Summary that explains the change (splitting the packed mlp_in Q8_0 weight into gate and up views, feeding ggml_swiglu_split, passi…
Full details: Title check

Explanation

The title names the main change: enabling Q8 MLP fusion for FLUX.3 on CUDA through a SwiGLU split of the packed mlp_in weight. It follows the conventional commit style and is specific. The title is short and fits in one sentence, so it meets the criteria.

Full details: Description check

Explanation

The description is mostly complete and follows the template. It has a Summary that explains the change (splitting the packed mlp_in Q8_0 weight into gate and up views, feeding ggml_swiglu_split, passing the graph to get_context(gf), and keeping the existing path for unsupported cases). It includes Additional Information with hardware, build reference, seeds, and benchmark tables for CUDA and Vulkan, plus stated untested configurations. The Related Issue / Discussion section is filled with N/A, which is acceptable. The checklist is ticked. The description does not link the PR objective's reference to PR #2373, but that is non-critical. The description is specific and technical, not vague.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 638f67b8-d03e-4e01-8ff9-1b7e50a2ca37
Commits

Reviewing files that changed from the base of the PR and between 532d2c1 and 1418d2c.

Files selected for processing: 2

  • src/model/common/ggml_block.hpp
  • src/model/diffusion/flux3.hpp

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant