Skip to content

MiniMax-H3 on 16 GB cards: layer-split OOM from never-released weight staging, CUDA chunk-budget no-op, f32 activation floor #2371

Open

Description

@Koreprobro
  • Project: leejet/stable-diffusion.cpp
  • Version tested: master-1106-532d2c1 (commit 532d2c1, right after the opaque-options refactor refactor(api): adopt opaque options, tensors, and results #2365). Baseline comparison against the previously packaged master-1104-5dcfa79.
  • Hardware: 4 × RTX 5070 Ti (16303 MiB each), PCIe, DDR5 ≥ 96 GB
  • Model: MiniMax-H3 AV (DiT video): minimax_h3_ref2va_pruned-Q8_0.gguf (20.5 GB weights, 53 transformer blocks), minimax_h3_video_vae_fp16.safetensors, Sakura TE (Q4_K_M, ~14 GB)
  • Reported: 2026-11-17, during development of our own project on top of stable-diffusion.cpp
  • Status: all four issues diagnosed with root cause; item 1 and item 2 already have working local patches (validated below; the working patch is provided in the first comment of this issue), item 3/item 4 are design gaps with suggested direction.

1. MAIN BUG — layer-split sessions OOM mid-run: weight staging copies are never released (lazy eviction + pin-order trap)

Symptom

With the model fitting via --offload-to-cpu and a split backend
(diffusion=cuda0&cuda1 --split-mode layer), short generations succeed (24f,
672×384). At HD 124f the run dies at the same place every time,
mid-sampling
:

[E] ggml: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 764.86 MiB on device 1: cudaMalloc failed: out of memory
[E] minimax_h3 compute failed: GGML status: error (failed to allocate memory) --- execution_engine.cpp:479
[E] minimax_h3 segment 37/54 (minimax_h3.blocks.33) failed during execution

Nothing about the failing block is special — it is simply the point where the
second card runs out. Per-segment capacity logs (free bytes at segment start,
device 1, HD 124f, blocks .30 → .33):

free 2708 MB → 2314 MB → 1918 MB → 1524 MB   (−394 MB per block executed, monotonic)
resident(tracked weights) 3150 → 3544 → 3938 → 4332 MB   (+394 MB per block, never freed)

394 MB is exactly one Q8_0 block. The device accumulates every block it has
ever executed
, plus the prefetched next block, until real allocations
overflow. The per-segment capacity check keeps "passing" because its estimate
(≈9.1 GB pending vs ≈10.1 GB free) still fits on paper; the actual
cudaMalloc during execution then fails with ~1 GB of phantom headroom.

Root cause

Two mechanisms combine:

  1. Eviction is pressure-only. Device-side staging copies of weights
    (ModelManager::compute_staging_blocks_) are released from
    release_compute_staging_blocks(false) / evict_compute_backend_block() —
    which are only reached from the make_enough_memory() failure path
    (execution_engine.cpp ~925) and from unregister/teardown. Normal segment
    completion never releases them. On the home card this is masked (the buffer
    cache reuses per-device staging buffers and the pressure loop caps growth at
    ~4.9 GB in our single-card traces); on a secondary split device every block
    copy just accumulates.

  2. The pin-order trap. prepare_params() pins the segment's tensors
    (state->pin_count++, model_manager.cpp ~1615). The pins are released by
    SegmentWeightPipeline::segment_end() →
    release_compute_backend_params(), which is called from the next
    segment's segment_start()
    (activate() → segment_end(),
    segment_weight_pipeline.cpp ~46/~106). So at every moment when eviction is
    attempted under pressure — i.e. during the next segment's preparation —
    the just-executed block (the ideal eviction candidate) is still pinned, and
    the fully_evictable checks in
    evict_compute_backend_params()/release_compute_staging_blocks() reject
    the block. The manager therefore cannot evict precisely the thing it needs
    to evict; "under pressure" arrives too late and the allocator — not the
    estimator — takes the hit.

Local fix (validated)

Eagerly evict the previous segment's staging copies at the point where their
pins are guaranteed to be gone — right after weights.segment_start(index) in
GraphExecution::run() (execution_engine.cpp):

if (!weights.segment_start(index, ensure_capacity)) {
    return fail_segment("weight preparation");
}
if (segmented && index > 0 && manager != nullptr) {
    // segment_start() has just released the previous segment's weight pins;
    // its device staging copies are now the cheapest memory on the card.
    // Release them before this segment allocates its workspace. The
    // prefetched segment lives in separate staging blocks and is pinned
    // out of eviction.
    manager->evict_compute_backend_params(weights.params(index - 1));
}

evict_compute_backend_params() is already strict (it frees only staging
blocks that are fully unpinned and intersect the target tensors), so this
cannot evict the current block or the prefetch window.

Validation (patched build, all runs --vae-tiling --sage-attn --offload-to-cpu --rng cpu)

Run Config Result
HD 124f, 1 GPU minimax_h3_mlp_chunk_budget_gib=1 done, 2m38s / 2 steps
HD 124f, split 2 same + --split-mode layer done, 3m10s / 2 steps
HD 124f, split 3 same done, 3m14s / 2 steps
HD 124f hires (draft 672×384 + HD refine), 1 vs split 2 same both done; SSIM = 1.000000
24f 672×384, split 2/3/4 vs baseline unpatched chunking SSIM = 1.000000
Regression: qwen-image / krea2 / moss t2i on patched binary defaults all done, no behavior change

Cost: staged block weights are re-uploaded from RAM per sampling step
(~394 MB/block at Q8_0). Observed overhead for split vs single card is
+10…20%, which is the price of capacity, not of the eviction fix itself.
A refinement worth considering upstream: keep an LRU window (e.g. 2–4 blocks)
instead of evicting at distance 1, to cut re-uploads across steps.


2. The seq-chunking machinery is a no-op on CUDA (ggml_backend_buft_get_max_size() = SIZE_MAX), so block intermediates are monolithic

minimax_h3.hpp (and the same idiom elsewhere) sizes sequence chunking from
the backend buffer-type maximum allocation size:

size_t chunk_budget = ggml_backend_buft_get_max_size(buft);  // CUDA: SIZE_MAX
chunk_budget -= chunk_budget % alignment;
...
chunk_size = min(seq_len, chunk_budget / token_bytes);       // => never chunks

On CUDA the buft max size is effectively unbounded, so both
FeedForward::forward() and Attention::project_qkv() fall back to
monolithic intermediates even though the chunked code path exists and works
(it was clearly written for backends like Vulkan/Metal with allocation caps).
Measured on HD 264f (seq ≈ 82.7k tokens, f32 graph):

  • attn.qkv_proj MUL_MAT output: 6780 MB in one buffer, followed by a
    6860 MB CONCAT (text-cond merge) — ~13.6 GB of live intermediates for one
    projection before attention even starts;
  • mlp.fc1 output: a multi-GB monolith until budgeted.

Two knobs were needed to make this configurable (local patch, --model-args):

  • minimax_h3_mlp_chunk_budget_gib — caps chunk_budget for both
    FeedForward::forward() and Attention::project_qkv() (shared block-level
    token-chunk budget);
  • the existing minimax_h3_attn_chunk_budget (head grouping) was already
    budgeted — the projection path simply wasn't.

Worth noting for whoever tunes estimates: capping FFN alone changed the
scheduler's pending estimate by exactly 0 bytes
(22248.99 MB before and
after, byte-identical) — the ggml sched-reserve simulation already reuses FFN
scratch perfectly. The estimate only moved when qkv projection chunking
was capped (22.2 → 21.0 GB). The monolithic qkv+concat, not the MLP, is the
real wall on CUDA.

Suggested upstream direction: give the chunk budget a first-class config
surface (per-model model-args at minimum) instead of deriving it purely from
buft_get_max_size(), and default it to something sane on CUDA for long-seq
DiTs (H3, Wan, LTX-class).


3. Activations are hardwired to f32; no bf16 knob — the per-block workspace floor blocks whole canvas classes

The H3 runner builds every intermediate in f32. There is no
--model-args/CLI knob for activation dtype (--type and
--tensor-type-rules are weight-only). Consequence — measured per-block
workspace floor (segment blocks.0, single device, chunk budgets already at
0.25 GiB, i.e. the floor is the seq-proportional q/k/v totals + residual
streams, not scratch):

Canvas seq tokens pending / real reserve fits 16 GB?
HD 124f (1344×768) ~41 k ~9.5 GB yes (with item 2's knobs)
HD 264f ~83 k 18.5 GB real (ggml_gallocr_reserve_n_impl single-arena attempt) no
FHD 124f (1920×1088) ~84 k ~22 GB no
FHD 264f ~165 k ~40 GB no

bf16 activations would halve this floor (HD 264f → ~9–10 GB, FHD 124f →
~11–12 GB) and make these classes feasible on 16 GB cards. This is not exotic:
neighboring runners in the same tree already use f16 opportunistically
(qwen_image_2_1.hpp f16 prefix KV cache under flash-attn, wan.hpp f16
loader defaults), and the vendor's own H3 artifacts are fp16/bf16. The H3
runner is simply still all-f32. A minimax_h3_act_dtype=bf16 (or a global
activation-dtype policy with per-op f32 accumulation guarantees) is the
suggested shape.

Note this also bounds what multi-GPU layer split can do at all: activation
workspace is per-block and per-token — it does not divide across cards. Layer
split fixes weight capacity; item 3 (or sequence parallelism) is what fixes
long-sequence capacity. The current need 22760.99 MB ... available 15127.11 MB wall on HD 264f is identical on master-1104 and master-1106
with 1 or 4 cards — not a regression, just a hard floor.


4. Row split mode is unusable on CUDA: "backend has no split buffer type"

--split-mode row fails during session setup on CUDA backends (the split
buffer type lookup has no CUDA path), leaving layer as the only usable mode
for diffusion=cuda0&cuda1. Layer split, per item 3, cannot split activations —
so today there is no way at all to scale a too-long sequence across cards. Not
blocking our envelope, but it removes the obvious answer to item 3's floor.


5. Minor observations (DX / estimator honesty)

  1. Failure phase mislabeling. The capacity gate that trips first is
    reached through weights.segment_start(index, ensure_capacity), but it is
    reported as failed during weight preparation — the message suggests weight
    I/O, while the actual reject is the workspace capacity estimate. The real
    phase only becomes visible after bypassing it ("workspace capacity check").
    Suggest distinct phase names (ensure_capacity vs weights load).
  2. Check vs allocator mismatch. The check's pending (sched-reserve
    estimate) and the real single-arena gallocr reserve differ materially
    (22.25 GB estimate vs 18.55 GB actual for the same HD 264f segment); and
    the check passes while the actual allocator dies later with ~1 GB of
    apparent headroom (cublas workspace / fragmentation are excluded by design:
    "Backend scratch buffers and pipelines are not included in graph
    measurements"
    , model_manager.cpp ~1792). A per-backend dynamic scratch
    allowance would make the gate predictive.
  3. Hardcoded 512 MB safety margin (model_manager.cpp ~1793) is a big
    fraction of a 16 GB card; a CLI knob (or auto from total/free) would help
    small-VRAM users.
  4. log_details in check_capacity() almost never fires (only on final
    failure), so tuning conversations start blind; a
    --log-level verbose-gated per-segment capacity line (pending/free/resident
    per backend) was invaluable for this diagnosis and is cheap to keep.

Suggested default for 16 GB cards (H3), after these fixes

--offload-to-cpu --vae-tiling --sage-attn \
--model-args minimax_h3_mlp_chunk_budget_gib=1,minimax_h3_attn_chunk_budget=0.5

HD 124f runs on one card; diffusion=cuda0&cuda1 --split-mode layer adds
weight headroom bit-identically (SSIM 1.000000 across 2/3-card splits, direct
and hires paths). HD 264f+ / FHD need item 3 (bf16 activations) or item 4 (row/seq
parallelism) to exist on 16 GB.


Filed by Qwen3.8-Next-Flash 180B (AI assistant) on behalf of the user. This investigation was conducted while building our own project on top
of stable-diffusion.cpp; full per-segment logs are available and we are glad
to test any upstream fix against this workload.

Activity

  1. Koreprobro commented on Nov 17, 2026

    @Koreprobro
    Author

    Working patch against 532d2c1 (validated: HD 124f direct + layer-split 2/3 cards + hires path, SSIM 1.000000; qwen-image / krea2 / moss regression clean):

    diff --git a/src/core/execution_engine.cpp b/src/core/execution_engine.cpp
    index 867fc73..423c824 100644
    --- a/src/core/execution_engine.cpp
    +++ b/src/core/execution_engine.cpp
    @@ -527,7 +527,8 @@ namespace sd {
             const auto params       = collect_used_param_tensors(graph);
             const auto& cached_plan = resolve_graph_cut_plan(graph);
             auto manager            = manager_;
    -        const bool can_segment  = !is_multi_device() && !sd_backend_is_cpu(session_.backend_) &&
    +        const bool can_segment  = (!is_multi_device() || session_.layer_split_enabled_) &&
    +                                  !sd_backend_is_cpu(session_.backend_) &&
                                       manager != nullptr && manager->segmented_compute_enabled() &&
                                       cached_plan.valid && cached_plan.has_cuts && cached_plan.segments.size() > 1;
             // An empty workspace request is a lower bound on the full graph's memory demand.
    @@ -577,6 +578,16 @@ namespace sd {
                             plan.segments.size(), plan.segments.size() == 1 ? "" : "s");
             }
             SegmentGraphBindings bindings(session_.cut_cache_, plan, graph);
    +        auto backend_for_tensor       = [&](const ggml_tensor* tensor) -> ggml_backend_t {
    +            auto it = session_.node_assignments_.find(tensor);
    +            return it == session_.node_assignments_.end() ? session_.backend_ : it->second;
    +        };
    +        auto segment_backend          = [&](const ggml_graph_cut::Segment& seg) -> ggml_backend_t {
    +            if (!session_.node_assignments_.empty() && !seg.internal_node_indices.empty()) {
    +                return backend_for_tensor(ggml_graph_node(graph, seg.internal_node_indices.front()));
    +            }
    +            return session_.backend_;
    +        };
             SegmentWeightPipeline weights(manager, session_.backend_, session_.owner_id(),
                                           graph, plan, params_tensor_set_,
                                           segmented && manager != nullptr && manager->prefetch_enabled());
    @@ -598,7 +609,7 @@ namespace sd {
                 };
                 session_.cut_cache_.prune(segment.live_cut_names);
                 bindings.reset(segment);
    -            if (!bindings.bind_cached_inputs(segment, task_.description.c_str())) {
    +            if (!bindings.bind_cached_inputs(segment, task_.description.c_str(), segmented ? segment_backend(segment) : nullptr)) {
                     return fail_segment("input binding");
                 }
                 ggml_context* segment_context = nullptr;
    @@ -626,7 +637,7 @@ namespace sd {
                     for (int output_index : segment.output_node_indices) {
                         auto tensor = ggml_graph_node(graph, output_index);
                         if (ggml_graph_cut::is_graph_cut_tensor(tensor) && segment.future_cut_names.count(tensor->name) &&
    -                        GraphCutTensorCache::can_bind_output(session_.backend_, tensor)) {
    +                        GraphCutTensorCache::can_bind_output(segmented ? backend_for_tensor(tensor) : session_.backend_, tensor)) {
                             cache_outputs.insert(tensor);
                         }
                     }
    @@ -657,6 +668,12 @@ namespace sd {
                 if (!weights.segment_start(index, ensure_capacity)) {
                     return fail_segment("weight preparation");
                 }
    +            if (segmented && index > 0 && manager != nullptr) {
    +                // segment_start released the previous segment's weight pins; now its device
    +                // staging copies are free to drop before this segment allocates its workspace.
    +                // Prefetched staging for later segments is pinned out of eviction by the pipeline.
    +                manager->evict_compute_backend_params(weights.params(index - 1));
    +            }
                 // Preparing weights can execute LoRA graphs and reclaim an idle workspace.
                 if (!session_.workspace_.prepare(measurement)) {
                     last_compute_status_ = GGML_STATUS_ALLOC_FAILED;
    @@ -665,7 +682,7 @@ namespace sd {
                 if (!ensure_capacity()) {
                     return fail_segment("workspace capacity check");
                 }
    -            if (!session_.cut_cache_.bind_outputs(cache_outputs)) {
    +            if (!session_.cut_cache_.bind_outputs(cache_outputs, segmented ? backend_for_tensor : GraphCutTensorCache::BackendResolver{})) {
                     last_compute_status_ = GGML_STATUS_ALLOC_FAILED;
                     return fail_segment("output cache allocation");
                 }
    @@ -697,7 +714,8 @@ namespace sd {
                 }
                 auto cache_status = session_.cache_.capture(segment_graph);
                 if (cache_status == GGML_STATUS_SUCCESS) {
    -                cache_status = session_.cut_cache_.capture(graph, segment, task_.description.c_str());
    +                cache_status = session_.cut_cache_.capture(graph, segment, task_.description.c_str(),
    +                                                           segmented ? backend_for_tensor : GraphCutTensorCache::BackendResolver{});
                 }
                 if (cache_status != GGML_STATUS_SUCCESS) {
                     last_compute_status_ = cache_status;
    @@ -742,11 +760,12 @@ namespace sd {
     
         size_t ExecutionSession::resident_bytes(ggml_backend_t backend) const {
             size_t bytes = workspace_.bytes(backend);
    -        if (backend == backend_) {
    -            const auto add = [](size_t a, size_t b) { return b > SIZE_MAX - a ? SIZE_MAX : a + b; };
    -            bytes          = add(bytes, cache_.resident_bytes(ggml_backend_get_device(backend)));
    -            bytes          = add(bytes, cut_cache_.resident_bytes(ggml_backend_get_device(backend)));
    -        }
    +        const auto add = [](size_t a, size_t b) { return b > SIZE_MAX - a ? SIZE_MAX : a + b; };
    +        // Cut and runner caches track their own buffer devices; count them per backend so
    +        // layer-split residency on secondary devices is visible to capacity checks.
    +        const auto dev = ggml_backend_get_device(backend);
    +        bytes          = add(bytes, cache_.resident_bytes(dev));
    +        bytes          = add(bytes, cut_cache_.resident_bytes(dev));
             return bytes;
         }
     
    diff --git a/src/core/runner_cache.cpp b/src/core/runner_cache.cpp
    index 9bbcae0..cdbc624 100644
    --- a/src/core/runner_cache.cpp
    +++ b/src/core/runner_cache.cpp
    @@ -159,8 +159,53 @@ namespace sd {
             return cached_tensor(tensors_, name);
         }
     
    +    ggml_tensor* GraphCutTensorCache::local_copy(ggml_backend_t consumer, const std::string& name, ggml_tensor* source) {
    +        auto& per_backend = local_copies_[consumer];
    +        auto found        = per_backend.find(name);
    +        if (found != per_backend.end()) {
    +            return found->second->tensor;
    +        }
    +        ggml_status status;
    +        auto entry = CachedTensor::copy(consumer, name, source, status);
    +        if (entry == nullptr || status != GGML_STATUS_SUCCESS) {
    +            LOG_ERROR("graph cut cache failed to copy tensor to consumer backend: %s", name.c_str());
    +            per_backend.erase(name);
    +            return nullptr;
    +        }
    +        ggml_tensor* result = entry->tensor;
    +        per_backend[name]   = std::move(entry);
    +        return result;
    +    }
    +
    +    ggml_tensor* GraphCutTensorCache::get_local(ggml_backend_t consumer, const std::string& name) {
    +        ggml_tensor* tensor = get(name);
    +        if (tensor == nullptr || consumer == nullptr) {
    +            return tensor;
    +        }
    +        ggml_backend_buffer_t buffer = ggml_graph_cut::tensor_buffer(tensor);
    +        if (buffer == nullptr) {
    +            return tensor;
    +        }
    +        ggml_backend_dev_t source_device   = ggml_backend_buft_get_device(ggml_backend_buffer_get_type(buffer));
    +        ggml_backend_dev_t consumer_device = ggml_backend_get_device(consumer);
    +        if (source_device == nullptr || source_device == consumer_device) {
    +            return tensor;
    +        }
    +        return local_copy(consumer, name, tensor);
    +    }
    +
    +    void GraphCutTensorCache::invalidate_local_copies(const std::string& name) {
    +        for (auto& per_backend : local_copies_) {
    +            per_backend.second.erase(name);
    +        }
    +    }
    +
         size_t GraphCutTensorCache::resident_bytes(ggml_backend_dev_t device) const {
    -        return sd::resident_bytes(tensors_, device);
    +        size_t bytes = sd::resident_bytes(tensors_, device);
    +        for (const auto& per_backend : local_copies_) {
    +            bytes += sd::resident_bytes(per_backend.second, device);
    +        }
    +        return bytes;
         }
     
         bool GraphCutTensorCache::can_bind_output(ggml_backend_t backend, const ggml_tensor* tensor) {
    @@ -168,16 +213,22 @@ namespace sd {
                    ggml_is_contiguous(tensor) && ggml_backend_supports_op(backend, tensor);
         }
     
    -    bool GraphCutTensorCache::bind_outputs(const std::unordered_set<const ggml_tensor*>& outputs) {
    +    bool GraphCutTensorCache::bind_outputs(const std::unordered_set<const ggml_tensor*>& outputs,
    +                                           const BackendResolver& backend_for_output) {
             for (const ggml_tensor* output : outputs) {
                 auto tensor = const_cast<ggml_tensor*>(output);
    -            auto entry  = CachedTensor::allocate(backend_, tensor->name, tensor);
    +            ggml_backend_t target = backend_for_output ? backend_for_output(tensor) : nullptr;
    +            if (target == nullptr) {
    +                target = backend_;
    +            }
    +            auto entry = CachedTensor::allocate(target, tensor->name, tensor);
                 if (entry == nullptr) {
                     return false;
                 }
                 tensor->buffer         = entry->tensor->buffer;
                 tensor->data           = entry->tensor->data;
                 tensor->extra          = entry->tensor->extra;
    +            invalidate_local_copies(tensor->name);
                 tensors_[tensor->name] = std::move(entry);
             }
             return true;
    @@ -215,11 +266,17 @@ namespace sd {
             for (auto it = tensors_.begin(); it != tensors_.end();) {
                 it = keep_names.count(it->first) ? std::next(it) : tensors_.erase(it);
             }
    +        for (auto& per_backend : local_copies_) {
    +            for (auto it = per_backend.second.begin(); it != per_backend.second.end();) {
    +                it = keep_names.count(it->first) ? std::next(it) : per_backend.second.erase(it);
    +            }
    +        }
         }
     
         ggml_status GraphCutTensorCache::capture(ggml_cgraph* graph,
                                                  const ggml_graph_cut::Segment& segment,
    -                                             const char* log_desc) {
    +                                             const char* log_desc,
    +                                             const BackendResolver& backend_for_output) {
             size_t copied_bytes = 0;
             size_t copied_count = 0;
             for (int index : segment.output_node_indices) {
    @@ -232,8 +289,12 @@ namespace sd {
                 if (cached != nullptr && output->buffer == cached->buffer && output->data == cached->data) {
                     continue;
                 }
    +            ggml_backend_t target = backend_for_output ? backend_for_output(output) : nullptr;
    +            if (target == nullptr) {
    +                target = backend_;
    +            }
                 ggml_status status;
    -            auto entry = CachedTensor::copy(backend_, output->name, ggml_graph_cut::cache_source_tensor(output), status);
    +            auto entry = CachedTensor::copy(target, output->name, ggml_graph_cut::cache_source_tensor(output), status);
                 if (entry == nullptr) {
                     LOG_ERROR("%s failed to capture graph cut tensor: %s", log_desc, output->name);
                     return status;
    @@ -241,6 +302,7 @@ namespace sd {
                 const size_t size = ggml_backend_buffer_get_size(entry->buffer);
                 copied_bytes      = size > SIZE_MAX - copied_bytes ? SIZE_MAX : copied_bytes + size;
                 ++copied_count;
    +            invalidate_local_copies(output->name);
                 tensors_[output->name] = std::move(entry);
             }
             ggml_backend_synchronize(backend_);
    diff --git a/src/core/runner_cache.h b/src/core/runner_cache.h
    index b6fe1c7..2b62e47 100644
    --- a/src/core/runner_cache.h
    +++ b/src/core/runner_cache.h
    @@ -1,6 +1,7 @@
     #ifndef __SD_CORE_RUNNER_CACHE_H__
     #define __SD_CORE_RUNNER_CACHE_H__
     
    +#include <functional>
     #include <map>
     #include <memory>
     #include <string>
    @@ -52,19 +53,30 @@ namespace sd {
         class GraphCutTensorCache {
             ggml_backend_t backend_;
             CachedTensors tensors_;
    +        // Consumer-side local copies for cross-device cut consumption (layer split).
    +        std::map<ggml_backend_t, CachedTensors> local_copies_;
    +        ggml_tensor* local_copy(ggml_backend_t consumer, const std::string& name, ggml_tensor* source);
    +        void invalidate_local_copies(const std::string& name);
     
         public:
    +        using BackendResolver = std::function<ggml_backend_t(const ggml_tensor*)>;
    +
             explicit GraphCutTensorCache(ggml_backend_t backend)
                 : backend_(backend) {}
             ggml_tensor* get(const std::string& name) const;
    +        // Cached tensor usable by a consumer on `consumer`: the shared entry when it already
    +        // lives on that device, otherwise a maintained local copy on the consumer's device.
    +        ggml_tensor* get_local(ggml_backend_t consumer, const std::string& name);
             size_t resident_bytes(ggml_backend_dev_t device) const;
             size_t estimate_output_bytes(ggml_cgraph* graph,
                                          const ggml_graph_cut::Segment& segment) const;
             static bool can_bind_output(ggml_backend_t backend, const ggml_tensor* tensor);
    -        bool bind_outputs(const std::unordered_set<const ggml_tensor*>& outputs);
    -        ggml_status capture(ggml_cgraph* graph, const ggml_graph_cut::Segment& segment, const char* log_desc);
    +        bool bind_outputs(const std::unordered_set<const ggml_tensor*>& outputs,
    +                          const BackendResolver& backend_for_output = nullptr);
    +        ggml_status capture(ggml_cgraph* graph, const ggml_graph_cut::Segment& segment, const char* log_desc,
    +                            const BackendResolver& backend_for_output = nullptr);
             void prune(const std::unordered_set<std::string>& keep_names);
    -        void clear() { tensors_.clear(); }
    +        void clear() { tensors_.clear(); local_copies_.clear(); }
         };
     }
     
    diff --git a/src/core/segment_graph_bindings.cpp b/src/core/segment_graph_bindings.cpp
    index ba72116..7ebb1d4 100644
    --- a/src/core/segment_graph_bindings.cpp
    +++ b/src/core/segment_graph_bindings.cpp
    @@ -7,6 +7,7 @@
     #include "core/runner_cache.h"
     #include "core/util.h"
     #include "ggml.h"
    +#include "ggml-backend.h"
     
     namespace sd {
         SegmentGraphBindings::SegmentGraphBindings(GraphCutTensorCache& tensor_cache,
    @@ -88,7 +89,8 @@ namespace sd {
     
         bool SegmentGraphBindings::bind_cached_inputs(
             const ggml_graph_cut::Segment& segment,
    -        const char* log_desc) {
    +        const char* log_desc,
    +        ggml_backend_t consumer_backend) {
             std::unordered_map<ggml_tensor*, ggml_tensor*> cached_view_sources;
             for (const auto& input : segment.input_refs) {
                 if (input.type != ggml_graph_cut::Segment::INPUT_PREVIOUS_CUT) {
    @@ -98,7 +100,7 @@ namespace sd {
                 if (input_tensor == nullptr) {
                     continue;
                 }
    -            ggml_tensor* cached_tensor = tensor_cache_.get(input.display_name);
    +            ggml_tensor* cached_tensor = tensor_cache_.get_local(consumer_backend, input.display_name);
                 if (cached_tensor == nullptr) {
                     LOG_ERROR("%s missing graph cut cache tensor: %s",
                               log_desc,
    diff --git a/src/core/segment_graph_bindings.h b/src/core/segment_graph_bindings.h
    index 08aba15..b9990f5 100644
    --- a/src/core/segment_graph_bindings.h
    +++ b/src/core/segment_graph_bindings.h
    @@ -3,6 +3,7 @@
     
     #include <array>
     #include <unordered_map>
    +#include "ggml-backend.h"
     #include "ggml.h"
     
     struct ggml_backend_buffer;
    @@ -27,7 +28,8 @@ namespace sd {
             void restore();
             ~SegmentGraphBindings() { restore(); }
             bool bind_cached_inputs(const ggml_graph_cut::Segment& segment,
    -                                const char* log_desc);
    +                                const char* log_desc,
    +                                ggml_backend_t consumer_backend = nullptr);
     
         private:
             struct ExternalBinding {
    diff --git a/src/model/diffusion/minimax_h3.hpp b/src/model/diffusion/minimax_h3.hpp
    index 2f5cb4f..13af03d 100644
    --- a/src/model/diffusion/minimax_h3.hpp
    +++ b/src/model/diffusion/minimax_h3.hpp
    @@ -23,6 +23,12 @@ namespace MiniMaxH3 {
     
         constexpr int H3_GRAPH_SIZE                   = 131072;
         constexpr float DEFAULT_ATTN_CHUNK_BUDGET_GIB = 2.f;
    +    // Optional token-chunk budget for the MiniMax-H3 MLP projections (GiB; 0 disables extra chunking).
    +    // Set via --model-args minimax_h3_mlp_chunk_budget_gib=<GiB>. Process-wide by design: one model per process.
    +    static inline float& h3_mlp_chunk_budget_gib() {
    +        static float value = 0.f;
    +        return value;
    +    }
         constexpr float FRAME_RESCALE                 = 5.f / 3.f;
         constexpr float VISUAL_COND_TIMESTEP          = 0.999f;
     
    @@ -276,6 +282,11 @@ namespace MiniMaxH3 {
                     size_t alignment    = ggml_backend_buft_get_alignment(buft);
                     size_t chunk_budget = ggml_backend_buft_get_max_size(buft);
                     chunk_budget -= chunk_budget % alignment;
    +                if (h3_mlp_chunk_budget_gib() > 0.f) {
    +                    size_t cap = static_cast<size_t>(h3_mlp_chunk_budget_gib() * (1024.f * 1024.f * 1024.f));
    +                    cap       -= cap % alignment;
    +                    chunk_budget = std::min(chunk_budget, cap);
    +                }
                     // Use the backend's preferred allocation size as a conservative chunk budget.
                     // Matmul outputs are F32 even for quantized weights.
                     size_t token_bytes = ggml_row_size(GGML_TYPE_F32, std::max(x->ne[0], 2 * ffn_hidden_size)) * x->ne[2] * x->ne[3];
    @@ -366,6 +377,11 @@ namespace MiniMaxH3 {
                     // This is a conservative chunk budget, not an operation support limit.
                     size_t chunk_budget = ggml_backend_buft_get_max_size(buft);
                     chunk_budget -= chunk_budget % alignment;
    +                if (h3_mlp_chunk_budget_gib() > 0.f) {
    +                    size_t cap = static_cast<size_t>(h3_mlp_chunk_budget_gib() * (1024.f * 1024.f * 1024.f));
    +                    cap       -= cap % alignment;
    +                    chunk_budget = std::min(chunk_budget, cap);
    +                }
                     size_t token_bytes = ggml_row_size(GGML_TYPE_F32, 3 * heads * head_dim) * inputs[0]->ne[2] * inputs[0]->ne[3];
                     chunk_size         = static_cast<int64_t>(std::min<size_t>(chunk_size, std::max<size_t>(1, chunk_budget / token_bytes)));
                 }
    @@ -1257,6 +1273,14 @@ namespace MiniMaxH3 {
                         } else {
                             LOG_WARN("ignoring invalid MiniMax-H3 model arg '%s=%s': expected a finite non-negative GiB budget", key.c_str(), value.c_str());
                         }
    +                } else if (key == "minimax_h3_mlp_chunk_budget_gib") {
    +                    float parsed = 0.f;
    +                    if (parse_strict_float(value, parsed) && std::isfinite(parsed) && parsed >= 0.f) {
    +                        h3_mlp_chunk_budget_gib() = parsed;
    +                        LOG_INFO("MiniMax-H3 MLP chunk budget: %.3g GiB (0 disables token chunking)", parsed);
    +                    } else {
    +                        LOG_WARN("ignoring invalid MiniMax-H3 model arg '%s=%s': expected a finite non-negative GiB budget", key.c_str(), value.c_str());
    +                    }
                     }
                 }
                 std::vector<GGMLBlock*> blocks;
    diff --git a/src/model_manager.cpp b/src/model_manager.cpp
    index bfdf262..99a4bf7 100644
    --- a/src/model_manager.cpp
    +++ b/src/model_manager.cpp
    @@ -1799,6 +1799,12 @@ ModelManager::CapacityCheck ModelManager::check_capacity(
     #endif
         result.required_device_bytes = add(add(request.pending_allocation_bytes, missing), safety_margin);
         result.required_budget_bytes = add(request.runtime_peak_bytes(), missing);
    +    if (log_details) {
    +        LOG_WARN("check_capacity %s: pending %.2f MB + missing %.2f MB (%zu states) + margin",
    +                 ggml_backend_name(request.compute_backend),
    +                 request.pending_allocation_bytes / (1024.0 * 1024.0),
    +                 missing / (1024.0 * 1024.0), states.size());
    +    }
         auto available_device_bytes  = [&](ggml_backend_t backend) {
             auto device = ggml_backend_get_device(backend);
             if (device == nullptr) {
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions