1. Motivation
AI hardware is improving faster than the software stack that is supposed to use it.
NVIDIA, AMD, and Intel all produce increasingly capable GPUs with better matrix acceleration, lower-precision compute, higher memory bandwidth, and stronger AI-oriented hardware. However, users still often have to choose a GPU based on software compatibility first and hardware capability second.
In practice, the first question is often not:
Which GPU has the best VRAM capacity, bandwidth, compute performance, power efficiency, and price?
Instead, it is:
Does this model, extension, quantization library, or attention kernel work with CUDA?
That is not a healthy long-term situation for a multi-vendor GPU market.
The goal of this proposal is **not to replace CUDA**.
CUDA, ROCm, and Intel XPU / oneAPI should continue to exist as optimized vendor backends.
The goal is to build a stronger common execution and memory-management layer above them so that application code, model code, and common AI workloads do not have to be tightly coupled to one GPU vendor.
2. Current Problems
Local AI users repeatedly encounter the same classes of problems:
- CUDA-only extensions
- incomplete or inconsistent ROCm support
- limited AMD support on Windows
- backend-specific differences in FlashAttention, xformers, bitsandbytes, and other libraries
- fragmented low-bit quantization implementations
- manual device mapping and offloading
- KV cache causing unexpected OOM failures
- temporary buffers consuming several additional gigabytes of VRAM
- dequantization buffers partially defeating the purpose of low-bit weights
- memory fragmentation
- GPUs with enough compute performance being unable to run a model because of VRAM limits
- CPU or NVMe offloading allowing a model to run, but often at unacceptable performance
The problem is therefore not just raw GPU compute.
The software stack needs better control over **memory placement, low-bit execution, backend abstraction, and runtime planning**.
3. A Concrete Target: 13B-Class Models on 12 GB GPUs
A useful practical target would be:
Make 13B-class INT4 inference practical on 12 GB consumer GPUs without requiring users to manually combine many separate libraries and backend-specific workarounds.
Approximate model weight sizes for a 13B model are:
- BF16: about 26 GB
- INT8: about 13 GB
- INT4: about 6.5 GB, plus scales and metadata
This means that INT4 weights can theoretically fit comfortably within 12 GB.
The real problem is everything else:
- KV cache
- activations
- attention workspace
- temporary tensors
- dequantization buffers
- kernel workspace
- allocator fragmentation
So the main challenge is not merely quantizing the weights. It is controlling the **entire runtime memory footprint**.
4. Memory-Budget-Aware Execution
Instead of forcing users to manually tune `device_map`, `max_memory`, offload folders, quantization settings, and backend-specific flags, the runtime should accept a simple memory budget.
For example:
```python
model = load_model(
path=“model”,
device=“gpu”,
memory_budget=“11GB”,
weight_dtype=“int4”,
kv_cache_dtype=“int8”,
offload=“auto”,
)
```
The runtime should automatically decide:
- which layers remain in local VRAM
- which tensors can use lower precision
- which tensors should be offloaded
- whether some layers should use INT4, INT8, FP8, or BF16
- which KV cache precision should be used
- which attention implementation should be selected
- which operations should be fused
- how much workspace can be allocated
- how temporary buffers should be reused
- which vendor backend should be selected
The user should be able to say:
My GPU has 12 GB. Make the best execution plan that fits.
The framework should then perform the planning.
5. Stronger Vendor-Neutral Execution
An ideal architecture could look like this:
```text
Application / Transformers / Local LLM Runtime
|
v
PyTorch Graph
|
v
Common GPU Runtime / IR
|
±----------±----------+
| | |
v v v
CUDA ROCm XPU
NVIDIA AMD Intel
```
Again, this does not require removing CUDA.
CUDA would remain the NVIDIA backend.
ROCm would remain the AMD backend.
XPU / oneAPI would remain the Intel backend.
The difference is that more model logic, memory logic, quantization logic, and kernel definitions would exist **above** those vendor-specific layers.
Ideally, users could write:
```python
model.to(“gpu”)
```
and the runtime would internally choose:
```text
NVIDIA → CUDA
AMD → ROCm
Intel → XPU / oneAPI
```
without requiring major changes in application code.
6. Low-Bit Data Types Should Be First-Class Runtime Features
INT4, INT8, FP8, and mixed precision should not behave like optional tricks provided by unrelated third-party libraries.
They should be first-class execution and memory-management concepts.
Examples include:
- INT4 weight-only quantization
- INT8 KV cache
- FP8 activations
- per-layer mixed precision
- per-block precision selection
- keeping sensitive layers in BF16 while storing most weights in INT4
- higher precision only for the output head or specific attention blocks
A critical point is that an INT4 model should not repeatedly allocate large BF16 copies of its weights during execution.
The preferred path should look more like:
```text
INT4 Weight
|
v
Fused Dequant + MatMul
|
v
BF16 / FP16 Output
```
rather than:
```text
INT4 Weight
|
v
Temporary BF16 Weight Copy
|
v
MatMul
```
The latter can destroy much of the memory advantage of quantization.
7. KV Cache Should Be Treated as a Core Memory Problem
Reducing weight size is not enough.
As context length grows, KV cache can become one of the largest consumers of VRAM.
A modern runtime should support:
- FP16 KV cache
- FP8 KV cache
- INT8 KV cache
- INT4 KV cache
- paged KV cache
- selective migration of old KV blocks to slower memory
- attention-window-aware retention
- prefix cache sharing
- automatic KV precision adjustment based on VRAM budget
For example, when given an 11 GB memory budget, the runtime might automatically decide:
```text
Weights: INT4
KV Cache: INT8
Attention: memory-efficient implementation
Context Length: automatically limited to available memory
```
This should be planned before execution, not after an OOM crash.
8. Hierarchical Memory
Current consumer systems effectively look like this:
```text
GPU VRAM
|
PCIe
|
System RAM
|
Storage
```
For AI workloads, a more explicit hierarchical memory model would be useful:
```text
Tier 0: GPU Local VRAM
Tier 1: GPU-Side Expansion Memory
Tier 2: System RAM
Tier 3: NVMe / mmap Storage
```
Tier 0: GPU Local VRAM
The fastest memory.
Best suited for:
- currently active layers
- hot KV cache
- attention working sets
- frequently accessed tensors
- temporary compute buffers
Tier 1: GPU-Side Expansion Memory
A future large-capacity memory tier physically close to the GPU.
Possible capacities:
- 32 GB
- 64 GB
- 128 GB
It may be slower than local VRAM but much faster and lower-latency than system RAM over PCIe.
Possible uses:
- inactive transformer layers
- cold KV cache
- adapters
- multimodal encoder weights
- rarely accessed blocks of a large model
Tier 2: System RAM
Large and relatively inexpensive, but slower and farther from the GPU.
Useful for less frequently accessed tensors and overflow capacity.
Tier 3: NVMe
A last-resort storage tier.
Useful for:
- model startup
- cold storage
- rarely used weights
- checkpoint data
- emergency offloading
It should not be treated as a substitute for fast working memory.
9. GPU-Side Expansion Memory
Consumer GPUs continue to gain compute performance faster than they gain VRAM capacity.
This creates situations where a GPU has enough compute throughput for a model but cannot hold the model and its runtime state.
One possible hardware direction would be:
```text
GPU
|
±- 12 GB High-Speed VRAM
|
±- 32-64 GB Expansion Memory
|
±- Dedicated High-Speed Link
```
The expansion memory does not need to be as fast as local GDDR or HBM.
However, it should be significantly better than ordinary system RAM or NVMe for GPU access.
Useful properties would include:
- high bandwidth
- low latency
- direct GPU access
- unified addressing
- coherent or semi-coherent memory management
- asynchronous page migration
- driver-level placement control
- framework-visible memory tiers
In such a design, 12 GB of VRAM could act as the high-speed working set while a much larger pool stores the rest of the model.
This could be especially valuable for local AI workloads.
10. Why NVMe Offloading Is Not Enough
NVMe offloading can prevent complete failure when a model is too large.
It does not solve the performance problem.
If every layer requires data to be read from storage and transferred to the GPU, then execution becomes transfer-bound:
```text
Compute Time << Data Transfer Time
```
At that point, additional GPU compute performance is mostly wasted.
NVMe is useful for:
- cold storage
- startup loading
- checkpoints
- infrequently accessed blocks
- emergency overflow
It is not an ideal primary memory tier for interactive inference.
The real solution is to reduce how much data must cross slow links during execution.
11. What PyTorch Could Improve
If PyTorch continues evolving toward a broader AI execution runtime, the following features could provide significant value.
11.1 Memory Budget API
Example:
```python
torch.set_memory_budget(“11GB”)
```
or a model-specific equivalent.
11.2 Hierarchical Tensor Placement
The runtime could understand placements such as:
```text
gpu_local
gpu_extended
system_ram
mmap
```
and move tensors automatically.
11.3 Native Low-Bit Tensor Support
Low-bit formats should be directly represented and optimized:
- INT4
- INT8
- FP8
- block quantization
- quantized KV cache
- mixed-precision tensors
11.4 Vendor-Neutral Kernel IR
An attention, quantization, or matrix-multiplication kernel could be defined once and compiled to:
- NVIDIA
- AMD
- Intel
backend-specific implementations.
11.5 Automatic Kernel Selection
The runtime should automatically select between:
- Flash-style attention
- memory-efficient attention
- fused MLP
- fused dequantization + matrix multiplication
- paged attention
- low-bit GEMM
depending on available hardware and memory.
11.6 Automatic Replanning Before OOM
Instead of crashing first and forcing the user to change settings, the runtime could estimate memory requirements before execution.
For example:
```text
Expected Usage: 13.8 GB
Available Budget: 11.0 GB
Replanned Execution:
- Layers 0-18 → local GPU memory
- Layers 19-31 → expansion memory or system RAM
- KV cache → INT8
- Attention workspace → memory-efficient mode
- Temporary buffer reuse → enabled
```
This is a better user experience than trial-and-error OOM debugging.
12. What Hugging Face Transformers Could Improve
From a Transformers user perspective, the ideal interface could be extremely simple:
```python
model = AutoModel.from_pretrained(
model_id,
device=“auto”,
memory_budget=“11GB”,
precision=“auto”,
)
```
Internally, the framework could coordinate:
- PyTorch
- torch.compile / Inductor
- TorchAO
- Triton
- CUDA
- ROCm
- Intel XPU
- quantization
- offloading
- KV cache
- attention backend selection
Today, users often have to read documentation for several separate libraries and manually combine them.
That complexity should gradually move into the runtime itself.
13. The Goal Is Not to Eliminate CUDA
This point is important.
CUDA is a mature and highly optimized NVIDIA backend.
There is no need to remove it.
The problem is that CUDA compatibility is often treated as a requirement for using a model, extension, or kernel at all.
The ideal market would allow users to choose GPUs based mainly on:
- VRAM capacity
- memory bandwidth
- compute performance
- power efficiency
- price
rather than mainly asking:
Does this require CUDA?
If NVIDIA provides the best hardware, users can choose NVIDIA.
If AMD provides more VRAM at a better price, users should be able to choose AMD without losing major software capabilities.
If Intel offers competitive hardware, it should also be a viable option.
That is healthier competition.
14. Expected Market Effect
A stronger common runtime could change how consumer GPUs compete.
Current situation:
```text
Software compatibility → GPU vendor decision
```
Preferred situation:
```text
Price / VRAM / Bandwidth / Efficiency / Performance → GPU vendor decision
```
This would encourage all three major GPU vendors to compete on actual hardware value while still maintaining their own optimized backend stacks.
The result should be better for users, developers, and the broader AI ecosystem.
15. Possible Roadmap
Stage 1
- stronger INT4 / INT8 / FP8 tensor support
- standard low-bit KV cache APIs
- memory-budget APIs
- backend-independent attention APIs
Stage 2
- common kernel IR for CUDA / ROCm / XPU
- vendor-neutral quantization kernels
- automatic memory planning
- OOM prediction before execution
Stage 3
- hierarchical system-memory support
- paged tensors
- asynchronous tensor migration
- improved direct memory access
Stage 4
- standardized GPU-side expansion memory support
- hardware memory-tier APIs
- runtime-controlled tensor migration
- stronger multi-vendor execution portability
16. Suggested Success Criteria
Test A
Run a 13B-class INT4 model on a 12 GB NVIDIA GPU.
Requirements:
- minimal manual device mapping
- automatic KV cache optimization
- useful context length
- no OOM during normal inference
Test B
Run the same model code on a 16 GB AMD GPU.
The application code should not require major modification.
Test C
Run the same model code on an Intel GPU.
Backend differences should remain mostly inside the framework.
Test D
When local VRAM is insufficient, automatically move selected tensors to another memory tier.
The user should not have to manually assign every layer.
17. Short Forum Version
Title
**Proposal: Vendor-neutral, memory-budget-aware execution for consumer GPUs**
Post
I would like to suggest a direction for PyTorch, Transformers, and local LLM runtimes.
Today, consumer GPU users face two major problems:
- Software support is fragmented between NVIDIA CUDA, AMD ROCm, and Intel XPU.
- VRAM is often the real bottleneck even when the GPU has enough compute performance.
A 12 GB GPU may have enough compute power for a 13B-class model, especially with INT4 weights, but KV cache, temporary buffers, backend-specific kernels, and fragmented memory allocation can still cause OOM.
I think frameworks should move toward a **memory-budget-aware execution model**.
For example:
```python
model = load_model(
path,
device=“gpu”,
memory_budget=“11GB”,
weight_dtype=“int4”,
kv_cache_dtype=“int8”,
offload=“auto”,
)
```
The runtime should automatically decide:
- which layers stay in local VRAM
- which tensors use lower precision
- how KV cache is stored
- which attention implementation is used
- which kernels are fused
- whether system RAM or another memory tier is used
- which backend is selected for NVIDIA, AMD, or Intel
The goal is **not to replace CUDA**.
CUDA, ROCm, and XPU can remain optimized vendor backends.
The goal is to build a stronger common execution layer above them so that model code does not need to be heavily rewritten or restricted to one GPU vendor.
I also think frameworks should prepare for hierarchical GPU memory:
```text
Tier 0: GPU local VRAM
Tier 1: future GPU-side expansion memory
Tier 2: system RAM
Tier 3: NVMe / mmap
```
NVMe offloading alone is not a real performance solution because bandwidth and latency are much worse than local GPU memory.
A future GPU-side expansion-memory tier could allow a fast 12 GB consumer GPU to use a much larger model without relying on system RAM or SSD as the primary working memory.
A concrete target could be:
Make 13B-class INT4 inference practical on 12 GB consumer GPUs without requiring users to manually combine many separate libraries and backend-specific workarounds.
There are already many relevant technologies:
- PyTorch
- torch.compile / Inductor
- TorchAO
- Triton
- Transformers
- Accelerate
- FlashAttention
- bitsandbytes
- llama.cpp-style quantization
However, these capabilities are still fragmented.
I think a future major version of the stack would be more valuable if it focused on unified memory management, low-bit inference, and hardware-neutral execution.
The end goal should be simple:
Users should choose GPUs based on VRAM, bandwidth, compute performance, power efficiency, and price — not primarily because a particular model or kernel only works on one vendor.
I would be interested to hear whether there are existing projects, RFCs, or research efforts moving in this direction.
18. One-Sentence Summary
**AI frameworks should hide GPU-vendor differences, automatically optimize models for a fixed VRAM budget, and prepare for future GPU-side expansion memory so that 12 GB consumer GPUs can run larger models efficiently.**