RedHatAI/Inkling-NVFP4-FP8-BLOCK

This is a quantized version of thinkingmachines/Inkling with MoE layers quantized to NVFP4 and attention layers quantized to FP8 Block.

Creation Script

from transformers import (
    AutoProcessor,
    InklingForConditionalGeneration,
)
from compressed_tensors.quantization.quant_scheme import (
    FP8_BLOCK,
    NVFP4,
    QuantizationScheme,
)
from compressed_tensors.utils import save_mtp_tensors_to_checkpoint
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.utils import load_context

MODEL_ID = "thinkingmachines/Inkling"
SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-NVFP4-FP8-BLOCK"

with load_context(InklingForConditionalGeneration):
    model = InklingForConditionalGeneration.from_pretrained(
        MODEL_ID,
        max_memory={"cpu": 500e9},
        device_map="auto_offload",
        offload_folder="./offload_folder",
    )
    processor = AutoProcessor.from_pretrained(MODEL_ID)
    processor.tokenizer.eos_token = "<|endoftext|>"
    processor.tokenizer.pad_token = "<|endoftext|>"

# Configure the quantization algorithm to run.
#   * quantize attention weights to FP8_BLOCK
#   * quantize mlp/MoE weights to NVFP4
recipe = QuantizationModifier(
    config_groups={
        "config_group_0": QuantizationScheme(
            targets=[
                r"re:.*attn\..*",
            ],
            **FP8_BLOCK,
        ),
        "config_group_1": QuantizationScheme(
            targets=[
                r"re:.*mlp\..*",
            ],
            **NVFP4,
        ),
    },
    ignore=[
        "lm_head",
        "model.llm.unembed",
        "model.llm.embed",
        "re:.*sconv.*",
        "re:.*norm.*",
        "re:.*bias$",
        "re:.*gate$",
        "re:.*global_scale$",
        "re:.*shared_experts.*",
        "re:.*rel_logits_proj.*",
        "re:.*act_fn.*",
        "re:.*visual.*",
        "re:.*vision.*",
        "re:.*audio.*",
        "re:model.mtp.*",
    ],
)

# Select calibration dataset.
DATASET_ID = "ultrachat-200k"
DATASET_SPLIT = "train_sft"

NUM_CALIBRATION_SAMPLES = 256
MAX_SEQUENCE_LENGTH = 4096

# Apply algorithms.
oneshot(
    model=model,
    processor=processor,
    recipe=recipe,
    dataset=DATASET_ID,
    splits={"calibration": f"{DATASET_SPLIT}[:{NUM_CALIBRATION_SAMPLES}]"},
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
)

# Save to disk compressed.
model.save_pretrained(SAVE_DIR, save_compressed=True, save_original_format=False)
processor.save_pretrained(SAVE_DIR)

save_mtp_tensors_to_checkpoint(
    source_model=MODEL_ID, dest_dir=SAVE_DIR, mtp_prefix="model.mtp"
)

Usage

This model is intended for deployment with vLLM. You can serve the model using 4xB200s:

vllm serve RedHatAI/Inkling-NVFP4-FP8-BLOCK \
  --tokenizer-mode inkling \
  --tensor-parallel-size 4 \ 
  --enable-auto-tool-choice \
  --tool-call-parser inkling \
  --reasoning-parser inkling \
  --enable-expert-parallel \
  --max-model-len 100000 \
  --max-num-seqs 32 \
  --enable-chunked-prefill \
  --trust-remote-code \

Evaluation

Evaluations were performed with inspectai:

inspect eval hf/Idavidrein/gpqa/diamond \
  --model RedHatAI/Inkling-NVFP4-FP8-BLOCK \
  --reasoning-effort high \
  --model-base-url http://localhost:8000/v1 \

Benchmark	thinkingmachines/Inkling	RedHatAI/Inkling-NVFP4-FP8-BLOCK
GPQA Diamond	87.2	84.5

Note: A bug in inspectai currently misscores markdown-formatted multiple choice answers (e.g. ANSWER: $B$ rather than ANSWER: B), which the Inkling models often include in their response. Scores are manually evaluated to account for discrepancy. More information at https://github.com/UKGovernmentBEIS/inspect_ai/issues/5145