Instructions to use inference-optimization/Qwen3.6-8B-A1.6B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use inference-optimization/Qwen3.6-8B-A1.6B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="inference-optimization/Qwen3.6-8B-A1.6B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("inference-optimization/Qwen3.6-8B-A1.6B")
model = AutoModelForMultimodalLM.from_pretrained("inference-optimization/Qwen3.6-8B-A1.6B", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use inference-optimization/Qwen3.6-8B-A1.6B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "inference-optimization/Qwen3.6-8B-A1.6B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Qwen3.6-8B-A1.6B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/inference-optimization/Qwen3.6-8B-A1.6B

SGLang

How to use inference-optimization/Qwen3.6-8B-A1.6B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "inference-optimization/Qwen3.6-8B-A1.6B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Qwen3.6-8B-A1.6B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "inference-optimization/Qwen3.6-8B-A1.6B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Qwen3.6-8B-A1.6B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use inference-optimization/Qwen3.6-8B-A1.6B with Docker Model Runner:
```
docker model run hf.co/inference-optimization/Qwen3.6-8B-A1.6B
```

Qwen3.6-8B-A1.6B

This is a tiny version of Qwen/Qwen3.6-35B-A3B created for testing and development.

Model Details

Base Model: Qwen/Qwen3.6-35B-A3B
Architecture: qwen3_5_moe (Qwen3_5MoeForConditionalGeneration)
Total Parameters: 7.81B
Activated Parameters: ~1.57B (8 of 256 routed experts per token, plus shared expert)

This is a multimodal (vision-language) mixture-of-experts model with a hybrid linear-attention / full-attention text backbone. The tiny model preserves the full architecture of the original: hybrid attention pattern (linear + full), 256 routed experts with top-8 routing, a shared expert, and the vision tower.

Configuration Changes

Only depth was reduced; all widths (hidden size, expert count, MoE intermediate size, attention head dims, vocab) match the original to keep the architecture faithful.

Parameter	Original	Tiny
`text_config.num_hidden_layers`	40	8
`text_config.layer_types`	30 linear + 10 full	6 linear + 2 full
`vision_config.depth`	27	2
`text_config.num_experts`	256	256 (unchanged)
`text_config.num_experts_per_tok`	8	8 (unchanged)
`text_config.moe_intermediate_size`	512	512 (unchanged)
`text_config.hidden_size`	2048	2048 (unchanged)
`text_config.vocab_size`	248320	248320 (unchanged)

The hybrid attention pattern (full_attention_interval=4) is preserved: text layers are [linear, linear, linear, full, linear, linear, linear, full], giving 6 linear-attention and 2 full-attention layers.

Checkpoint Structure

Sharded safetensors checkpoint with model.safetensors.index.json, matching the original repository layout. Routed experts are stored in the original fused format (one 3D tensor per layer):

...mlp.experts.gate_up_proj → [256, 1024, 2048]
...mlp.experts.down_proj → [256, 2048, 512]

All non-MTP tensor name patterns match the original checkpoint exactly. The MTP (multi-token-prediction) layers from the original are intentionally omitted.

Usage

import torch
from transformers import Qwen3_5MoeForConditionalGeneration, AutoTokenizer

model = Qwen3_5MoeForConditionalGeneration.from_pretrained(
    "inference-optimization/Qwen3.6-8B-A1.6B", dtype=torch.bfloat16, device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Qwen3.6-8B-A1.6B")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model claude skill.

Reduced text depth to 8 layers (preserving the hybrid attention pattern) and vision depth to 2, keeping all other dimensions.
Randomly initialized weights, then fine-tuned on a small toy text dataset until perplexity converged (train loss ≈ 0.02, perplexity ≈ 1.0).
Converted the fine-tuned checkpoint's per-expert tensors into the original fused-expert format and re-sharded to match the original repo layout.

Validation

Loads with Qwen3_5MoeForConditionalGeneration (no missing/unexpected keys).
Perplexity on the validation text ≈ 1.0 (target ≤ 10).
Greedy generation is coherent on the fine-tuning distribution.

Notes

The weights are randomly initialized and fine-tuned only on a tiny toy dataset; this model is for testing and development only and has no real-world language or vision capability.
The vision tower is included for architecture coverage but was not trained on image data.

Downloads last month: 202

Safetensors

Model size

8B params

Tensor type

BF16

Model tree for inference-optimization/Qwen3.6-8B-A1.6B

Base model

Qwen/Qwen3.6-35B-A3B

Finetuned

(256)

this model

Collection including inference-optimization/Qwen3.6-8B-A1.6B

Tiny Models

Collection

Tiny models used for testing • 22 items • Updated about 20 hours ago • 3