---
license: mit
base_model:
  - Qwen/Qwen3.8-Flash-Next
tags:
  - qwen4_exp
  - fixture
  - testing
  - moe
  - linear-attention
pipeline_tag: text-generation
---

# Qwen3.8-Flash-Next tiny architecture fixture (qwen4_exp)

A ~0.29M-parameter random-init TEXT checkpoint plus a reduced config that keeps the
real **Qwen3.8-Flash-Next** `qwen4_exp` field names, so loaders, quant planners,
and CI jobs can exercise the new architecture's config parsing, expert-table
sizing, and safetensors load path without the real weights.

## What this is
- Base model: `Qwen/Qwen3.8-Flash-Next` (released 2026-08-24),
  `model_type: qwen4_exp`, `architectures: ["Qwen4ExpForConditionalGeneration"]`.
  The base is a large multimodal MoE with a hybrid linear+full attention stack,
  ngram vocab, an attention indexer, PLE, hyper-connections, and an MTP head, so it
  cannot be instantiated in a unit test, in CI, or on a laptop.
- What this fixture is: a byte-reproducible **random-init** text checkpoint plus a
  reduced config that preserves the real top-level wrapper fields (`model_type`,
  `architectures`, `image_token_id`, `video_token_id`, `vision_start/end_token_id`,
  `language_model_only`, `text_config`, `vision_config`) and the real
  `qwen4_exp_text` field names inside `text_config` — including `layer_types`
  (the linear/full attention schedule), `linear_*`, `ngram_*`, `indexer_*`,
  `ple_*`, `hc_*`, the nested `mtp` block, and `rope_parameters`.
- How it is changed from the base: same schema, tiny geometry, float32, and a
  **reduced standard-attention + MoE tensor set**. It is a schema/load fixture, not
  a quantization or a distilled model.
- What it is **not**: not trained, not distilled, not a quality or benchmark claim.
- Why it is useful: it lets you test config parsing, weight-name mapping,
  expert-table sizing, router/top-k bookkeeping, and safetensors load paths in
  milliseconds, and it exercises the `qwen4_exp` multimodal wrapper shape that a
  plain text-only fixture does not.

## Fixture geometry (what this checkpoint actually contains)
| Field | Value |
|---|---|
| num_hidden_layers | 4 |
| hidden_size | 64 |
| num_attention_heads / num_key_value_heads | 4 / 2 |
| head_dim | 16 |
| layer_types | linear, linear, linear, full |
| num_experts | 8 |
| num_experts_per_tok | 2 |
| shared_expert_intermediate_size | 32 |
| moe_intermediate_size | 32 |
| dtype | float32 |
| vocab_size | 256 |

Total: 138 tensors, 1,157,376 data bytes = 289,344 float32 parameters.

## Intentional omissions (documented, not silent)
- **Text-only.** No vision encoder or projector tensors; `vision_config` is a
  placeholder. A full multimodal loader must supply vision/projector tensors.
- The tensor set is a **reduced standard-attention + MoE convention**. The real
  model's linear-attention (conv/ssm) tensors, ngram-embedding parts, attention
  indexer, PLE, hyper-connection, and MTP tensors are NOT included.
- No `lm_head` tensor; a loader must tie to `model.embed_tokens.weight` or supply
  its own head.
- Tokenizer metadata files are placeholders (no vocab file); use your own tokenizer.

## Verification actually performed (stdlib only, no torch in this environment)
- safetensors header parses: 138 tensors, contiguous `data_offsets`, header padded
  to 8-byte alignment; `__metadata__` records the generator string.
- `config.json` parses; `model_type` is `qwen4_exp` with a nested `qwen4_exp_text`
  `text_config` carrying the real field names listed above.
- `checksums.txt` records the SHA-256 of every tensor blob.
- Deterministic regeneration: SplitMix64 seed 20260903, Box-Muller normals, scale
  0.02, float32 row-major, consumed in sorted-name order.
- Generator script is included as `build_fixture.py` so the folder can be rebuilt
  and diffed.

**Not yet verified:** loading under a specific `transformers` version (no
torch/transformers in the build environment), and whether
`Qwen4ExpForConditionalGeneration` accepts this reduced text-only geometry without
the linear-attention / ngram / indexer / PLE / MTP tensors. Treat those as open
until run against a real install.

## How to run it
Read the tensors with the standard library (no torch needed, matching how this was
built):

```python
import json, struct
with open("model.safetensors", "rb") as f:
    n = struct.unpack("<Q", f.read(8))[0]
    header = json.loads(f.read(n))
    # header[name] = {"dtype", "shape", "data_offsets"}; data starts at byte 8+n
```

Or with the `safetensors` package:

```python
from safetensors.torch import load_file
tensors = load_file("model.safetensors")   # {name: tensor}
```

To exercise a real loader, build a config from `config.json` (the `qwen4_exp`
model type; use `AutoConfig.from_pretrained(..., trust_remote_code=True)` where
needed) and feed these weights in. There is no `lm_head` tensor, no vision stack,
and the tokenizer files are placeholders, so supply your own head/tokenizer/vision.

## License
The generated fixture content (random weights, config, scripts) is released under
MIT (see `LICENSE`). The `qwen4_exp` architecture and config schema belong to the
base model `Qwen/Qwen3.8-Flash-Next` under its own terms, which were **not**
independently re-verified this cycle — check the base repository before
redistribution.

## Citation
Qwen Team, Qwen3.8-Flash-Next, 2026.

---

Buy me a beer:
**BTC** `bc1q5ayht3fxhj0v95fk0z8l2f6900g3awdsw5842p`