cstr/jina-embeddings-v5-text-small-retrieval-onnx-int8

jina-embeddings-v5-text-small-retrieval — INT8 ONNX (SmoothQuant α=0.8)

INT8-quantized ONNX of jinaai/jina-embeddings-v5-text-small-retrieval, built with the SmoothQuant α=0.8 outlier-migration recipe (same approach used for the F2LLM and Octen Int8 exports).

1.06 GB (50 % memory of FP32).

Quality

SmoothQuant α=0.8 + per-channel dynamic INT8 keeps cos_min ≈ 0.99 vs PyTorch FP32 reference on the 6-text canonical probe (vanilla quantize_dynamic collapses on the same probe — same Qwen3-style activation outlier issue documented for Octen / F2LLM Int8).

Files

File	Size	Description
model.int8.onnx	~5 MB	ONNX header (external data)
model.int8.onnx.data	~1 GB	SmoothQuant α=0.8 + INT8 weights
tokenizer.json, config.json, tokenizer_config.json	small	tokenizer + model config

Conversion

1. scripts/smoothquant_onnx.py --alpha 0.8 — graph rewrite that migrates activation outliers into weights via Mul nodes (mathematically (X * 1/s) @ (W * s) = X @ W).
2. scripts/quant_smoothed_int8.py — standard ORT dynamic INT8 on the smoothed FP32.

See Xiao et al. 2023, SmoothQuant for the underlying technique.

Use via fastembed-rs

let embedder = TextEmbedding::try_new(
    InitOptions::new(EmbeddingModel::JinaEmbeddingsV5SmallInt8))?;

Pooling: last-token. Asymmetric retrieval prefixes "Query: " / "Document: " are recommended.

License

Apache 2.0, inherited from the base model.

Provenance and EU AI Act Art. 53 note

• Upstream model: jinaai/jina-embeddings-v5-text-small-retrieval — published by jinaai.
• Upstream licence: cc-by-nc-4.0. This repository redistributes under the same terms; it grants no rights the upstream licence does not.
• What was done here: format conversion and/or quantisation only (ONNX, INT8 precision). No training, no fine-tuning, no merging, no distillation, no change to architecture, vocabulary or capability. Only the numeric representation of the upstream weights differs.
• Training data: documented — where it is documented at all — by the upstream provider; see the upstream model card. No training data was used, added or selected by this repository.
• Provider status: under Regulation (EU) 2024/1689 the upstream authors remain the provider of this model. Converting the serialisation format does not make this repository the provider of a new general-purpose AI model, and no such claim is made. Questions about training content, copyright policy or model capability belong upstream.