Qwen_quantized_dynamic_8-bit _(INT8)
Quantization was performed using dynamic 8-bit integer quantizatio. The process leverages asymmetric weight quantization (symmetric for activations) without a calibration dataset, as dynamic quantization does not require one.
Quantization Type: Dynamic, 8-bit (INT8)
Calibration Dataset: None (dynamic quantization)
Operators for Quantization: MatMul, Add, Gather, EmbedLayerNormalization
Quantization Configuration:
- Weight Quantization: Symmetric
- Activation Quantization: Asymmetric
- Per-Channel Quantization: Enabled
- Reduce Range: Disabled
- MatMulConstBOnly: Enabled
Used Configuration:
extra_options = {
"WeightSymmetric": True,
"ActivationSymmetric": False,
"MatMulConstBOnly": True,
}
operators_to_quantize = [
"MatMul",
"Add",
"Gather",
"EmbedLayerNormalization"
]
quantize_dynamic(
model_input=input_model_path,
model_output=output_model_path,
op_types_to_quantize=operators_to_quantize,
nodes_to_exclude=[],
per_channel=True,
reduce_range=False,
weight_type=QuantType.QInt8,
use_external_data_format=use_external_data,
extra_options=extra_options
)