多种量化方式联合使用

VLLM支持使用llmcompressor同时使用SmoothQuant和GPTQ这2种量化方式：

from llmcompressor.transformers import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.modifiers.smoothquant import SmoothQuantModifier

# Configure the quantization algorithms
recipe = [
    SmoothQuantModifier(smoothing_strength=0.8),
    GPTQModifier(targets="Linear", scheme="W8A8", ignore=["lm_head"]),
]

# Apply quantization
oneshot(
    model=model,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
)

# Save the compressed model
SAVE_DIR = MODEL_ID.split("/")[1] + "-W8A8-Dynamic-Per-Token"
model.save_pretrained(SAVE_DIR, save_compressed=True)
tokenizer.save_pretrained(SAVE_DIR)

VLLM文档：INT8 W8A8 — vLLM

SmoothQuant是W8A8，GPTQ是W8A16，两者一起使用，可以得到精度损失更小的W8A8量化。

DeepSeek的解释：

• SmoothQuant mathematically "migrates" quantization difficulty from activations to weights, making activations easier to quantize.

• GPTQ then optimizes the 8-bit weight quantization using the smoothed activations as a calibration dataset.

• Together, they achieve W8A8 with minimal accuracy loss compared to FP16/FP32 models.