symmetric vs asymmetric quantization

Symmetric vs. Asymmetric Quantization refers to how the quantization range is mapped to the original floating-point value range, specifically whether the zero point is fixed or learned.

Symmetric Quantization

• Zero-Point Fixed: The quantized zero is mapped to the floating-point zero. The quantization range is symmetric around zero.
• Formula: $q = \text{round}(x / s)$ where $s$ is the scale factor.
• Range: For 8-bit signed integers, the range is [-127, 127], with 0 mapping to 0.
• Advantages: Simpler implementation, faster inference (no zero-point offset calculation), better for hardware acceleration.
• Disadvantages: Wastes one quantization level if the data distribution is asymmetric (e.g., ReLU activations are always non-negative).

Asymmetric Quantization

• Zero-Point Learned: The quantized zero can map to any floating-point value. The quantization range is asymmetric.
• Formula: $q = \text{round}(x / s + z)$ where $s$ is scale and $z$ is the zero-point offset.
• Range: For 8-bit unsigned integers, the range is [0, 255], with the zero-point $z$ learned to minimize quantization error.
• Advantages: Better utilizes the quantization range for asymmetric distributions (e.g., post-ReLU activations), lower quantization error.
• Disadvantages: Slightly more complex, requires storing and applying the zero-point offset.

When to Use Each

• Symmetric: Weights (typically centered around zero), when hardware acceleration is critical, when simplicity matters.
• Asymmetric: Activations (especially after ReLU, which are non-negative), when minimizing quantization error is the priority.

Example

Consider values in range [0.5, 3.5]:

• Symmetric: Maps [-3.5, 3.5] to [-127, 127], wasting half the range on negative values that don't exist.
• Asymmetric: Maps [0.5, 3.5] to [0, 255], using the full quantization range efficiently.

Practical Impact

Most modern quantization frameworks (TensorFlow Lite, PyTorch) use:
- Symmetric quantization for weights (simpler, hardware-friendly).
- Asymmetric quantization for activations (better accuracy for ReLU outputs).

The choice between symmetric and asymmetric quantization is a fundamental design decision that impacts both model accuracy and inference efficiency.