You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/source/3x/PT_MXQuant.md
+36-9Lines changed: 36 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -70,32 +70,59 @@ Neural Compressor seamlessly applies the MX data type to post-training quantizat
70
70
</left>
71
71
</a>
72
72
73
-
The memory and computational limits of LLMs are more severe than other general neural networks, so our exploration focuses on LLMs first. The following table shows the basic MX quantization recipes in Neural Compressor and enumerates distinctions among various data types. The MX data type replaces general float scale with powers of two to be more hardware-friendly. It adapts a granularity falling between per-channel and per-tensor to balance accuracy and memory consumption.
73
+
The memory and computational limits of LLMs are more severe than other general neural networks, so our exploration focuses on LLMs first. The following table shows the basic MX quantization recipes in Neural Compressor and enumerates distinctions among various data types. The MX data type replaces general float scale with powers of two to be more hardware-friendly.
| Zero point | 0 (None) | $2^{bits - 1}$ or $-min * scale$ | 0 (None) |
79
79
| Granularity | per-block (default blocksize is 32) | per-channel or per-tensor | per-channel or per-tensor |
80
80
81
-
The exponent (exp) is equal to torch.floor(torch.log2(amax)), MAX is the representation range of the data type, amax is the max absolute value of per-block tensor, and rmin is the minimum value of the per-block tensor.
81
+
The exponent (exp) is equal to clamp(floor(log2(amax)) - maxExp, -127, 127), MAX is the representation range of the data type, amax is the max absolute value of per-block tensor, and rmin is the minimum value of the per-block tensor.
82
82
83
83
84
84
## Get Started with Microscaling Quantization API
85
85
86
-
To get a model quantized with Microscaling Data Types, users can use the Microscaling Quantization API as follows.
86
+
To get a model quantized with Microscaling Data Types, users can use the AutoRound Quantization API as follows.
87
87
88
88
```python
89
-
from neural_compressor.torch.quantization import MXQuantConfig, prepare, convert
0 commit comments