Abstract
Post-training quantization (PTQ) is a widely adopted technique for compressing large language models (LLMs) without retraining. Existing second-order PTQ methods, including GPTQ, construct quantization objectives exclusively from input activation statistics, effectively assuming that all output channels contribute equally to the layer-wise reconstruction objective. We propose KronQ, a PTQ framework that challenges this assumption by introducing the gradient covariance into the quantization pipeline. Under the Kronecker-factored Hessian approximation, the quantization loss depends jointly on both the activation and gradient covariances, and KronQ exploits this at two complementary levels. (1) KronQ introduces bidirectional incoherence processing, extending the existing input-side random rotation to the output dimension using the gradient covariance, reducing weight magnitude variance across both input and output dimensions. (2) KronQ derives a new sensitivity metric for inter-layer mixed-precision allocation, driven by the gradient and activation Hessian traces. Notably, in the case of 2-bit weight-only quantization on LLaMA-3-70B, while GPTQ and GPTAQ diverge or produce degenerate quantizations (>2000 perplexity on WikiText-2), KronQ achieves 7.93 perplexity.
Community
What got me was the gradient covariance angle. I've run GPTQ enough times to know it treats every output channel like it matters equally, and that's always felt like a shortcut. KronQ adds the gradient covariance into the quantization objective via a Kronecker-factored Hessian approximation, which means it actually weighs which output channels matter more for the reconstruction loss. Makes intuitive sense — not all weights pull their weight in the forward pass.
The practical question is whether the extra compute at quantization time pays off. Second-order methods already cost more than RTN or AWQ, and KronQ adds another matrix factorization step on top. For a one-time quant on a model you'll serve millions of times, sure. For iterating on quant recipes during development, the overhead might sting.
I'd be curious to see perplexity numbers at 3-bit and 2-bit specifically. That's where GPTQ tends to fall apart and where a better loss objective would show its teeth. If KronQ holds at 3-bit where GPTQ starts hallucinating, that's the real win.
Thank you for your interest in our work and for the thoughtful comments!
On quantization time: as reported in Table 8, KronQ costs slightly more than GPTAQ due to the additional rotation. A Llama-2-7B model quantizes in about 20-25 minutes depending on the hardware. We also need the gradient covariance H_G beforehand, but this is a one-time offline pass (a single backward) that is reused across all bit-widths. We publish the precomputed H_G caches on the Hub, so feel free to use them.
On performance: we ran extensive experiments at 3- and 2-bit, and that's exactly where the gap opens up. Per-channel WikiText-2 PPL (GPTQ → KronQ):
Llama-2-13B: W3 9.41 → 5.16, W2 35.89 → 6.90
Llama-2-70B: W3 5.47 → 3.67, W2 9.54 → 5.41
Llama-3-70B: GPTQ diverges (>2000 PPL at both W3 and W2), while KronQ stays at 4.48 (W3) and 8.43 (W2).
So right where GPTQ starts to fall apart, KronQ holds up. The gains are largest at exactly the low-bit regime you're pointing at.
Numbers and packed checkpoints (Llama-2/3, 7B-70B, W2/W3/W4): https://huggingface.co/donghyunli
H_G precomputation code is on GitHub: https://github.com/Intelligent-Computing-Lab-Panda/KronQ
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- MGVQ: Synergizing Multi-dimensional Sensitivity-Aware and Gradient-Hessian Fusion for Vector Quantization (2026)
- TwinQuant: Learnable Subspace Decomposition for 4-Bit LLM Quantization (2026)
- TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization (2026)
- Joint Structural Pruning and Mixed-Precision Quantization for LLM Compression (2026)
- Beyond Activation Alignment:The Alignment-Diversity Tradeoff in Task-Aware LLM Quantization (2026)
- OffQ: Taming Structured Outliers in LLM Quantization by Offsetting (2026)
- DynamicPTQ: Mitigating Activation Quantization Collapse via Residual-Stream Dynamics (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Models citing this paper 30
Browse 30 models citing this paperDatasets citing this paper 5
Browse 5 datasets citing this paperSpaces citing this paper 0
No Space linking this paper