Entropy-Weighted Quantization (EWQ): A Breakthrough in Compressing Large Language Models

Entropy-Weighted Quantization shows that selectively quantizing lower-entropy transformer layers preserves accuracy, reduces perplexity, and significantly decreases memory usage, unlocking efficient deployment of powerful AI models.

‍

Deploying large language models efficiently poses significant challenges due to their extensive computational and memory requirements. In our latest research paper, published on arXiv, webAI researchers have introduced Entropy-Weighted Quantization (EWQ), an innovative method that addresses these challenges through selective, architecture-agnostic quantization.

Unlike traditional methods that uniformly quantize all model layers—often degrading performance—EWQ strategically quantizes transformer blocks based on their entropy. By aggressively quantizing lower-entropy blocks that encode less critical information and preserving higher precision for high-entropy, performance-critical blocks, EWQ substantially reduces model size while maintaining accuracy.

‍

Key Advantages of EWQ

Accuracy Preservation

EWQ consistently achieves accuracy within 0.5% of full-precision models on rigorous benchmarks, such as MMLU. This level of accuracy retention surpasses traditional uniform quantization methods, making EWQ suitable for mission-critical applications.

Enhanced Perplexity through Regularization

In some cases, EWQ reduces perplexity compared to full-precision models. By selectively lowering precision, EWQ acts like regularization, improving the model’s predictions and output quality while also compressing its size.

Universal Applicability Across Architectures

EWQ is architecture-agnostic and consistently effective across diverse LLM architectures and sizes, eliminating the need for tailored quantization strategies. This simplifies deployment across various AI systems and environments.

FastEWQ: Accelerating Entropy Analysis

To optimize EWQ further, webAI developed FastEWQ, an efficient method using a supervised Random Forest classifier. FastEWQ predicts suitable transformer blocks for quantization without analyzing full model weights, based on three attributes:

‍

• Number of parameters per transformer block: Measures block complexity.
• Execution index: Indicates the block’s hierarchical position in inference.
• Total number of blocks: Contextualizes model scale and structure.

Performance Validated Across LLM Architectures

In addition to maintaining model accuracy within 0.5% of unquantized models, FastEWQ is able to significantly reduce model size and memory usage by 18 to 22% as observed in our benchmarking:

The classifier analysis highlights that blocks later in the inference chain tolerate quantization best, aligning with their focus on high-level abstractions.

‍

Conclusion

EWQ and FastEWQ provide groundbreaking tools for deploying advanced AI models, achieving optimal balance among efficiency, accuracy, and predictive confidence. This breakthrough significantly enhances practical AI deployment across industries. Both EWQ and FastEWQ significantly enhance the feasibility of running state-of-the-art LLMs on resource-constrained devices, such as 16GB consumer hardware, without performance degradation.

Explore the full details in the EWQ research paper.