A widely used technique to make AI models more efficient, quantization, has its limits, and the industry may be approaching them, according to a recent study by researchers from Harvard, Stanford, MIT, Databricks, and Carnegie Mellon. Quantization, which involves reducing the number of bits needed to represent information, has been a convenient way to make AI models less computationally demanding. However, the study suggests that quantization may have more trade-offs than previously assumed, particularly when it comes to large-scale AI models.

In the context of AI, quantization refers to lowering the number of bits needed to represent information. This process is analogous to providing a more approximate answer to a question, such as saying "noon" instead of "oh twelve hundred, one second, and four milliseconds." While quantization can make AI models more efficient, the study reveals that it may not be as effective as previously thought, especially when dealing with large models trained on vast amounts of data.

The researchers found that quantized models perform worse if the original, unquantized version of the model was trained over a long period on lots of data. This could spell bad news for AI companies training extremely large models and then quantizing them to make them less expensive to serve. In fact, developers and academics have already reported that quantizing Meta's Llama 3 model tended to be "more harmful" compared to other models, potentially due to the way it was trained.

Tanishq Kumar, a Harvard mathematics student and the first author on the paper, emphasized that the number one cost for everyone in AI is and will continue to be inference, and that reducing inference costs will not work forever. He noted that AI model inferencing, or running a model, is often more expensive in aggregate than model training. For instance, Google spent an estimated $191 million to train one of its flagship Gemini models, but if the company were to use a model to generate just 50-word answers to half of all Google Search queries, it would spend roughly $6 billion a year.

The study's findings have significant implications for the AI industry, which has largely embraced training models on massive datasets under the assumption that "scaling up" will lead to increasingly more capable AI. However, evidence suggests that scaling up eventually provides diminishing returns, and there's little sign that the industry is ready to meaningfully move away from these entrenched scaling approaches.

One potential solution to mitigate the limitations of quantization is to train models in "low precision," which can make them more robust. Kumar and his co-authors found that training models in lower precision can help reduce the impact of quantization on model performance. Hardware vendors like Nvidia are also pushing for lower precision for quantized model inference, with their new Blackwell chip supporting 4-bit precision. However, Kumar notes that extremely low quantization precision might not be desirable, and that precisions lower than 7- or 8-bit may see a noticeable step down in quality.

The study's authors acknowledge that their research was conducted at a relatively small scale, but they plan to test it with more models in the future. Kumar believes that at least one insight will hold: there's no free lunch when it comes to reducing inference costs. "Bit precision matters, and it's not free," he said. "You cannot reduce it forever without models suffering. Models have finite capacity, so rather than trying to fit a quadrillion tokens into a small model, in my opinion, much more effort will be put into meticulous data curation and filtering, so that only the highest quality data is put into smaller models."

The study's findings serve as a reminder that AI models are not fully understood, and that known shortcuts that work in many kinds of computation don't work here. As the AI industry continues to grapple with the challenges of scaling up, it's clear that a more nuanced approach to model development and inference is needed. By acknowledging the limitations of quantization, researchers and developers can work towards creating more efficient and effective AI models that meet the demands of real-world applications.