Generative AI has dazzled the world with its capacity to produce text, images, and code on demand. Yet each impressive output carries a considerable environmental burden. Training large language models gobbles enormous amounts of electricity—frequently generated from non-renewable sources—while inference at scale creates relentless energy demand and water consumption for cooling data centers. As AI adoption speeds up, grasping and reducing these environmental costs becomes urgent. This article explores the energy and water use of AI training and inference, examines efficiency gains from model distillation and smaller models, and considers the case for standardized disclosure of AI compute footprints.
1. Energy and Water Use of Training and Inference at Scale
Training large language models is an energy-intensive endeavor. Training GPT-3, for instance, is estimated to consume 1,287 megawatt-hours (MWh) of electricity—enough to power over 120 average U.S. homes for a year. That energy often comes from fossil fuels, adding to carbon emissions. Training also requires significant water for cooling data centers. A study by the University of California, Riverside, found that training GPT-3 consumed 700,000 liters of water—equivalent to the daily water needs of 1,000 people.
Inference at scale, especially for real-time applications such as chatbots and image generation, drives continuous energy demand. Each query to a large language model uses energy, and when multiplied by millions of users, the aggregate impact is staggering. A single query to ChatGPT may consume about 0.001–0.01 kWh of electricity, depending on model size and response length. While this seems small, at scale—billions of queries per day—the energy consumption adds up. Water usage for cooling is also continuous. Data centers often rely on evaporative cooling, which consumes water. In water-stressed regions, this can worsen local water scarcity.
Case studies highlight the strain on local resources. In drought-prone areas like California, data centers compete with agriculture and residential users for water. Some tech companies have faced criticism for their water consumption. Google's data center in The Dalles, Oregon, for example, used over 1.2 billion gallons of water in 2022, primarily for cooling, sparking concerns among local residents and environmental groups.
2. Efficiency Gains from Model Distillation and Smaller Models
Model distillation is a powerful technique to shrink AI's environmental footprint. It involves training a smaller, more efficient model (the student) to mimic the behavior of a larger, pre-trained model (the teacher). The student model learns to generalize from the teacher's outputs, achieving comparable performance on many tasks while requiring significantly less compute for inference. DistilBERT, a distilled version of BERT, for example, retains 97% of BERT's performance on language understanding tasks while being 40% smaller and 60% faster. This translates to substantial energy savings: inference with DistilBERT consumes about 60% less energy than BERT.
Smaller models and optimized architectures can also reduce training and inference costs without sacrificing much accuracy. MobileBERT, designed for mobile devices, achieves competitive performance on NLP tasks while being much more efficient. Efficient architectures like Transformers with sparse attention or pruning techniques further reduce compute requirements. A trade-off analysis between model size and performance reveals that for many applications, smaller models can deliver acceptable accuracy at a fraction of the environmental cost. In sentiment analysis, for instance, a distilled model may achieve 95% of the accuracy of a large model while using 10% of the energy.
These efficiency gains are crucial for sustainable AI infrastructure. By adopting distillation and smaller models, organizations can significantly lower their energy and water consumption, reducing operational costs and environmental impact. Moreover, efficient models enable deployment on edge devices, reducing the need for data center cooling and transmission losses.
3. Disclosure and Reporting Standards for AI Compute Footprints
Currently, there is a lack of standardized disclosure of AI's environmental impact, making it difficult to assess and compare models. A survey of existing reporting practices shows that few AI developers publish detailed energy or water usage data. Voluntary disclosures, such as Google's environmental report, provide some insights but are not standardized and often lack granularity. This opacity hinders accountability and informed decision-making by consumers, investors, and policymakers.
Emerging frameworks and regulations aim to mandate disclosure. The EU AI Act includes provisions requiring high-risk AI systems to disclose energy consumption and encourage sustainable practices. Industry initiatives like the AI Environmental Impact Score (AIEIS) propose standardized metrics for energy and water usage, enabling comparisons. These efforts are steps toward transparency, but broader adoption is needed. Standardized reporting would allow stakeholders to make informed choices, incentivize efficiency improvements, and drive investment in sustainable AI infrastructure.
Disclosure also fosters innovation. When companies measure and report their environmental footprint, they are more likely to invest in efficiency measures. Microsoft has committed to being carbon negative by 2030 and reports on its data center water usage. Such commitments, when backed by transparent metrics, can drive industry-wide change.
Conclusion
The environmental and infrastructure costs of generative AI are significant and growing. Training and inference consume vast amounts of energy and water, often in regions facing scarcity. However, solutions exist: model distillation and smaller models offer substantial efficiency gains, and standardized disclosure can drive accountability and sustainability. As AI continues to permeate every aspect of our lives, addressing its environmental footprint is not just an ethical imperative but also a business necessity. By optimizing energy and water use, leveraging efficiency techniques, and embracing transparency, we can harness the benefits of AI while minimizing its hidden price.
