TLDR: Mark Zuckerberg announced that Meta is working on its Llama 4 model, expected to launch later this year, using a massive AI GPU cluster with over 100,000 NVIDIA H100 GPUs. This setup reportedly cost over $2 billion.* Based on the content by Anthony Garreffa below.
Mark Zuckerberg has provided a small update on Meta's work on its new Llama 4 model, which is being trained on a cluster of AI GPUs "bigger than anything" Zuck has seen.
Meta is cooking its new Llama 4 right now, with Zuckerberg telling investors and analysts on an earnings call this week that the initial launch of Llama 4 is expected later this year. Zuck said: "We're training the Llama 4 models on a cluster that is bigger than 100,000 H100s, or bigger than anything that I've seen reported for what others are doing. I expect that the smaller Llama 4 models will be ready first".
Meta's new AI supercomputer with its 100,000+ NVIDIA H100 AI GPUs reportedly cost over $2 billion for the H100 AI GPU chips alone, which means Mark Zuckerberg is signing some fat cheques to NVIDIA. Speaking of which, NVIDIA CEO Jensen Huang recently spoke with Meta CEO Mark Zuckerberg, where Jensen said Meta now has 600,000+ NVIDIA H100 AI GPUs to which Zuck replied saying that Meta were "good customers for NVIDIA".
Meta is in a race with xAI and Elon Musk, with Musk's xAI to double the size of its Colossus AI supercomputer cluster, which has 100,000 NVIDIA Hopper AI GPUs and is getting upgraded to 200,000 NVIDIA Hopper AI GPUs. But the fight of the biggest AI supercluster is damn interesting to watch, just don't show me the power bills.