700 Tokens Per Second: Google's New AI Chips Run in Meta's Data Centers

Google has officially connected its custom AI chips to Meta's infrastructure. The tech giant announced that its Tensor Processing Units, or TPUs, are now running AI workloads in production at Meta. This development marks a significant shift in the industry, as it is the first time Google has sold its TPUs to a major external customer. The move directly challenges the dominance of Nvidia, which has long controlled the market for AI hardware.

The integration works through new versions of two popular inference engines, SGLang and vLLM. Previously, these tools required Google's JAX framework to run on TPUs. Now, they support PyTorch natively. This change allows Meta to use the hardware without rewriting its core software. Meta's PyTorch-TPU lead, Zongwei Zhou, confirmed that the new SGLang version is live. He described the result as a mature serving stack developed through a three-way collaboration between Meta, Google, and the SGLang team.

Jana van Greunen, a director of PyTorch engineering at Meta, emphasized the strategic importance of this shift. She stated that Meta cannot be limited to a single hardware architecture. According to Greunen, the new setup enables the company to deploy its PyTorch workloads directly to Google's TPUs. Google co-developed this compatibility layer with Meta's PyTorch team to ensure a smooth transition for the social media giant.

This is not the first time Meta has experimented with Google's chips. An older bridge called PyTorch/XLA allowed some use of TPUs from 2020 to 2023. However, those contracts were canceled. The new TorchTPU solution is a more direct successor to that earlier effort. Reports suggest that Google's push was specifically aimed at Meta, which had resisted migrating its entire software stack to JAX. The new native support removes that barrier.

The physical location of these chips remains somewhat unclear. Industry talks last year suggested a potential rental agreement in 2026, with purchases planned for 2027. Meta reportedly signed a multi-year rental deal, though the exact data centers hosting the hardware have not been publicly detailed. The announcement came just one day after Amazon open-sourced its own PyTorch backend for its Trainium chips. Both moves appear designed to break Nvidia's software lock-in.

Performance metrics highlight the potential of this new setup. A tuned vLLM configuration reportedly achieved over 700 tokens per second for a single user. This speed was recorded on the seventh-generation Ironwood TPU while serving a massive 2.8-trillion-parameter model. The test was conducted by Inferact, a startup that helped build the new tool. It used 16 Ironwood chips with a technique called speculative decoding to boost speed.

For comparison, a similar setup using 16 Nvidia GB200 chips reached 452 tokens per second in the same test. Without speculative decoding, the new TPU kernels still delivered roughly 1.4 to 2 times the throughput of the Nvidia baseline at small batch sizes. These results suggest that Google's custom silicon is becoming a serious competitor for high-performance AI inference tasks.

The ecosystem is set to expand further. SGLang-TorchTPU will add support for more models, starting with DeepSeek V4.1 Flash and Reflection Beam. The developers also aim for Day 0 support for new open models by the first quarter of 2027. This means the hardware will be ready to run new AI releases on the day they launch. Google, Meta, and RadixArk plan to share more details at a summit in San Francisco in November.

Open questions remain about the long-term stability of this partnership. While the initial performance numbers are strong, the scale of the deployment is still being defined. The success of this alliance will likely depend on how well the software continues to mature. For now, the arrival of Google's chips in Meta's production environment signals a changing landscape for AI infrastructure in the Southeast and beyond.