is now utilizing tensor processing units in production through a -native version of the inference engine. This integration follows the open-source release of a PyTorch-native engine, both of which previously relied on Google's JAX.
The transition is supported by TorchTPU, a solution co-developed by Google and Meta's PyTorch team to succeed PyTorch/XLA. Startups RadixArk and Inferact, founded by the creators of SGLang and vLLM, are driving these optimizations for custom AI hardware.
Performance benchmarks show a vLLM setup on 16 seventh-generation Ironwood TPUs reaching 700 tokens per second on the Kimi K3 model. This exceeds the 452 tokens per second achieved by 16 Nvidia GB200s, highlighting the efficiency of the new PyTorch stack.