Google adds "TPU 8i Max" and "TPU 8tx" to 8th-generation TPUs, for long context and confidential inference, its first 4 systems in a year

Two systems added to the 8th-generation TPU

Google announced on its official blog on Nov. 9 that it has added new systems, "TPU 8i Max" and "TPU 8tx," to the 8th generation of "TPU," its in-house semiconductors for AI. TPU 8i Max targets very long context (the range of input an AI handles at once) and high compute density per rack (the shelves that house servers), while TPU 8tx targets secure, confidential inference. They join "TPU 8t" for training and "TPU 8i" for inference, announced in April. Google said it is the first time it has released four TPU systems in a single year.

Amin Vahdat, the blog's author and the senior vice president (chief technologist) who oversees AI and infrastructure at Google, also shared the announcement on X (posted at 10:38 p.m. Nov. 9 Japan time).

Of the software named in the post, the blog itself says TorchTPU was open-sourced last month. Agent Compute Engine was announced in October and is not a new announcement this time. TPU Sync is software Google said it is now open-sourcing; it handles multiple storage devices together and cuts the transfer latency of the KV cache (data in which an AI holds intermediate computation results) and the time to first token (TTFT) by more than 90%, according to the company.

TPU 8i Max puts two TPU 8i chips in one package

TPU 8i Max is a system that puts two low-latency TPU 8i chips in a single package. Compute performance, SRAM (fast on-chip memory) and HBM (high-bandwidth memory stacked next to the chip) bandwidth are each double those of TPU 8i. Because it fits twice the silicon into the footprint of an existing rack, data centers can install more computing power per unit of floor space and per unit of power, the company said.

According to Google, memory bandwidth becomes a bottleneck when dozens of AI agents simultaneously handle an entire software codebase or hours of video. It said that as companies have begun running agents in real business operations, two challenges have emerged, hardware-level data protection and context on the scale of 1 million tokens, and that the two new systems are configured to address them. It did not give absolute figures for compute performance or bandwidth.

TPU 8tx protects data in use with hardware

TPU 8tx is a system for secure, confidential inference and is also suited to small-scale training, the company said. It targets heavily regulated companies and governments that run frontier models, proprietary models and AI agents together with confidential data.

Data is encrypted when stored and transmitted, but it must be decrypted in memory while it is being computed. TPU 8tx combines the same silicon as TPU 8t with a host carrying AMD's 5th-generation "EPYC" CPUs, building security mechanisms into the hardware. Google said it protects and isolates confidential data and model weights even while they are in use.

For networking, it uses "Boardfly," a low-latency network built for real-time inference. Google said latency is up to half that of TPU 8t, with almost no loss of performance. It did not give the measurement conditions. Model fine-tuning, distillation from larger models to smaller ones and reinforcement learning can be done on the same system without moving confidential data to a separate cluster (group of computers), the company said.

New configurations combining existing chips

In this announcement, Google distinguishes between new chips and new systems. Designing and manufacturing chips takes years, but systems are built by combining the compute, networking, storage and packaging around the chips. Because it has designed TPUs to be assembled like building blocks, it can release purpose-built supercomputers for specific uses in a matter of months, the company said.

Google has said that TPU 8t and TPU 8i, announced at its annual cloud event "Google Cloud Next" in April, offer up to 2.7 times the training performance and 80% better inference price-performance, respectively, compared with the previous generation. It did not give such comparisons with the previous generation for the two new systems.

Google uses TPUs to train and serve its own Gemini and for services for Google Cloud customers, and recently made them available for installation in customers' own data centers. TPU customers number close to 100, including frontier AI developers, trading firms, retailers and universities, according to the company.

No word on pricing or availability

In the blog, Google did not disclose pricing, availability dates or regions for the two systems. It said details of the new TPU systems would be explained in a livestream starting at 9 a.m. Pacific time that day (2 a.m. Nov. 10 Japan time).