Deep Dive: GPU vs. TPU Architectures for LLMs

This article provides a detailed comparison of GPU and TPU architectures, focusing on their core compute units, memory hierarchies, and networking capabilities. Using the H100 and B200 GPUs as examples, it meticulously dissects the internal workings of modern GPUs, including Streaming Multiprocessors (SMs), CUDA Cores, Tensor Cores, and the interplay between various memory levels (SMEM, L2 Cache, HBM). The article also contrasts GPU and TPU performance in collective communication (e.g., AllReduce, AllGather), analyzing the impact of different parallelism strategies (data parallelism, tensor parallelism, pipeline parallelism, expert parallelism) on large language model training efficiency. Finally, it summarizes strategies for scaling LLMs on GPUs, illustrated with DeepSeek v3 and LLaMA-3 examples.