@boringbiz_ — 6x breakout selling Agent X

Loading video…

Dylan Patel on NVIDIA vs AMD chips

@boringbiz_

6.1K followers · usually 1.4K views a post

Views

8.8K

Likes

141

Comments

Promoted product

Agent X

Why it worked

The video provides valuable technical insights into optimizing AI inference performance by highlighting the interplay between hardware (GPUs, memory, storage) and software, which is highly relevant to a technical audience.

Summary

The speaker discusses how the performance of AI inference depends not just on the GPU, but also on memory and storage solutions. They explain that while Nvidia GPUs are generally faster, AMD GPUs with more memory can sometimes outperform them, and that optimizing storage can also lead to better results than simply using the best GPU.

Free account

Transcript, structure and on-screen text

6 beats, a 383-word transcript and 4 lines of on-screen text — the parts you need to write your own version.

Create a free account to see the full breakdown

Original caption

Dylan Patel of SemiAnalysis says a worse GPU with better storage and memory now beats the best chip without them, so buying the newest GPU alone no longer wins inference. So, an AMD GPU with more memory can outperform Nvidia in some cases. "So what we have is we have over $80 million of compute, GPUs from Nvidia, AMD, TPUs from Google, Trainium from Amazon, and we run this benchmark constantly on the newest inference engine, newest drivers, newest PyTorch version, whatever it is." "Every day it runs on an automated CI, and we run it on all the latest Chinese models, from GLM, Zhipu, Moonshot, Kimi, Alibaba, all these models we run." "Initially, when we were benchmarking the difference between these chips and different engines, different schemes for parallelism, we were just running it fixed context length." "But now with Agent X, we've analyzed over $5 million worth of Claude Code traces. This is real production traffic that people have donated to us as well as internally generated. Now we know what the actual agent workload looks like." "And then as we implement that and run those benchmarks, it turns out yes, the chip you're using is very important, but now even more important is how are you handling this memory offload?" "And so while an Nvidia GPU is faster than an AMD GPU in most cases, because AMD GPUs have more memory, they actually end up outperforming in some cases." "Or you can have a worse GPU, but a much better storage solution, and now you can outperform what the best GPU can do without those solutions. So just buying the newest and latest GPU alone doesn't get you the best inference economics." "Actually, you need to layer in all these other innovations including storage and memory." [ Who's the top player on your chart? ] "That really is a difficult multivariable problem. And generally that means you need to have, yes, you need to have the best GPU, a GB300, but you also need to have the best storage solutions. And so I won't spoil who's the best right here, but I will say that storage solutions matter a lot and memory solutions matter a lot, as does your front-end networking. That matters a lot." #business #motivation #successful #nvidia