Giant MoE models. Consumer GPUs.

@hatsunama

3.2K followers · usually 1.1K views a post

Views

3.4K

Likes

184

Comments

57

Shares

27

SoundTikTok sound · original sound - hatsunama

Promoted product

Pulsar

Why it worked

The video likely performed well because it addresses a cutting-edge topic (running large AI models on consumer hardware) in an accessible way. It offers a practical solution and demonstrates tangible results, appealing to tech enthusiasts and AI developers.

Summary

The video demonstrates how to run large AI models (700 billion parameters) on consumer GPUs by using a new Rust + CUDA engine called Pulsar. This engine streams data from the SSD directly into the GPU's VRAM, allowing for efficient execution of these models at home.

Structure

1Introduction to running large AI models on consumer GPUs
2Explanation of the Pulsar engine and its function
3Demonstration of performance metrics for different model sizes
4Highlighting the efficiency and speed improvements
5

Original caption

Consumer GPU just got a major Ai upgrade!! #ai #news #localai #gpu #fyopppppppppppppppppppp

More from @hatsunama

Product placement

The video is a demonstration of a new AI engine called Pulsar, which allows large AI models to run on consumer GPUs. The engine's functionality and performance are explained and highlighted throughout the video.

Call to action

Hit that save button. Let's get in.

On-screen text

Giant MoE models. Consumer GPUs. A 700 BILLION PARAMETER AI MODEL ON TWO 16 GIGABYTE GRAPHIC CARDS AND I'M GONNA SHOW YOU HIT THAT SAVE BUTTON. LET'S GET IN. LET'S GET IN AGAIN. WE'RE NOT SHRINKING THE MODEL. WE'RE DOING SOMETHING VERY SIMILAR TO WHAT WE DID IN THE LAST VIDEO, BUT WE'RE STREAMING FROM THE SSD STRAIGHT INTO THE GPU. THIS TIME. THIS IS PULSAR A NEW RUST + CUDA ENGINE IT'S JOB IS TO KEEP THE BRAIN OF THE MODEL THE CURRENT BRAIN THE MOST USED EXPERTS AND THE CURRENTLY BEING USED LAYER ONLY THOSE IN YOUR VRAM SO NOW THESE GIANT MOE. MODELS THAT USED ENTIRE DATA CENTER, YOU CAN NOW RUN AT HOME. IT'S NOT SET UP FOR KIMI K 3 BUT IT WILL RUN KIMI K 2.7 AT 1.2 TOKENS/SEC GLM 5.2 HAS 743 B PARAMETERS AND IT'LL RUN AT 2.7 TOKENS/SEC IF YOU USE THE 295 YOU'LL GET 7 TOKENS/SEC WHICH IS WAY FASTER THAN THE PREVIOUS METHOD THIS IS REALLY COOL THAT IT TREATS YOUR SSD BASICALLY AS AN EXTENSION OF YOUR GPU SO NOW WE CAN BASICALLY DO ANYTHING WE WANT. I TOLD YOU, THE TECH IS GONNA CONTINUE TO IMPROVE

Transcript

Giant MoE models. Consumer GPUs. You can now run a 700 billion parameter AI model on two 16 gigabyte graphics cards and I'm gonna show you hit that save button. Let's get in. Let's get in again. We're not shrinking the model. We're doing something very similar to what we did in the last video, but we're streaming from the SSD straight into the GPU this time. This is Pulsar, a new Rust + CUDA engine. Its job is to keep the brain of the model, the current brain, the most used experts and the currently being used layer, only those in your VRAM. So now these giant MoE models that used to require an entire data center, you can now run at home. It's not set up for Kimi K3, but it will run Kimi K 2.7 at 1.2 tokens/sec. GLM 5.2 has 743 billion parameters and it'll run at 2.7 tokens/sec. If you use the 295 billion parameter model, you'll get 7 tokens/sec, which if you know from the last video is way faster than the previous method. This is really cool that it treats your SSD basically as an extension of your GPU. So now we can basically do anything we want. I told you, the tech is gonna continue to improve.