IF YOU WANT TO RUN QWEN3.8

IF YOU WANT TO RUN QWEN3.8 Qwen 3.8-27B 27 BILLION PARAMETERS MODEL YOU WILL NEED AROUND 60 GIGABYTES OF RAM 60 GB YOU WILL NEED AROUND 60 GIGABYTES OF RAM BUT THERE IS A WAY TO MAKE THIS MODEL SMALLER SO IT WOULD FIT REGULAR CONSUMER HARDWARE TODAY WE TALK ABOUT QUANTIZATION THE MAJOR MODEL OPTIMIZATION TECHNIQUE AND WE WILL SEE WHERE ALL THESE CRAZY NUMBERS ARE COMING FROM MY NAME IS IVAN I'M A COMPUTER SCIENCE TEACHER FOUNDER OF EASYBITS AND I HELP ADULTS TO KEEP UP WITH TECH LET'S TALK QUANTIZATION SO WE HAVE MODELS AND MODELS HAVE PARAMETERS Parameters -0.310 -1.899 1.775 -0.380 -0.615 -0.947 1.670 0.926 -0.437 -0.062 1.284 -1.146 0.533 1.902 -0.771 0.118 -1.624 1.456 0.807 -0.293 -1.337 0.671 1.119 -0.854 0.442 -1.981 1.035 0.294 -0.529 1.748 -0.306 -1.472 0.985 1.341 -0.688 0.057 -1.758 1.026 0.713 -0.944 1.583 -0.227 -1.095 -0.364 -1.238 0.536 0.781 1.604 1.817 -1.453 -0.899 -1.663 -1.887 0.472 1.051 -0.349 0.231 -1.541 1.426 -0.912 AND WE STORE THESE NUMBERS IN OUR COMPUTER'S RAM RAM/VRAM OR V RAM FAST MEMORY BECAUSE WE NEED TO HAVE THEM AVAILABLE REALLY QUICK AND JUST AS ANY DATA ON OUR COMPUTERS EACH OF THESE NUMBER REQUIRES SOME AMOUNT OF MEMORY SOME AMOUNT OF BITS OR BYTES ON THE MODEL PAGE WE USUALLY SEE HOW MUCH MEMORY AND EVERY PARAMETER TAKES Qwen 3.8-27B FOR THIS PARTICULAR QUINN MODEL WE HAVE TENSOR TYPE BF16 WHICH APPROXIMATELY MEANS THAT EVERY PARAMETER TAKES 16 BITS OR TWO BITES SO 27 BILLION PARAMETERS TWO BITES EACH IT GIVES US ROUGHLY 54 GIGABYTES AND THIS IS WHAT WE NEED JUST FOR PARAMETERS WITHOUT ANYTHING ELSE BUT WHY WE NEED TWO BITES BECAUSE WE WANT TO HAVE SOME LEVEL OF PRECISION BYTE bit ZERO OR ONE SO ONE BITE IS 256 DIFFERENT COMBINATIONS AND IT MEANS 256 DIFFERENT VALUES AND THIS IS JUST ONE BITE WE END UP WITH MORE THAN 65,000 DIFFERENT VALUES IT DOESN'T SOUND THAT IMPRESSIVE BUT IMAGINE THAT ALL OF YOUR PARAMETERS ARE NUMBERS BETWEEN MINUS 2 AND 2 SO WITH TWO BYTES YOU CAN ENCODE 65 THOUSANDS OF VALUES BETWEEN THEM MEANS YOU REACH EXTREMELY HIGH PRECISION -0.9483 BUT WHAT IF I WOULD USE JUST ONE BITE INSTEAD OF TWO YES PRECISION WILL BE MUCH LOWER AND I WILL NOT BE ABLE TO ENCODE AS MANY NUMBERS BETWEEN MINUS 2 AND 2 AS BEFORE BUT IT STILL MAY WORK AND THIS PROCESS IS CALLED QUANTIZATION SO YOU USE SPECIAL TECHNIQUES TO DECREASE THE AMOUNT OF MEMORY EVERY PARAMETER NEEDS AND WAS THAT OVERALL MEMORY REQUIREMENTS FOR A MODEL GO DOWN THERE ARE DIFFERENT LEVELS OF QUANTIZATION 8 BITS 6 BITS EVEN 4 BIT AND ON EACH LEVEL MODEL BECOMES LESS AND LESS ACCURATE BUT MORE AFFORDABLE IMPORTANT CAVEAT HERE JUST CALCULATING HOW MUCH MEMORY PARAMETERS REQUIRE IS NOT ENOUGH TO UNDERSTAND WHAT KIND OF HARDWARE YOU NEED TO RUN THE MODEL BUT WE WILL GO DEEPER INTO IT IN THE NEXT EPISODES SO FOLLOW ME TO NOT MISS IT