The video effectively breaks down a complex AI concept (vector embeddings) using simple, relatable analogies and visual aids like sticky notes and drawings. The creator's clear explanation and step-by-step approach make the topic accessible to a general audience, driving engagement and understanding.
Summary
The video explains how Large Language Models (LLMs) process text by converting it into numerical tokens and then into vectors. It illustrates that different words have distinct vector representations based on their meaning and relationships, and that the dimensionality of these vectors allows for encoding complex information. The creator uses simple examples like 'cat', 'dog', and 'plane' to demonstrate how similar concepts have closer vector representations, while dissimilar ones are further apart.
Structure
1Introduction to tokens and LLMs
2Explanation of vectors and embedding matrices
3Example of word vectors (cat, plane, dog, bird)
4
Original caption
What is Vector? | AI basics #ai #aibasics #aiforbeginners
More from @easybits.tech.room
More breakout videos from this creator.
Concept of vector dimensions
5Stacking vectors into a matrix for LLM input
Product placement
easybits.tech (ACTS - provides educational content about AI and tech, creator is founder of this entity, removing it would change what happens in the video), LLM (ACTS - Large Language Model, the core technology discussed and used in the video, removing it would change what happens in the video), TIKTOKENIZER (ACTS - a website used to demonstrate tokenization, removing it would change what happens in the video)
Call to action
so follow me to not miss it
On-screen text
THE
SKY
WAS
LLM
1820
13180
574
6437
THE
SKY
WAS
BLUE
Token count
4
the
sky
was
blue
1820, 13180, 574, 6437
Show whitespace
Built by dqbd. Created with the generous help from Diagram.
EVERY TOKEN THAT COMES INTO THE LLM IS A NUMBER
AND LLM USES THESE NUMBERS TO CALCULATE THE NEXT TOKEN
THE ONLY THING HERE IT DOESN'T USE THIS ONES
TODAY WE'LL TALK ABOUT VECTORS
AND HOW INFORMATION IS REPRESENTED IN A MODEL
MY NAME IS IVAN I'M A COMPUTER SCIENCE TEACHER
FOUNDER OF ISABIT
AND I HELP ADULTS TO KEEP UP WITH TECH
LET'S TALK VECTORS
WE KNOW THAT WHEN TEXT COMES INTO A MODEL
WE CONVERT INTO SEQUENCE OF TOKENS
AND EVERY TOKEN IS BASICALLY A NUMBER
WE CAN OPEN WEBSITES LIKE TIKTOKENIZER AND SEE
REAL NUMBERS THERE BUT IN THE REALITY
THESE NUMBERS THEY ARE JUST LABELS
IT'S LITERALLY MEANS THAT THIS IS TOKEN NO. 574
AND ONCE THE SEQUENCE OF THESE LABELS
OR AS THEY CALLED TOKEN IDS COMING INTO A MODEL
IT ACTUALLY FALLS INTO SOMETHING THAT IS CALLED
EMBEDDING MATRIX AND CONCEPTUALLY
THIS EMBEDDING MATRIX IS A HUGE TABLE
WHERE WE HAVE THE ID OF THE TOKEN
AND ITS VECTOR REPRESENTATION
SO WHAT IS VECTOR ESSENTIALLY VECTOR
VECTOR IS THE ARRAY OF NUMBERS
SO KIND OF ORDERED SEQUENCE
AND YOU MAY ASK WHY WE NEED MULTIPLE NUMBERS
IF WE ALREADY HAVE THIS ONE
BECAUSE WITH ONE NUMBER
WE ARE QUITE LIMITED
WE NEED MULTIPLE NUMBERS TO HAVE A FLEXIBILITY
AND REPRESENT DIFFERENT PROPERTIES OF THE WORD
SEE WHEN MODEL PRODUCING
THE TEXT
WE ARE NOT WORKING WITH
WORDS AS SET OF
CHARACTERS
WE ACTUALLY WORK ALSO
WITH THE MEANING OF THE
WORD
AND WITH THE RELATIONS OF
WORDS BETWEEN EACH
OTHER
AND WE NEED TO ENCODE
QUITE A BIT OF INFORMATION
LET'S TAKE ONE VERY SIMPLE
EXAMPLE
IMAGINE WE HAVE TWO
WORDS
CAT AND PLANE AND WE WANT
TO ENCODE THEM WITH
VECTORS
AND WE SEE THAT THE
VALUES
THEY ARE QUITE DIFFERENT
BECAUSE CAT IS VERY
DIFFERENT FROM A PLANE
AND IF WE WANT NOW ADD
DOG HERE
WE UNDERSTAND THAT
VALUES WILL BE QUITE CLOSE
TO THE CAT
NOT THE SAME ONES BECAUSE
IT'S A DIFFERENT SPECIES
BUT IT'S WILL STILL BE
PRETTY FAR FROM A PLANE
BUT IF WE WILL DECIDE TO
ADD FOR EXAMPLE BIRD
BIRD IS ALREADY A LITTLE BIT
CLOSER TO A PLANE
BECAUSE IT ALSO CAN FLY
AND IT HAVE WINGS
SO YOU CAN SEE THAT
DIFFERENT WORDS HAVE
DIFFERENT RELATIONS IN
DIFFERENT CHARACTERISTIC
AND THE AMOUNT OF
NUMBERS
IN OUR VECTOR IS CALLED
DIMENSION
SO HERE IN OUR
OVERSIMPLIFIED EXAMPLE
WE HAVE THREE DIMENSIONAL
VECTOR
BUT IN THE REALITY
WE HAVE SOMETHING LIKE 4,0
96 DIMENSIONAL VECTOR
AND ONCE WE PROCESSED
ALL OUR TOKENS
THROUGH THIS EMBEDDING
MATRIX
FOR EACH OF THEM WE HAVE A
SEPARATE VECTOR
AND AFTER THAT WE KIND OF
STACK THESE VECTORS
TOGETHER
AND WE HAVE SOMETHING
THAT IS CALLED IN MATH
MATRIX
AND THIS MATRIX WITH SOME
ADDITIONAL INFORMATION
IS ACTUALLY WHAT GOES INTO
LLM
FOR CALCULATION THIS SET
OF FACTORS
IT'S NOT GOING THROUGH
JUST ONE MATHEMATICAL OPERATION
IT ACTUALLY GOES THROUGH
MULTIPLE LAYERS
BUT WE WILL TALK ABOUT IT
IN THE NEXT EPISODES
SO FOLLOW ME TO NOT MISS IT
FOLLOW ME
Transcript
Every token that comes into the LLM is a number. And LLM uses these numbers to calculate the next token. The only thing here it doesn't use this ones. Today we'll talk about vectors and how information is represented in a model. My name is Ivan, I'm a computer science teacher, founder of easybits.tech, and I help adults to keep up with tech. Let's talk vectors. We know that when text comes into a model, we convert into sequence of tokens. And every token is basically a number. We can open websites like Tiktokenizer and see real numbers there. But in reality, these numbers, they are just labels. It's literally means that this is token no. 574. And once the sequence of these labels or as they called token IDs coming into a model, it actually falls into something that is called embedding matrix. And conceptually, this embedding matrix is a huge table where we have the ID of the token and its vector representation. So what is vector? Essentially, vector is the array of numbers, so kind of ordered sequence. And you may ask, why we need multiple numbers if we already have this one? Because with one number, we are quite limited. We need multiple numbers to have a flexibility and represent different properties of the word. See, when model producing the text, we are not working with words as set of characters. We actually work also with the meaning of the word and with the relations of words between each other. And we need to encode quite a bit of information. Let's take one very simple example. Imagine we have two words: cat and plane. And we want to encode them with vectors. And we see that the values, they are quite different because cat is very different from a plane. And if we want now add dog here, we understand that values will be quite close to the cat, not the same ones because it's a different species. But it's will still be pretty far from a plane. But if we will decide to add, for example, bird, bird is already a little bit closer to a plane because it also can fly and it have wings. So you can see that different words have different relations in different characteristics. And the amount of numbers in our vector is called dimension. So here in our oversimplified example, we have three-dimensional vector. But in the reality, we have something like 4096 dimensional vector. And once we processed all our tokens through this embedding matrix, for each of them, we have a separate vector. And after that, we kind of stack these vectors together. And we have something that is called in math, matrix. And this matrix with some additional information is actually what goes into LLM for calculation. This set of factors, it's not going through just one mathematical operation. It actually goes through multiple layers. But we will talk about it in the next episodes. So follow me to not miss it.
