LET'S BUILD A
neural network
PYTHON
SO YOU CAN
BETTER UNDERSTAND
LET'S START BY
PICTURING
OF PROBLEM
NETWORK IS MEANT
TO SOLVE
A TABLE, AN IMAGE,
A SOUND CLIP
OF DATA
GIVEN EXAMPLES
FROM THE PAST
AND EACH EXAMPLE
ANSWER THAT WE
COMPUTER
TO PREDICT
AT PICTURES
OF HANDWRITTEN
FOR EXAMPLE, THEN
THE LABEL MIGHT
VALUES OF EVERY
SINGLE PIXEL,
AND THE OUTPUT
IS THE NUMBER
REPRESENTS
THE NEURAL
NETWORK IS TO
LOOK AT THOUSANDS
OF THESE EXAMPLES
THAT CONNECT
THE INPUT,
THEN WHEN IT
NEW EXAMPLE
LATER,
A SENSIBLE
OF JUST GUESSING
BUT FOR TODAY,
LET'S USE A SIMPLE
IMAGE EXAMPLE
IMAGINE EACH ROW
IN OUR DATASET
OF A
HANDWRITTEN DIGIT,
TELLS US WHETHER
THAT PICTURE IS
A 0, 1,
2, 3 AND
TO US, RECOGNISING
DIGIT IN THAT
IMAGE FEELS
BECAUSE WE CAN
SEE THE CURVES,
LINES AND SPACING
BUT TO OUR
COMPUTER, THE
IMAGE ISN'T REALLY
INSTEAD, IT'S JUST
A GRID OF NUMBERS
WHERE EACH
NUMBER REPRESENTS
CERTAIN PIXEL IS
BEFORE THE MODEL
CAN EVEN SEE
IT HAS TO
PATTERNS
PIXEL NUMBERS
FINAL LABEL
AT THE VERY
THE NEURAL
NETWORK KNOWS
NOTHING USEFUL
STRUCTURE, BUT NO
KNOWLEDGE YET
YOU CAN THINK
A GIANT SYSTEM
OF TINY DECISION
CALLED NEURONS
EACH NEURON
RECEIVES NUMBERS,
ADJUST THEM USING
WEIGHTS,
BIAS, AND THEN
PASSES THEM
NEXT LAYER
THE WEIGHTS ARE
IMPORTANT PART
BECAUSE THEY
STRONGLY A CERTAIN
INFORMATION
SHOULD MATTER
WHICH MEANS THE
NETWORKS
PRETTY TERRIBLE
IT A PICTURE
7
2 OR 5
OR ANYTHING ELSE
AND THAT'S BECAUSE
IT HAS NOT
PIXEL PATTERNS
ACTUALLY MATTER
FIRST LAYER OF
THE NEURAL NETWORK
INPUT LAYER,
WHERE RAW
ENTERS THE MODEL
HERE, A
DIGIT EXAMPLE
THAT EVERY SINGLE
PIXEL VALUE FROM
THE IMAGE
GETS PASSED INTO
A NUMBER
NETWORK IS JUST
THINKING ABOUT
THE IMAGE AS
A FULL OBJECT
IT'S JUST RECEIVING
A LONG LIST
FROM THERE, THOSE
NUMBERS MOVE INTO
ONE OR MORE
HIDDEN LAYERS
ARE WHERE THE
REAL LEARNING
BECAUSE EACH
TO FIND A
SMALL PATTERN IN
ONE NEURON MIGHT
VERTICAL LINE,
ANOTHER MIGHT
RESPOND TO A
DETECT A SHAPE
THAT LOOKS LIKE
AN 8
OF A 7
MOST IMPORTANT
NOT MANUALLY TELL
TO LOOK FOR
NETWORK DISCOVERS
USEFUL PATTERNS
BY ADJUSTING ITS
ONCE THOSE NUMBERS
MOVE THROUGH ALL
THE HIDDEN LAYERS,
REACH AN OUTPUT
LAYER
IS WHERE THE
MODEL TURNS ALL
CALCULATIONS
INTO A FINAL
IF WE ARE
DIGITS FROM 0
TO 9,
THE MODEL MIGHT
POSSIBLE SCORES,
ONE FOR EACH
SCORE FOR A
7 - 0.96
1 - 0.03
AND ALMOST NO
OTHER DIGITS
AT PREDICTION TIME,
THE MODEL CHOOSES
THE HIGHEST SCORE
I THINK THIS
IMAGE IS A 7
BUT DURING TRAINING,
WE DO NOT
WHAT THE MODEL
WE CARE ABOUT
HOW WRONG IT
THE CORRECT ANSWER
THE LOSS FUNCTION
COMES IN
FUNCTION IS
MODEL'S
AFTER THE MODEL
MAKES A PREDICTION,
THE LOSS FUNCTION
COMPARES THAT TO
AND PRODUCES A
NUMBER THAT TELLS
US HOW BADLY
THE MODEL WAS
GIVEN A PICTURE
AND GAVE THE
A 7,
IS SMALL
7 AND GAVE
THE HIGHEST SCORE
THEN THE LOSS
THE MODEL IS
SIMPLY TRYING TO
FIGURE SMALLER
AND SMALLER OVER
BUT THE KEY
"How does the
model know what to c
TO CHANGE?
KNOW WHAT
HAS THOUSANDS OR
EVEN MILLIONS
AND ANY ONE
BE THE REASON
A WRONG GUESS
THIS IS WHERE
BACK PROPAGATION
COMES IN
PROPAGATION
OF SENDING THE ERROR
THROUGH THE MODEL
WHICH WEIGHTS
CONTRIBUTED TO THE
AND HOW THEY
SHOULD BE ADJUSTED,
RANDOMLY CHANGING
HOPING IT IMPROVES
GIVES THE NETWORK
A WAY TO
IT WORKS BACKWARDS
FROM THE OUTPUT
LAYER AND THROUGH
THE HIDDEN LAYERS,
AND CALCULATES HOW
WEIGHT INFLUENCE
THE FINAL ERROR
KNOWS THE DIRECTION
IT SHOULD MOVE
EACH WEIGHT
GRADIENT DESCENT
IS KNOWN AS THE
UPDATE STEP,
IT SLIGHTLY CHANGES
THE WEIGHTS
THAT SHOULD REDUCE
THE LOSS
AVOIDS MAKING ONE
BIG CHANGE
INSTEAD, IT TAKES
SMALLER STEPS
EACH FULL PASS
THROUGH THE TRAINING
AN EPOCH
AND HOPEFULLY WITH
THE MODEL GETS
A LITTLE BIT BETTER
INPUTS TO THE
CORRECT OUTPUT
FIRST, THE MODEL'S
ALMOST RANDOM,
BUT AFTER GOING
THROUGH ENOUGH
IT STARTS TO
INTERNAL
EARLY LAYERS MIGHT
LEARN
SIMPLER PATTERNS
MIGHT CONNECT THOSE
MEANINGFUL SHAPES,
AND THE FINAL
THOSE SHAPES
INTO SOMETHING THAT
STRONGLY SUGGESTS A
NEURAL NETWORKS ARE
SO POWERFUL IN
TODAY'S WORLD
THE LEARNING
PROCESS STOPPED
THE WEIGHTS ARE
SO THE MODEL
THE CORRECT
ANSWER ANYMORE
A NEW IMAGE,
PUSHES THE PIXEL
VALUES THROUGH THE
SAME LAYERS,
PATTERNS THAT
TRAINING,
FINAL OUTPUT THAT
6
WE CAN SEE
HAS LEARNED WELL,
THEN A BRAND
NEW HANDWRITTEN 3
ACTIVATE THE KINDS
OF NEURONS,
BUT LEARN TO
RECOGNISE THE CURVES
ASSOCIATED WITH A
3,
SCORE FOR THE
3 - 0.97
THE STRONGEST
THAT FINAL
PREDICTION IS WHERE
ALL THE ABSTRACT
NEURONS, WEIGHTS,
PROPAGATION AND
GRADIENT DESCENT ALL
COLLAPSE INTO ONE
FINAL IDEA
A NEW EXAMPLE
AND MAKES AN
BASED ON THE
PATTERNS IT LEARNED
FROM THE PAST
WE'LL ACTUALLY BE
CODING OUT A
NETWORK
HANDWRITTEN
IN PYTHON
YOU FOLLOW ALONG
MISS IT
YOU THEN