HELLO, TODAY WE'RE GOING TO BE BUILDING A NEURAL NETWORK COMPLETELY FROM SCRATCH

Hello, today we're going to be building a neural network completely from scratch with no libraries except numpy, so you can use this for your portfolio if you're trying to break into data science and machine learning. By the end, I expect you to understand exactly what's happening inside these things instead of treating them like magic boxes. Here's the plan. A neural network is just layers of neurons stacked together, right? Data comes in, flows through the hidden layers and a prediction comes out. Each connection has a weight, and training is just the process of tuning those weights until the predictions are good. We're going to build this with three classes, where we have our activation functions, our layers and the network itself. And each class is going to map to one of these ideas. Let's zoom into one neuron, it multiplies each input by a weight, adds them up and then adds an arbitrary value called the bias before passing the result through an activation function. Activation function adds non linearity to the network and allows it to learn complicated patterns like curves and bends instead of just straight lines. So let's start with those activation functions because they're the smallest piece and everything else builds on them. We make activation its own class. And notice each one has a forward and a backward method. Forward is just what happens when data flows through, and backward is what happens during learning. ReLU, which stands for rectified linear unit, is dead simple. It just kills any negative number and keeps the positives, whereas a sigmoid squeezes any number down between 0 and 1. We'll be using ReLU in the middle of the network and sigmoid at the very end. And why we need two different activations is because ReLU is fast and it's great for hidden layers because it avoids the problem called the vanishing gradient, where basically the network learning slows to a crawl to the point any updates in our gradients is so tiny that it doesn't really contribute to training. But our final answer needs to be between 0 and 1, like a probability. So the output layer uses sigmoid. So ReLU is used to think and sigmoid is used to decide. Now for our layer class. This is pretty much the heart of the code. When we create a layer, it sets up its own weights and biases randomly. Then it has a forward method which is just multiplying the weights, adding a bias and applying the activation function. It also incorporates a backward method, which is where learning happens. So the layer figures out how wrong its weights are and not just them in the right direction, and then passes the error back to the layer before it. Every layer is fully self contained, and it knows how to run itself forward and pretty much how to fix itself. Forward direction is just a forward pass where data flows through the network to make a prediction, and the backward direction is back propagation, which is the error flowing back through the network so each layer can learn from it. It's using the same network, just different directions. Finally we have the network's class and look how simple it is. It's literally just a list of layers. The forward method runs the data through every layer in order. The backward method runs the error through every layer in reverse. That reverse loop right there is the entire idea behind the word back propagation. We're propagating the error backward and that's it. To actually learn, the network needs to know how wrong it is, which is the loss. We compare the prediction to the correct answer and measure the gap. The whole goal of training is to shrink this number, epoch after epoch, which is just one pass through the data set. So now finally we're gonna do the training loop which is wrapped up in a clean method. And for every epoch, we're gonna run the forward pass, measure the loss and then send the error backwards so every layer updates itself. And this forward measuring backward repeating cycle is done thousands of times in a loop, and this little loop is the heartbeat of every neural network ever made, including the massive ones powering the AI you use everyday. Now we're actually going to initialise our network. We're going to teach it the XOR problem which is a classic problem that you cannot solve with a straight line, which makes it the perfect test for whether our hidden layers actually work. And look how readable this is, right? You just have one line for the input data, two lines to define the entire architecture, including a hidden layer with Rayloo and an output layer with sigmoid. And this is exactly how real frameworks like Karas feel to use. Except, now you know what's actually happening underneath every line. Okay, I did a bit of a skip because I had to debug and that took quite a bit because, you know, you forget nuances like the commas and the right parentheses. But, um, but now we can watch the loss drop. It starts around 0.25, which is basically the network random guessing and for every thousand epox, it drops all the way down to 0.0007, which is the network just getting smarter in real time right in front of us. And the final result, we asked for it to give us a pattern of 0110 and it gave us numbers that were really close to those values and that's a correct solve, so we built something that learns.