The video opens with a sloppy 28 by 28 pixel 3 that a brain recognises effortlessly, and sets out to show what a neural network actually is, assuming no background, using handwritten digit recognition as the example.
A neuron is a thing that holds a number between 0 and 1: the first layer has 784 neurons, one per pixel's grayscale value, the last layer has ten neurons for the digits, and two hidden layers of 16 neurons sit in between.
The hope for the layered structure is that the second layer picks up little edges, the third picks up subcomponents such as loops and lines, and the output combines them into digits, the same layers of abstraction found in parsing speech.
Each connection carries a weight, the weighted sum gets a bias such as negative 10 and is squished into 0 to 1 by the sigmoid function, giving almost exactly 13,000 weights and biases that learning must set.
Written compactly as a matrix-vector product plus a bias vector inside a sigmoid, the whole network is a function from 784 numbers to 10, and guest Leisha Lee notes that modern networks mostly use ReLU instead of sigmoid because it is much easier to train.
Brief overview
A neural network is just a function: layers of numbers linked by weights and biases that recognise handwritten digits.
Neurons hold numbers, the network is a functionEach neuron holds an activation between 0 and 1, and the network maps 784 pixel values to 10 digit scores.
Weights pick patterns, biases set thresholdsWeights choose which pixel pattern a neuron responds to; the bias decides how high the weighted sum must be before it activates.
ReLU has largely replaced the sigmoidLeisha Lee says sigmoids were hard to train and ReLU worked very well for very deep networks.
Questions this recording answers
5 questions, each answered where it is said
What does a neuron in a neural network hold?
A neuron is simply a thing that holds a number between 0 and 1, called its activation. In the digit network, each of the 784 first-layer neurons holds the grayscale value of one pixel, from 0 for black up to 1 for white, and a neuron is lit up when its activation is high.
How many layers and neurons does the digit-recognition network use?
It has an input layer of 784 neurons, one for each pixel of a 28 by 28 image, an output layer of ten neurons, one per digit, and two hidden layers of 16 neurons each. The video admits the two hidden layers and the number 16 were fairly arbitrary choices.
What is the bias in a neural network?
The bias is an extra number added to a neuron's weighted sum before the sigmoid, for example negative 10 if the neuron should only be active when the sum is bigger than 10. Weights say which pixel pattern a neuron picks up on; the bias says how high the weighted sum must be before it becomes meaningfully active.
How many parameters does this network have?
Almost exactly 13,000 weights and biases. The first hidden layer alone has 784 times 16 weights plus 16 biases, and the other layers add more. Learning means getting the computer to find a valid setting for all of these numbers so the network solves the problem.
Why do modern networks use ReLU instead of the sigmoid?
Leisha Lee explains that sigmoid networks were very difficult to train, and ReLU, the rectified linear unit that takes a max of zero and the input, turned out much easier to train and worked very well for incredibly deep neural networks.
Key Quote
“when I say neuron, all I want you to think about is a thing that holds a number, specifically a number between 0 and 1.”
— But what is a neural network? | Deep learning chapter 1, at 2:51▶ watch at 2:51
Key Quote
“the brightest neuron of that output layer is the network's choice for what digit this image represents.”
— But what is a neural network? | Deep learning chapter 1, at 5:23▶ watch at 5:23
Key Quote
“So the weights tell you what pixel pattern this neuron in the second layer is picking up on”
— But what is a neural network? | Deep learning chapter 1, at 11:24▶ watch at 11:24
Key Quote
“this network has almost exactly 13,000 total weights and biases, 13,000 knobs and dials”
— But what is a neural network? | Deep learning chapter 1, at 12:18▶ watch at 12:18
Key Quote
“so much of machine learning comes down to having a good grasp of linear algebra”
— But what is a neural network? | Deep learning chapter 1, at 14:14▶ watch at 14:14
Key Quote
“Really the entire network is just a function, one that takes in 784 numbers as an input and spits out 10 numbers as an output.”
— But what is a neural network? | Deep learning chapter 1, at 15:35▶ watch at 15:35
Cite this
“But what is a neural network? | Deep learning chapter 1.” https://www.youtube.com/watch?v=aircAruvnKk. Transcript summary by WhipScribe, https://whipscribe.com/transcript-pages/but-what-is-a-neural-network-deep-learning-chapter-1.html.