In this exercise, you will continue to implement the neural network framework, that you started in exercise e06_nn_framework. At the end of this exercise, the framework should be extended by a convolutional layer and a pooling layer so that you can create a simple ConvNets. You want your operations, especially the convolution, to be efficient, so it will not slow down the training process to an unacceptable ratio. Therefore your goal is to implement vectorized versions of the layers in the exercise.
@@ -15,16 +15,16 @@ In this course you will extend the logistic regression model to a fully connecte
## Notebooks
In [Introduction to Machine Learning](introduction-to-ml.md)(highly recommended), you calulated the gradient by hand and just used the final formula. In this exercise you will learn how to just derive the single individual functions and chain them programatically. This allows to programmatically build computational graphs and derive them w.r.t. certain variables, only knowing the derivatives of the most basic functions.
Knowing how to compute the forward pass and the backward pass with backpropagation, you are ready for a simple neural neutwork. First you will refresh your knowledge about logistic regression, but this time, implement it using computational graph. Then you will add a hidden layer. Further you will understand what happens with the data in the hidden layer by plotting it.
For a better understanding of neural networks, you will start to implement a framework on your own. The given notebook explains some core functions and concepts of the framework, so all of you have the same starting point. Our previous exercises were self-contained and not very modular. You are going to change that. Let us begin with a fully connected network on the now well-known MNIST dataset. The Pipeline will be
In the previous exercises we used to initilize our weights with a normaldistribution centered arount 0. Although it was not the worst way we could have done this, there also exist better ways. One is the *Xavier* initilization \[GLO10\], which is still practically used in state-of-the-art neural network architectures.