Commit e4c7d255 authored by Klaus Strohmenger's avatar Klaus Strohmenger

applied new folder structure

parent 2a18bd8f
......@@ -36,9 +36,10 @@ The following pen & paper exercise will provide you with the theoretical backgro
In this exercise, you will continue to implement the neural network framework, that you started in exercise e06_nn_framework. At the end of this exercise, the framework should be extended by a convolutional layer and a pooling layer so that you can create a simple ConvNets. You want your operations, especially the convolution, to be efficient, so it will not slow down the training process to an unacceptable ratio. Therefore your goal is to implement vectorized versions of the layers in the exercise.
* [exercise-cnn-framework](../notebooks/feed-forward-networks/exercise-cnn-framework.ipynb)
After finishing the exercise, you should have a working and efficient neural network framework. Do not forget to move all classes into the corresponding script files of the framework. A good follow up is to repeat your experiments on a dataset of your interest from **exercise-nn-framework** but with a ConvNet instead of a standard neural network.
One can never have enough training. Additionally in real worl problems, labeling the data by experts can become very expensive. The following exercise will teach you the most basic concepts for data augmentation with images.
One can never have enough training data. Additionally in real world problems, labeling the data by experts can become very expensive. The following exercise will teach you the most basic concepts for data augmentation with images.
* [exercise-images-data-augmentation-numpy](../notebooks/machine-learning-fundamentals/exercise-images-data-augmentation-numpy.ipynb)
......
......@@ -15,7 +15,7 @@ The following gif depicts the whole picture:
## Notebooks
Training and using regression models, and therefore also neural networks, require either a lot of loops, or better, matrix and vector operations. The first exercise is intended to refresh mathematic skills.
* [exercise-matrix-vector-operations](../notebooks/machine-learning-fundamentals/linear-algebra/exercise-matrix-vector-operations.ipynb)
* [exercise-matrix-vector-operations](../notebooks/math/linear-algebra/exercise-matrix-vector-operations.ipynb)
In Python, the most efficient way to handle vectors, matrices and n-dimensional arrays is via the package numpy. Its core functions are implemented in C and Fortran, which makes it extremely fast. Numpy provides a lot of functions and features to access and manipulate n-dimensional arrays. Having a basic knowledge about what is possible with numpy will ease any data-scientists daily work with Python.
* [exercise-python-numpy-basics](../notebooks/machine-learning-fundamentals/exercise-python-numpy-basics.ipynb)
......@@ -35,3 +35,6 @@ With the use of the sigmoid (or logistic) function, the linear regression can ea
Until now, you have only been working with training data. But only using training data, which the model used to learn its parameters, it is hard to really tell how the model performs on unseen data. In this notebook we introduce the out-of-sample error (E\_out), in contrast to the error on the training data (E\_in). E\_out is composed of the variance and the bias. The ratio between these two key figures is determined by the complexity of our model and if it did not fit to the training data enough (underfitting) or even too much (overfitting).
* [exercise-bias-variance-tradeoff](../notebooks/machine-learning-fundamentals/exercise-bias-variance-tradeoff.ipynb)
Of course machine learning is not all about neural networks (and therefore linear and logistic regression). There exist a lot of other algorithms, which still have their merit and justification. One example are decision trees and random forests. But before learning about the later, take a look at decision trees and the concept of entropy.
* [exercise-entropy](../notebooks/information-theory/exercise-entropy.ipynb)
* [exercise-decision-trees](../notebooks/machine-learning-fundamentals/exercise-decision-trees.ipynb)
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
daft = "*"
numpy = "*"
"pymc3" = "*"
matplotlib = "*"
tensorflow = "*"
scipy = "*"
pandas = "*"
theano = "*"
graphviz = "*"
mkl = "*"
torch = "==0.4.1"
torchvision = "*"
"pyro4" = "*"
[dev-packages]
[requires]
python_version = "3.6"
Formatiert,
Aufgabe ohne Solution vorhanden (in educational-material),
keine Ergänzungen
prio: 1=sehr hoch, 2=hoch, ...
- /exercise-1d-gmm-em.ipynb (prio: 2.2)
- XXX_DONE - /exercise-sensorfusion-and-kalman-filter-1d.ipynb (prio: 4)
- XXX_DONE - /exercise-simple-example-for-EM.ipynb
Nicht formatiert / nur halb formatiert,
Aufgabe ohne Solution NICHT vorhanden (in educational-material),
keine Ergänzungen
- /exercise-bundesliga-predictor-pymc3.ipynb (3,2)
- /exercise-probabilistic-rankings-pymc3-solution.ipynb (3,3)
- /exercise-pymc3-examples.ipynb (prio: 3,1)
- /exercise-topic-models-latent-dirichlet-allocation-pymc3.ipynb (prio:3,5)
- /exercise_hierarchical_model_for_univariate_gaussian_data.ipynb (prio: 2) merge (???) with exercise-univariate-gaussian-likelihood
- /exercise-linear-regression-polynominal-tensorflow_probability.ipynb (3.9)
- /exercise-bayesian-linear-regression-via-variational-em.ipynb (prio: 3)
NEW FILES
- /exercise-variance-dependence-on-sample-size.ipynb
- /exercise-backprop-through-a-matrix.ipynb
This diff is collapsed.
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2D convolution as matrix operation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Strictly speaking we are working here with cross correlation and not with a convolution."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Kernel:\n",
"\n",
"$$\n",
"\\begin{pmatrix}\n",
"w_{11} & w_{12} \\\\\n",
"w_{21} & w_{22} \n",
"\\end{pmatrix}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Image:\n",
"\n",
"$$\n",
"\\begin{pmatrix}\n",
"x_{11} & x_{12} & x_{13} & x_{14}\\\\\n",
"x_{21} & x_{22} & x_{23} & x_{24}\\\\\n",
"x_{31} & x_{32} & x_{33} & x_{34}\\\\\n",
"\\end{pmatrix}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Exercise\n",
"\n",
"The 2D cross correlation can be expressed as a matrix multiplication:\n",
"\n",
"$$\n",
"\\vec y = W \\cdot \\vec x\n",
"$$\n",
"\n",
"How must the matrix $W$ and the vector $\\vec x$ looks like?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Exercise:\n",
"\n",
"Describe how this can be extended if we have more than one input (and output) channel in a CNN. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Programming exercise\n",
" \n",
" \n",
"Implement a function with\n",
"\n",
"- Input:\n",
" - image (with k-channels)\n",
" - k convolutional kernel(s)\n",
"- Output\n",
" - convolution kernels as matrix\n",
" - image as vector "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"celltoolbar": "Slideshow",
"kernelspec": {
"display_name": "deep_teaching_kernel",
"language": "python",
"name": "deep_teaching_kernel"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise: Backprop through a matrix"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Forward pass\n",
"Assume we have the following mathematical expressions:\n",
" \n",
"$$\n",
"\\vec y = W \\cdot \\vec x\n",
"$$\n",
"\n",
"with a matrix $W$.\n",
"\n",
"and a function $f$:\n",
"\n",
"$$\n",
"f(\\vec y) = f(\\vec y(\\vec x))\n",
"$$\n",
"\n",
"We want compute the gradient of $f$ w.r.t. $\\vec x$:\n",
"$\\nabla_{\\vec x} f$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Backward pass\n",
"\n",
"The gradient (error responsibilities) w.r.t. $\\vec y$ is:\n",
"\n",
"$$\n",
"\\nabla_y f = \\vec \\delta^{(y)} = \\left(\\frac{\\partial f}{\\partial y_1}, \\frac{\\partial f}{\\partial y_2}, \\dots, \\frac{\\partial f}{\\partial y_n}\\right)^T \n",
"$$\n",
"\n",
"\n",
"The gradient (error responsibilities) w.r.t. $\\vec x$ is:\n",
"$$\n",
"\\nabla_x f = \\vec \\delta^{(x)} = \\left(\\frac{\\partial f}{\\partial x_1}, \\frac{\\partial f}{\\partial x_2}, \\dots, \\frac{\\partial f}{\\partial x_n}\\right)^T\n",
"$$\n",
"\n",
"Note: The error responsibilities are propagated backward through the computational graph: Backpropagation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Show that\n",
"$$\n",
"\\vec \\delta^{(x)} = W^T \\vec \\delta^{(y)}\n",
"$$\n",
"\n"
]
}
],
"metadata": {
"celltoolbar": "Slideshow",
"kernelspec": {
"display_name": "deep_teaching_kernel",
"language": "python",
"name": "deep_teaching_kernel"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
This diff is collapsed.
......@@ -420,10 +420,10 @@
"targets = torch.tensor([0,2,1,0], dtype=torch.int64)\n",
"pred[np.arange(4), targets]\n",
"\n",
"costs = cross_entropy(pred, t).item()\n",
"costs = cross_entropy(pred, targets).item()\n",
"print(costs)\n",
"\n",
"# costs shoudl be a float >= 0.0\n",
"# costs should be a float >= 0.0\n",
"assert costs >= 0.0"
]
},
......
......@@ -640,7 +640,11 @@
"source": [
"#### Cost (Loss) over Iterations\n",
"\n",
"Plot of the cost progress vs. iterations."
"Plot of the cost progress vs. iterations.\n",
"\n",
"The output should look similar to the following:\n",
"\n",
"<img src=\"https://gitlab.com/deep.TEACHING/educational-materials/raw/dev/media/klaus/shrinking_costs.png\" alt=\"Internet connection needed\"></img>"
]
},
{
......@@ -670,7 +674,11 @@
"source": [
"#### Decision Boundary After Training\n",
"\n",
"The following function plots the data with the decision boundaries after the training. The model should be trained well enough to seperate most (roughly ~95%) of the data correctly. Use the following code for plotting."
"The following function plots the data with the decision boundaries after the training. The model should be trained well enough to seperate most (roughly ~95%) of the data correctly. Use the following code for plotting.\n",
"\n",
"The output should look similar to the following:\n",
"\n",
"<img src=\"https://gitlab.com/deep.TEACHING/educational-materials/raw/dev/media/klaus/linear_decision_boundary_3_classes.png\" alt=\"Internet connection needed\"></img>"
]
},
{
......
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment