MSc/MEng Data Mining and Machine Learning (2024) Lab 4 – Neural NetworksPython

Java Python MSc/MEng Data Mining and Machine Learning (2024)

Lab 4 – Neural Networks

Problem

The challenge is to implement the Error Back-Propagation (EBP) training algorithm for a multi-layer perceptron (MLP) 4-2-4 encoder [1] using Matlab (or Python if you wish).   Intuitively the structure of the encoder is as shown in Figure 1.

Figure 1: MLP structure for 4-2-4 encoder

The MLP has an input layer with 4 units, a single hidden layer with 2 hidden units, and an output layer with 4 units.  Each unit has a  sigmoid activation function. The task of the encoder is to  map the following inputs onto outputs:

Table 1: Input-output pairs for the 4-2-4 encoder

The problem is that this has to be achieved through the 2-unit “bottle-neck” hidden layer.  Rumelhart, Hinton and Williams demonstrate that to achieve this, the MLP learns binary encoding in the hidden layer.

Input (and target) pattern

There are 4 input patterns, and the targets are equal to the inputs (Table 1).  Recall that the output oj of the jth  unit in the network is given by:

where netj  is the input to the jth unit.  The values of oj  converge towards 0 and 1 as the magnitude of netj  becomes large, but the values 0 and 1 are never realised.  Hence for practical purposes it is better to replace, for example,  1, 0, 0, 0  in Table 1 with a `softer’ version  0.9, 0.1, 0.1, 0.1.

Since there are only these 4 input/output pairs, the training set consists of just 4 input/output pairs.

Structure of the program

The program needs to run the EBP weight updating process multiple times.  So you will need a variable N for the number of iterations and an outer loop (for n=1:1:N).  You could terminate the process when the change in error drops below a threshold but this is simpler for the moment.  In addition you will need a second inner loop (for d=1:1:4) to cycle through the 4 input patterns in each iteration. But before you do this you need to set up some basic structures:

•     You will  need two arrays W1 and W2 to store the weights  between the input and hidden, and hidden and output layers, respectively.  I suggest that you make W1of size 4x2 and W2of size 2x4. You will need to initialize these arrays (randomly?).  Given an output x from the input layer, the input y to the hidden layer is given by:

y = W1’*x;

Note the transpose!

•     The output from the hidden layer is obtained by applying the  MSc/MEng Data Mining and Machine Learning (2024) Lab 4 – Neural NetworksPython ;sigmoid function to y, so you will need to write a function to implement this function.

•     Once you have propagated the input to the output layer you can calculate the error.   In fact the only use you have for the error is to plot it to help confirm that your code is working.

•     Now you need to back-propagate to calculate δj  for every unit j in the output and hidden layers. First you need to calculate δj  for every output unit (see Eq. (12) in the slides).  Then you need to apply back-propagation to calculate δj  for every hidden unit (again see Eq. (12) in the slides).  To back-propagate the vector of δjs from the output layer to the hidden layer you just need to multiply by W2 (no transpose this time):

deltaH = W2*deltaO;

where deltaO and deltaH are the deltas in the output and hidden layers, respectively.  Note that the above equation is only a part of the equation needed (just to indicate the use of the matrices for doing this calculation) – the full equation is Eq. (12) in the slides.

Once you have calculated the δjs for the output and hidden layers you can calculate ∆wij  = −θδjoi

(see slide 19 from the lecture).

•     Finally, you can update the weights: wij   → wij  + ∆wij

I suggest you do all this about 2,000 times (for n=1:1:N, N=2000) and plot the error as a function of n.

Practical considerations

If you implement all of this properly you will see that the error decreases as a function of n.  You should explore the learning rate, different initialisations of the weight matrices, different numbers of iterations, `softening’ the input/output patterns, etc.   However, you will find that whatever you do you cannot get the error to reduce to zero.  To do this I think you need to add bias units to the input and hidden layers.

Lab-Report Submission

Submit your lab report and source code.   Your  lab report should include thorough comments on experimental evaluations and results obtained.  Include the listing of your source code in appendix of the lab report.

References

[1] D. E. Rumelhart, G. E. Hinton, and R. J. Williams (1986), “Learning Internal Representations by Error Propagation”,  In:  Rumelhart,  D.E.,  McClelland,  J.L.  and  the   PDP  Research  Group,   Eds.,   Parallel Distributed  Processing:  Explorations  in  the  Microstructure  of  Cognition,  Vol.  1:  Foundations,  MIT Press, Cambridge, MA, 318-362