Improve existing Classical Neural Networks to cope with images

Utilising Classical Neural Networks to cope with images

Pengwei Yang
krysertim@gmail.com
SCSLab The University of Sydney J12/1 Cleveland St Sydney, NSW 2006 Australia

Abstract

Convolutional neural networks(CNNs) are used worldwide now, it shows outstanding performance when dealing with big data, especially with image data.
In this report, we build two kinds of convolutional neural networks based on two well-known CNN architectures(i.e. LetNet-5 and VGGNet-16) and select partial EMNIST-ByClass data as our dataset(i.e. 30000 for training, 5000 for testing). Besides, we use a support vector machine(SVM) to see whether CNNs are outperforming other models when dealing with images. We use two main methods for our model tunning, namely Keras Tuner and plot the graph. The former is used for model structure tunning and the latter is used for batch size and epoch number tunning.
This report delineates how we change the structure of the original model and how we tune the hyperparameters that we choose. And furtherly compare three different types of models and go in-depth research on why one of them outperforms the others.
Keywords—CNNs, Keras Tuner, SVM, LetNet-5, VGGNet-16, EMNIST-ByClass

1. Introduction

Machine learning(ML) techniques are playing an important role both in daily life and in the laboratory. As a core part of Machine learning, Convolutional Neural Networks(CNNs) are shown an outstanding performance when dealing with image data(Witten et al, 2016). In 1998, one extraordinary paper was released by Professor LeCun showing a brand new CNN architecture(LeCun et al, 1998) that can get high performance when dealing with the MINIST dataset. It can be seen as a milestone in the development of CNNs. Even though it is quite small compared with CNN architectures nowadays, LeNet-5 include all fundamental components(i.e. Convolutional layer, pooling layer and fully connected layer) of the current CNN architecture. After that, more high-performance CNN architectures are released in the ImageNet Large Scale Visual Recognition Challenge(ILSVRC), such as AlexNet(Krizhevsky et al, 2017), VGGNet etc.
In this report, we build three types of classification models. One is based on the Support Vector Machine(SVM) algorithm and the remaining two are both based on CNN architecture. We aim to discover the different performances between the CNN model and other models to see which is the better choice when dealing with the EMNIST-ByClass dataset. Then, we compare the difference between two different types of CNN architecture to find which one is better. According to our experiment result, VGGNet-16 is our best model and SVM does not perform very well in this case.
In terms of our dataset option, we choose EMNIST-ByClass as our dataset, which is the handwriting image dataset. Each image has a size of 28×28. One problem is that, as for VGGNet-16 model, the original input size is 244×244 (Simonyan&Zisserman,2014), which means that we have to change the architecture to match our dataset. This report will show how we finely change the architecture and how we build each model and tune the hyperparameters.

2. Previous Work

Handwriting letter and letter recognition is a classical topic not only in pattern recognition but in the relevant field of ML. With the development and application of CNN architecture, model performance has witnessed significant progress. But what about the performance of other algorithms such as SVM? We searched for a range of papers and found most of them are not using SVM alone, instead of that, they prefer using methods to extract core features in advance of using SVM, like hybrid Discrete Wavelet Transform(Ghadekar et al, 2018). Feature extracting methods can actually enhance the performance of traditional classifiers to some extent.
LeNet-5, a very classical neural network architecture, is released by Professor LeCun(LeCun et al, 1998). This CNN model provides new ideas to academics who research in the deep learning field. Our first CNN model is built based on this architecture. But compared with the original LeNet-5 model, our model change the activation function and introduce some new techniques such as dropout(Baldi et al, 2014). After the release of the LeNet-5 structure, several years later, another outstanding neural network structure, known as VGGNet, is released by Simonyan. Simonyan first published the result of two neural network architectures(Nejad. 2021), which are known as VGGNet-16 and VGGNet-19. In this work, we choose the VGGNet-16 structure. We try to find an optimal neuron number of its fully connected layer and the best learning rate, then we discover that when we tune the batch size and epoch number, the performance will be influenced. So, we select batch size and epoch number as our hyperparameters as well.

3. Methodology

3.1 Pre-processing techniques

Principal component analysis(PCA) is one of the dimensionality reduction methods. This technique is often used in dealing with high dimensional data. The key idea of this technique is to try to map the data into smaller dimensional data and keep the majority variance of data. So, the most important data will be kept and the minor data(i.e. Highly correlated features) will be dropped accordingly, which means that it can save lots of time(Abdi & Williams, 2010). We use the PCA technique in our SVM model in order to speed up our model grid-search time and training time.
Normalisation is usually used to avoid the influences taken by the dominance of attributes over attributes with small values. One well-known normalisation method is what we call MinMax scaling. We use this method in our SVM model. Besides, we try to map our training data into a range of [0,1] when we build our CNNs architecture in order to avoid the gradient vanishing. Normalisation has a significant effect, especially on distance-based Machine Learning algorithms(Witten et al, 2016).

3.2 Classifying methods:

3.2.1 SVM

The support vector machine is a linear classifier with the largest interval in the feature space. In order to correctly divide the training data set, the optimal separation hyperplane (w x+b=0) is obtained by using the interval maximization, which can be formalized as solving convex two optimization algorithms for sub programming.

3.2.2 LeNet-5

Fig.1 (Lecun et al，1998).

LeNet-5 model structure is a series mode, Fig. 1. clearly illustrates the structure of LeNet-5. Some features of LeNet-5 are shown as follows:
a. Each convolutional layer contains 3 parts: convolution, pooling and nonlinear activation function.
b. Capture advanced spatial features using convolutions.
c. Add pooling layer to downsample.
d. Use the activation function of the hyperbolic tangent (Tanh).
e. Finally, use MLP as the classifier.

3.2.3 VGGNet-16

Fig.2 (Simonyan&Zisserman, 2014)

The model structure of VGGNet-16 is shown in Fig. 2. Some features of this model can be seen as follows:
a. A total of 16 layers (excluding Max pooling layer and softmax layer)
b. All convolution kernels use a size of 3×3, and pooling kernels use a size of 2×2
c. Max pooling with stride=1 and padding=0
d. The depth of the convolutional layer is 64 -> 128 -> 256 -> 512 -> 512

4. Experiment and Result

We select EMNIST-ByClass(Cohen et al, 2017) as our dataset, which includes data from more than 800,000. In order to avoid a quite long tunning time, we select the former 30,000 data from the training dataset and 5,000 data from the test dataset. And we normalise all data before using them for model training. Additionally, we split a small set of data(i.e. Validation data) from the training dataset, and use it in the latter two CNNs models to monitor its generalisation performance. This part has shown how we implement and tune three types of models, namely SVM, LetNet-5 and VGGNet-16.

4.1 SVM

In terms of our SVM model. For the data preprocessing, we use MinMax Scaling to map the original data into a new range(i.e.[0, 1]). Besides, On the first try of building our SVM model, we do not use the Principal Component Analysis(PCA) technique. As a result of that, our first SVM model is running for almost two hours which is quite a long time. So, we decide to use PCA for dimensionality reduction. As for EMNIST-ByClass data, which are 28×28(784 dimensions) grey image data. We keep 95% variance of original data and use PCA to get 103 dimensions from 784 dimensional original data.
We select parameter C, kernel function and gamma type as our hyperparameters, and use grid search to search for the combination of these hyperparameters that give the best performance. After the grid search, we get the result of this combination:
{‘C’: 10, ‘gamma’: ‘scale’, ‘kernel’: ‘rbf’}
Finally, we build our final SVM model based on the above combination of hyperparameters. As for model evaluation, by using the classification_report function from the sklearn library(Pedregosa et al, 2011), we get the accuracy, precision and recall of our model. Details of model accuracy, precision and recall are shown separately as follows: 0.80, 0.79 and 0.80. Further analysis comes from the model confusion matrix. In order to clarify our model performance, instead of using the plot_confusion_matrix function from (Pedregosa et al, 2011), we use the heatmap function from the seaborn library. Fig. 3. shows the good performance of our SVM model due to a few scattered dots that are apart from the diagonal line, which means that the majority of data are correctly classified by our SVM model.

Fig. 3. Confusion matrix heatmap of our SVM model, horizontal and vertical lines demonstrate the existed class number. The range of shades demonstrates the different amounts of data.

4.2 LeNet-5

In light of our LeNet-5 model, this is the first CNNs architecture that we choose. LeNet-5 is a very classical CNNs architecture that includes 3 convolutional layers and 2 fully connected layers. The original model structure, which was designed by Lecun(Lecun et al，1998), includes 3 convolutional layers and 2 fully connected layers. Besides, this original model uses the pooling(MaxPooling) technique, with two pooling layers each after the first two convolutional layers. Furthermore, it uses the sigmoid activation function in the second convolutional layer and in the first fully connected layer. Apart from that, in terms of the output layer, the Euclidean distance between the input vector and parameter vector is used in each output Radial Basis Function(RBF) unit to give the final prediction.
In comparison with the original model, our baseline LeNet-5 model uses the softmax activation function instead of RBF, because Keras does not support the RBF activation function in the fully connected layer. And we set an Adam optimiser(Kingma et al, 2014), which is a computationally efficient method with little memory requirement. We get our LeNet-5 baseline model accuracy equals 0.80.
In regard to our LeNet-5 model tunning, firstly, we use the Relu activation function instead of the sigmoid due to its advantages(i.e. faster computational speed and less vanishing gradient)(Thakur, 2022). Besides, we use the dropout technique(Baldi&Sadowski, 2014) in order to save model tunning time and avoid the overfitting problem, and we set dropout = 0.2, which means that our model randomly drops 20% of neurons in the first fully connected layer. Next, we use Keras Tuner(Keras Team, n.d.) to search for the filter number in the first convolutional layer and the neuron number in the first fully connected layer. After searching, we get the best combination, 14 instead 6 and 90 instead 84. Fig. 4. shows our best Lenet model architecture.
After getting our best LeNet-5 model structure, we try to search for the optimal number of epochs because a too big or too small amount of epochs might cause worse performance or overfitting issues when we train neural networks. Our method is firstly to select a relatively big number of epochs, then plot the carve to demonstrate the relation between epoch number and performance, after which we select a suitable epoch number according to that line graph. Fig. 5. demonstrates the above-mentioned relation. According to this graph, we select epochs number 12.

Fig. 4. Our final LeNet-5 model architecture.

As for the evaluation of our final LeNet-5 model, it gets a good performance at accuracy, precision and recall, namely 0.82, 0.80 and 0.82. We can see its good performance in the heatmap of the confusion matrix as well. Fig. 6. shows

Fig. 5. Relation between epoch number and performance.

the detail of the confusion matrix heatmap of our final LeNet-5 model.

Fig. 6. Confusion matrix heatmap of our LeNet-5 model, horizontal and vertical lines demonstrate the existed class number. The range of shades demonstrates the different amounts of data.

4.3 VGGNet-16

VGGNet architecture is designed in 2014(Simonyan&Zisserman, 2014), with a concise structure. This neural network structure deploys the same kernel size(i.e. 3×3) in all convolutional layers. By stacking convolutional layers and pooling layers, Simonyan and Zisserman offer six variants of VGGNet, namely A, A-LRN, B, C, D and E. This report adopts the D structure of VGGNet, which is also known as VGGNet-16. In general, it includes five stacks of convolutional layers, with each including two or three convolutional layers and a pooling layer after that. Apart from that, the classical VGGNet-16 model includes two fully connected layers and one output layer. Additionally, the padding technique is used in every convolutional layer.
Our VGGNet-16 baseline model is built based on the above-mentioned architecture. But compared with the input data size of the original model(i.e. 244×244), our input data size is 28×28, which is much smaller than the former. According to the below formula, which is relevant to the downsampling process,
W2=(W1-F)/S+1 (1)
H2=(H1-F)/S+1 (2)
D2=D1 (3)
W, H and D separately represent width, height and depth of the convolutional layer. F represents the filter size. By using this formula, we decide to remove the pooling layer of the fifth convolutional stack to let this model match our input data. Result from that, the only difference between our baseline VGGNet-16 model and classical VGGNet-16 is dropping the pooling layer in the fifth convolutional stack. We get an accuracy of 0.78, which is not very well when compared with the former two algorithms.
So, we try to select several hyperparameters. Firstly, we want to change the filter size of each convolutional layer. But this size is fixed because it plays an important role in extracting image features(Simonyan&Zisserman, 2014).
As a result of that, for the Neural Network architecture, we only choose the neuron number(from 128 to 640, interval=128) of the second fully connected layer and learning rate(0.01, 0.001, 0.0001) as our first hyperparameter set. One function of the Keras(Keras Team, n.d.) library called Hyperband is very useful in this case. After the searching period, we get the best performance set, neuron number equals 256 and learning rate equals 0.0001. One thing that is worth mentioning is that the best number is the same as the classical model. Fig. 7. shows the detail of our best VGGNet-16 model.

Fig. 7. Our final VGGNet-16 model architecture.

Batch and epoch numbers can influence the CNNs model performance(He et al, 2018). So, after getting the final neural network model, we try to get the best combination of batch size and epoch number in the next step. We choose a small batch size set starting from 32 and ending up to 256 because VGGNet is a rather big model that includes too many layers, which means that with larger batch size, the model possible occupies much more memory. The key idea that how we choose the optimal combination of batch size and epoch number is plotting different line graphs with different batch sizes and showing the relation between the epoch number and model performance. Fig. 8. demonstrate the above-mentioned relation.

Fig. 8. Line graphs with different batch sizes show the relation between epoch number and model performance.

According to Fig. 8., it is obvious that the batch size of 32 is the best choice,

Fig. 9. Confusion matrix heatmap of our VGGNet-16 model, horizontal and vertical lines demonstrate the existed class number. The range of shades demonstrates the different amounts of data.

and the epoch number should be 20. Finally, based on the above-mentioned hyperparameters, we train our model using 30000 training data and get the best accuracy among all of our algorithms, namely, accuracy equals 0.83, precision equals 0.80 and recall equals 0.83. Then, we plot the confusion matrix heatmap, which is shown in Fig. 9.

4.4 Comparision

This report shows three different classification algorithms, namely SVM, LeNet-5 and VGGNet-16. Making comparisons among the above-mentioned methods, VGGNet-16 gives the best performance.
Firstly, as for running time consumption, we train our CNN models based on Colaboratory Pro, which offers us Tesla P100 GPU. Our LeNet-5 model consumes 30 seconds of training time, and our VGGNet-16 costs 6 minutes for training. Besides, we do not use GPU for our SVM model training. As a result of that, our SVM model costs 1 minute for training in this case. According to the above-mentioned time consumption, we discover that VGGNet-16 consume the longest time for training and LeNet-5 cost the shortest time. It is possible because of three factors, namely its architecture of model, batch size and epoch number.

Table 1 clearly illustrates the evaluation results of each algorithm. According to Table 1, it is obviously that convolutional neural network architecture outperforms the SVM algorithm. The reason is that the CNNs algorithm includes the convolutional layer and the pooling layer, which are the outstanding way to extract features from the original image. Then, all of the important data are sent to the fully connected layer, and by using the backpropagation method, the classification model will get good trained and in turn, get excellent performance(H. et al., 2016).
In terms of the various structures of CNNs, LeNet-5 and VGGNet-16 in this case, the latter show better performance when dealing with the EMNIST dataset. Why does it happen? We pay attention to their different layer setting. Having a comparison between the above two models, we find that VGGNet-16 has more convolutional layers and a smaller kernel size. Further, when LeNet-5 get a feature map size of 1×1 by using a 5×5 kernel(Lecun et al，1998), VGGNet uses two convolutional layers that use 3×3 kernels(Simonyan&Zisserman, 2014) to get the same feature map size. In this way, the above-mentioned structure might enforce the learning performance by extracting features with more details. So, that is why our VGGNet model gets the best performance in this case.

5. Conclusion

In this work, we test the performance of three types of classification models by using the EMNIST-ByClass dataset. And we slightly change their structure in order to further match the above-mentioned dataset. We discover VGGNet-16 based model has the best performance and SVM not doing very well in this case, which demonstrates that the Convolutional Neural Network models have a good performance when dealing with image data, and shows that various architectures of CNNs might get different performances.
As for the shortage of this work, firstly, we are not select kernel size in the convolutional layer as our hyperparameter. Besides, we are not searching for a large range of values of each hyperparameter due to the overwhelming running time. Thirdly, we only choose partial data from the original dataset, which means that we are not training our model using the full dataset. So, in the future, we will use the whole dataset for model training and use more data preprocessing techniques to do the Feature Engineering. Besides, we will try more architectures to see which one has the best performance.

6. References

1.Abdi, & Williams, L. J. (2010). Principal component analysis. Wiley Interdisciplinary Reviews. Computational Statistics, 2(4), 433–459. doi: 10.1002/wics.101
2.Baldi, P., & Sadowski, P. (2014). The dropout learning algorithm. Artificial Intelligence, 210, 78–122.https://doi.org/10.1016/j.artint.2014.02.004
3.Cohen, G., Afshar, S., Tapson, J., & van Schaik, A. (2017, February 17). EMNIST: an extension of MNIST to handwritten letters. Retrieved May 11, 2022, from https://arxiv.org/abs/1702.05373
4.Ghadekar, P., Ingole, S., & Sonone, D. (2018). Handwritten Digit and Letter Recognition Using Hybrid DWT-DCT with KNN and SVM Classifier. 2018 Fourth International Conference on Computing Communication Control and Automation (ICCUBEA), 1–6. IEEE. https://doi.org/10.1109/ICCUBEA.2018.8697684
5.He, T., Zhang, Z., Zhang, H., Zhang, Z., Xie, J., & Li, M. (2018, December 4). Bag of Tricks for Image Classification with Convolutional Neural Networks. Retrieved May 12, 2022, from https://arxiv.org/abs/1812.01187
6.Keras Team. (n.d.). Keras documentation: KerasTuner API. Retrieved May 14, 2022, from https://keras.io/api/keras_tuner/
7.Kingma, D. P. (2014, December 22). Adam: A Method for Stochastic Optimization. Retrieved May 17, 2022, from https://arxiv.org/abs/1412.6980
8.Krizhevsky, A., Sutskever, I., & Hinton, G. (2017). ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6), 84–90. https://doi.org/10.1145/3065386
9.Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278–2324. https://doi.org/10.1109/5.726791
10.Nejad, A. (2021, January 19). Convolutional Neural Network Champions — Part 3: VGGNet (TensorFlow 2.x). Retrieved May 18, 2022, from https://towardsdatascience.com/convolutional-neural-network-champions-part-3-vggnet-tensorflow-2-x-ddad77492d96
11.Thakur, A. (2022, May 11). ReLU vs. Sigmoid Function in Deep Neural Networks. Retrieved May 17, 2022, from https://wandb.ai/ayush-thakur/dl-question-bank/reports/ReLU-vs-Sigmoid-Function-in-Deep-Neural-Networks-Why-ReLU-is-so-Prevalent–VmlldzoyMDk0MzI
12.Witten, I. H. (Ian H. ., Frank, E., Hall, M. A., & Pal, C. J. (2016). Data mining practical machine learning tools and techniques (4th ed.). Amsterdam: Elsevier

7. Appendix

7.1 Code Running Illustration

7.1.1 SVM model

‘Group97_other_algorithm1.ipynb’ is the code of our SVM model. The whole process is clarified as follows(for each cell):
Part1: Setup and data preprocessing
Install emnist library→Import all useful libraries→Downloead and read original training dataset and test dataset→Extract 30000 training data and 5000 test data and using MinMax Normalisation, then concatenate all data→Using PCA technique on all data→Extract training data and test data
Part2: Grid-Search
Using grid search to get best combination of hyperparameters, and print the result and accuracy
Part3: Build final SVM model and evaluate it
Build model based on the best hyperparameters, then get predicts→Print predicts→Print accuracy, precision and recall of each class→Plot confusion matrix

7.1.2 LeNet-5 model

‘group97_other_algorithm2.ipynb’ is the code of our LeNet-5 model. The whole process is clarified as follows(for each cell):
Part1: Setup and data preprocessing
Install emnist library→Install keras-tuner→Import useful libraries→Download and read original training dataset and test dataset→Extract 30000 training data and 5000 test data and doing Normalisation, then split training data to a new training data and validation dataset→Add a new dimension
Part2: Build a baseline LeNet-5 model
Build baseline model architecture and train model→Get the baseline accuracy of using test dataset
Part3: Using Keras Tuner for hyperparameters tunning
Build the model that can be tuned by Keras Tuner→check the hyperparameters→searching the best combination of hyperparameters→check the model with the best hyperparameters→Plot the relation between epoch number and model performance
Part4: Build the final LeNet-5 model
Build the best model architecture→Concatenate training data and validation data, and use the new dataset train model→Turn the probability(i.e. The format of y_pred) to category(i.e. 1,2 etc.)→Print accuracy, precision and recall→Plot confusion matrix

7.1.3 VGGNet-16 model

The whole process is clarified as follows(for each cell):
Part1: Setup and data preprocessing
Install emnist library→Install keras-tuner→Import useful libraries→Download and read original training dataset and test dataset→Extract 30000 training data and 5000 test data and doing Normalisation, then split training data to a new training data and validation dataset→Add a new dimension
Part2: Build a baseline VGGNet-16 model
Build baseline model architecture and train model→Get the baseline accuracy of using test dataset
Part3: Using Keras Tuner for hyperparameters tunning
Build the model that can be tuned by Keras Tuner→check the hyperparameters→searching the best combination of hyperparameters→check the model with the best hyperparameters→Plot the relation among batch size, epoch number and model performance
Part4: Build the final VGGNet-16 model
Build the best model architecture→Concatenate training data and validation data, and use the new dataset train model→Turn the probability(i.e. The format of y_pred) to category(i.e. 1,2 etc.)→Print accuracy, precision and recall→Plot confusion matrix→Save our trained model

7.2 Hardware Environment

Colaboratory Pro: GPU: Tesla P100

7.3 Software Environment

Version of Python: Python 3
Used Packages: Pandas, Sklearn, Numpy, Emnist, Seaborn, Kerastuner

SVM

#**SVM** model:

#Setup and Data Preprocessing

pip install emnist

import numpy as np

from emnist import extract_training_samples, extract_test_samples

from sklearn.preprocessing import MinMaxScaler
from sklearn.decomposition import PCA

from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV

from sklearn import metrics
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix

import seaborn as sn
import pandas as pd

X_ta, y_ta = extract_training_samples('byclass')
X_te, y_te = extract_test_samples('byclass')
X_ta.shape
X_te.shape

#Select 30000 data from training dataset and 5000 data from test dataset.

#Using **MinMax Normalisation**

X_train = X_ta[:30000]
y_ta = y_ta[:30000]
X_test = X_te[:5000]
y_te = y_te[:5000]
X_train = X_train.reshape(30000, 28*28)
X_test = X_test.reshape(5000, 28*28)
tool = MinMaxScaler() 
tool.fit(X_train)
X_train = tool.transform(X_train)
X_test = tool.transform(X_test) 

all = np.concatenate((X_train,X_test))
all.shape

#Using **PCA** technique

pca=PCA(n_components=0.95) #Choose 95% variance
all_P = pca.fit_transform(all)
all_P.shape

X_train_P = all_P[:X_train.shape[0]] #Extract training data
X_test_P = all_P[X_train.shape[0]:]
X_test_P.shape

#Using **grid search** for best hyperparamter combination searching

param_grid = {
   
   'kernel':