Khedidja Derdour, Hayet Mouss and Rafik Bensaadi

Industrial Engineering Department, Automatic and Manufacturing Laboratory Batna 2 university, Algeria khadidja.derdour@univ-msila.dz, h.mouss@univ-batna2.dz, r.bensaadi@univ-batna2.dz

Abstract: In this paper, we present a system for handwriting digit recognition using different invariant features extraction and multiple classifiers. In the feature extraction we use four types: cavities, Zernike moments, Hu moments, Histogram of Gradient (HOG). Firstly, the features are used independently by five classifiers: K-nearest neighbor (KNN), Support Vector Machines (SVM) one versus one, SVM one versus all, Decision Tree, MLP. Then to achieve the best possible classification performance in terms of recognition rate, three methods of classifiers Combination rule employed: majority vote, Borda count and maximum rule. Experiments are performed on the well-known MNIST database of handwritten digits. The results demonstrated that the combination of KNN using HOG features with SVMOVA using Zernike moments by Borda count rule have considered to be good based on a geometric transformation invariance.

Keywords: Handwritten digit recognition, Invariant features extraction, Multiple classifier, classification.

1. Introduction

Handwritten digit recognition has been successfully applied in various fields where, the accuracy of recognition is significant such as recognition of the postal codes on the envelope and banking account number or amount written on the check, automatic reading of zip codes and many more, several approaches have been used in handwritten recognition. However, the accuracy of recognition varies between one and another. The task of handwritten digit classification is hard problem and complex in pattern recognition and have a high level of research difficulty, since the variation of digits in each class of digit, this is due to the personal style of each writer and different form of digit. writer can represent the digit in different ways, all people have different writing style that is not easily understand by the machine. Past and recent works in this field have concentrated on various languages including Chinese and Japanese Persian Latin, Devanagari, Bangla and Urdu, Arabic, Farsi, etc.

The two important phases in recognition system are Extraction of features and classification methods. A suitable features extraction and good classifier play a very important role in a recognition system to improve the recognition accuracy. The features selected to represent a digit affect on the performance of digit recognition systems. The selected feature sets must verify a large separation between class and a low intraclass variance. in [51] we found several features extraction methods used with Neural Network based Algorithms. In literature many different features are proposed, including, vertical and horizontal projection histograms with dynamic thresholding. Projection histograms are combined with other feature sets and usually used in printed digit recognition. Zoning, Cavity, profile, Freeman code, end point, Histogram of Oriented Gradient (HOG), Fourier transform, Discret cosine transform, Hough transform, wavelet transform, Gabor filter, …etc., Zernike invariant moments, Hu moments, complex moments, geometric invariant moments, affine invariant moments, Legendre moments, Fourier Mellin moments and pseudo-Zernike moments, Mojette transform etc. are the current choices for features. Various moments have been used in Arabic digit or other languages, In the literature Some works based on moment features is discussed in [5, 6, 10, 11, 30, 34, 40].

Different classifiers can be used for handwritten digit recognition include artificial neural network (ANN) asthe Multilayer perceptron, (MLP), Decision tree, Linear discriminant function (LDF), Bayes classifiers, Parzen windows, Hidden Markov Models (HMM), K-nearest neighbor

Received: November 12th, 2019. Accepted: March 20th, 2021

DOI: 10.15676/ijeei.2021.13.1.9

(KNN) classifier and support vector machines (SVM) which are more successfully implemented because of it has great classification potential and large discrimination abilities. Recently, approaches based on the deep neural networks (DNNs) have shown excellent performance in many applications of pattern recognition and machine learning. Many methods in the state of the art on handwritten digit recognition problems employ a deep neural networks gives good accuracies, see recent works [1, 2, 3, 4, 7, 20, 21, 12, 13, 14, 15,19, 38, 42]. The paper [51] is a survey on using neural network based algorithms, present the past works done on the handwritten digit recognition.

The combination of classifier decisions is a topic studied, recently, multiple classifier systems based on the integration of different classifiers have been proposed for the classification performance improvement, The Goal digit recognition system is to achieve the best possible recognition rate, this objective motivated the interest of classifiers combining. Many strategies of classifiers combination can be used in order to improve recognition error rates, in [23, 25, 26, 27, 28, 19, 31, 32, 35, 55] combining classifiers method have been studied extensively.

The outline of the paper is as follows: a brief review of some recent work and previous approaches of handwritten digit recognition is presented in Section 2. Then we introduce our proposed system in section 3. Whereas, section 4 describes the features extraction used, and we present the classifiers implemented in section 5. Section 6 exposes the experimental results. Finally, we conclude the paper in section 7.

2. Related Work

In last year Recent work propose different method for Arabic digit recognition and various languages, in [32] an optimization proposed to adjust recent swarm intelligence bat algorithm. Prove that with using weak set of features as histogram projections, the proposed algorithm would give good results. Complementary set of 115 features and the ensemble classifier algorithm is used in [36].

The paper [37] presents reliable approaches for recognition of handwritten digits by different classifiers as Naïve Bayes, Multilayer Perceptron, Random Forest, Support Vector Machine, Bayes Net, J48 and Random Tree has been used for the recognition of digits using WEKA, the recognition process achieves 90.37%.

In [39] a novel system for classification of writer independent off-line handwritten Persian two digit numerals is proposed based on a combination of HMM and SVM, employing Gabor filter bank (24, 12, 6 and 3 scales) in 6 directions (0, 30, 60, 90, 120, 150 degrees). The digit recognition rate is 98.75 %, while the SVM recognition rate is 98.58% on isolated characters and 95.93% with HMM.

Reference [40] presents different features types based on gradient features, geometric invariants and Zernike moments, normalized by several techniques using SVM (radial basis function) classifier. The results are promising on MNIST database. Gradient features achieve the best results 99.16%. Various handwritten numeral databases (MNIST, CENPARMI, CEDAR, USPS) used in [41] employing skeleton features, number of contours, Number of watersheds, and ratio between the upper half part and lower half-part pixels number of the digit image with K–NN and SVM classifiers. The proposed method novelty is size invariant.

Different features which include average zone values, number of starting, chain vector and intersection points are used in [42] for 600 images of eastern Arabic handwriting digit. Features based comparison and several classifiers neural network, SVM, deep neural network, decision tree, nearest, ensemble classifiers are presented.

Authors in [43] use several techniques based on Convolutional Neural Networks (CNN), Deep Belief Network (DBN), CNN with dropout, CNN with dropout and Gabor filters, CNN with dropout and Gaussian filters, for Bangla digit recognition. These networks have a high degree of invariance to pattern distortions and transformation geometric (scaling, translation and other). The work presented in [44] study the offline handwritten numeral recognition of multilingual digit data set of Devanagari and English digits, using the representation learning for dimensionality reduction implemented in SVM-based classifier.

In reference [45] a study is made to recognize, Telugu, Devanagari, Bangla and Arabic using Mojette transform with the Principal Component Analysis (PCA) for feature dimensionality reduction and also shortening the training time. Finally, multiple classifiers (SVM, MLP, Naïve Bayes, Random forest, Bagging) are used to test 48-element feature vector on CMATER handwritten digit databases, an accuracy mean of 98.17 % is given by MLP classifier.

In the paper [46] a triangle features are improved by the combination of the gradient and ratio features, using four types of datasets MNIST, BANGLA, IFCHDB and HODA. In the experiment an accuracy of 93.18% is achieved with SVM classifier and 96.51% using MLP.

The paper [47] propose a novel technique for Arabic/Farsi handwritten digit recognition. an efficient and invariant feature set is constructed by combination of Histogram of Oriented Gradient (HOG) and four directional Chain Code Histogram (CCH), using SVM classifier with radial basis function kernel.

Reference [48] presents a handwritten digit recognition system for Indian subcontinent digit scripts, namely, Devanagari, Indo-Arabic, Roman, Bangla, and Telugu. A combination of six different types of moments (affine moment invariant, geometric moment, Zernike moment, moment invariant, Legendre moment, and complex moment) give 130-element feature, is evaluated using multiple classifiers on CMATER and MNIST databases. Results prove that recognition accuracies attained are satisfactory and MLP classifier outperforms the others.

The paper [49] presents novel feature extraction methods which is distance and slope based curvature coding. Using cascaded classifiers of KNN and SVM. KNN does the primary classification while the postprocessor SVM used to select of two probable close classes the final output class. An acceptable accuracy of 99.26% is achieved on Arabic digits.

Handwritten digit recognition model given by an adaptation of a previous theory of face recognition is presented in [12]. The model realizes rotation invariance and translation in a principled way instead of extensive learning based from big data masses. accuracy achieved is 96.10%.

A survey of several techniques used for handwritten digit recognition system is carried out in [50]. This review discusses the neural network application and its variants, and a survey of the use of NN for digit recognition is presented.

The current State of the art accuracy in handwritten digit classification of MNIST database is reported in reference [8, 29], Many methods have been tested with this database, details about the methods are given in cited paper.

3. Recognition System Architecture

Framework of our approach using isolated handwritten digits of MNIST database is presented in "Fig 1." The intermediate steps involved are summarized as below: image preprocessing where the input images are stored in a matrix and binarized using the operation of binarization. Then we employed various features type include Hu moments, Zernike moments, Hog and cavities. We were interested to extract the most discriminating characteristic structures of digit "the cavities" which are most effective in preserving the separability of the classes. We have chosen the moments Hu and the moments of Zernike because they verify the properties of Independence to the geometric transformation (translation, rotation and scale). The HOG technical realize the geometric and photometric transformations invariance, except for object orientation.

To achieve automatic recognition of digits, it is necessary to provide these parameters (the calculate discriminating parameters) to a classifier. Classification (recognition) step used multiple classifiers: KNN, MLP, Decision tree, the most used and the most efficient classifiers in the field of writing, and the very powerful classifier SVM (one versus one –SVMOVO-, one versus all –SVMOVA-). Finally, in this study, classifier combination with three rules used to improve the recognition rate so to determine the necessity or no of the classifiers combination with these different features.

Figure 1. A framework of proposed system

The experiments were carried out on the MNIST data set available at [8] of 60000 images of digits for training and 10000 images of digits for testing, it is a subset from NIST. The images were centered in a 28x28 pixel by computing the gravity center of digits, and translating the image where this point placed at the center of the 28x28 field. Figure 2 shows examples from MNIST database. We note that we use this database without any preprocessing to enhancement image quality such as noise removal etc.

Figure 2. Subset of MNIST Arabic handwritten numeral

4. Feature Extraction

Extraction of features have a significant role for patterns differentiation. In this stage we present features used: cavity, Histogram of gradient (HOG), hole number and two types of invariant features, Hu moment and Zernike moments. A lot of useful information about a binary Image can be obtained from the moments.

A. Hu moments

There are seven Hu moments [52] are characterized to be invariant to orientation, size and position of digit, the computation steps of Hu moments are described as below:

Compute the geometrical moments value, M_ij of order (i + j) for binary image until third order:

\[\begin{aligned} M_{ij} &= \sum_{k=1}^{K} \sum_{l=1}^{L} (k)^{i} (l)^{j} g(k,l) \\ g(k,l) &: \text{ is image function (pixel).} \\ K, L &: \text{ are image dimensions.} \end{aligned} \tag{1}\]

• Compute the intensity moment (centroid or gravity center of image) (x_g y_g) of image with formula:

\[x_g = \frac{M_{10}}{M_{00}}\], \(y_g = \frac{M_{01}}{M_{00}}\) (2)

• Compute the geometrical central moments \(\mu_{ij}\) with formula(3), \(\mu_{ij}\) are invariants to translation

\[\mu_{ij} = \sum_{k=1}^{K} \sum_{l=1}^{L} (k - x_g)^{i} (l - y_g)^{j} g(k, l)\] (3)

• Compute normalized Geometrical central moment \(\eta_{pq}\) which are invariant to scaling

\[\eta_{ij} = \frac{\mu_{ij}}{\mu_{00}^{(i+j+2)/2}} \tag{4}\]

Finally, Compute Hu moments, Ø₁ to Ø₇ with respect to scale, translation and rotation invariants:

\[\text{[rumus tidak dapat ditampilkan dengan baik — lihat PDF asli]}\]

The 7 Hu moments defined form normalized geometrical central moment \(\eta_{ij}\) of order 2 and 3, which are invariant to scaling, translation and rotation.

B. Zernike Moments

Zernike moments \(A_{nl}\) are complex number (complex polynomial) The computation steps are described as below:

• Compute the geometrical moments \(M_{ij}\) (1), geometrical central \(\mu_{ij}\) (3)
• Compute Zernike moment with respect the invariance at translation, scale and rotation with formula [10] [11]

\[A_{nl} = [(n+1)/\pi] \sum_{k=l}^{n} \sum_{j=0}^{q} \sum_{m=0}^{l} (-i)^{m} {q \choose j} {l \choose m} B_{nlk} \ \mu_{k-2j-l+m,2j+l-m}\] with : q=(k-l)/2, (l-k) et (n-k) even and

\[B_{nlk} = (-1)^{(n-k)/2} [(n+k)/2]! / [(n-k)/2]! [(l+k)/2]! [(k-l)/2]!\] (13)

\(\binom{j}{s} = \binom{j!}{(s! * (j-s)!)}\) is the s element combination from j.

The List of 12 order of invariant Zernike moments, is given in "Table 1" which contains 49 variables, used in experiments.

Table 1. The 12 Order Zernike Moments

In the following we propose the expression of nine moments calculated from the application of previous equations, and we note that the first two moments are independent (have fixed values), So we can do without in experiments and use and use 47 moments instead of 49:

\[A_{00} = \mu_{00}/\pi = 1/\pi \tag{14}\]

\[A_{11} = A_{1-1} = 0 \tag{15}\]

\[A_{22} = 3(\mu_{02} - \mu_{20} - 2i\mu_{11})/\pi \tag{16}\]

\[A_{20} = 3(2\mu_{20} + 2\mu_{02} - 1)/\pi \tag{17}\]

\[A_{33} = 4[(\mu_{03} - 3\mu_{21} - i(\mu_{30} - 3\mu_{12})]/\pi\] (18)

\[A_{31} = 12[(\mu_{03} - \mu_{21} - i(\mu_{30} - \mu_{12})]/\pi\] (19)

\[A_{44} = 5[\mu_{40} - 6\mu_{22} - \mu_{04} + 4i(\mu_{31} - \mu_{13})]/\pi\] (20)

\[\begin{aligned} A_{44} &= 5[\mu_{40} - 6\mu_{22} - \mu_{04} + 4i(\mu_{31} - \mu_{13})]/\pi \\ A_{42} &= 5\{4(\mu_{04} - \mu_{40}) + 3(\mu_{20} - \mu_{02}) - 2i[4(\mu_{31} - \mu_{13}) - 3\mu_{11}]\}/\pi \end{aligned} \tag{20}\]

\[A_{40} = 5[6(\mu_{40} + 2\mu_{22} + \mu_{04}) - 6(\mu_{20} + \mu_{02}) + 1]/\pi\] (22)

C. Histograms of Oriented Gradients

Figure 3. Blocks and cells of image

The HOG [47, 53, 54] shows shape information of image by the distribution of density of gradient and edge. The basic idea is to divide the window of image into "cells" (size of cells is in pixels), then local regions constructed named blocks by accumulating of cells. Their relation is shown by "Fig 3." In this paper we adopt default parameter settings and we use a block of 8 x8 and the size of cell is 4 x 4.

D. Cavities

The Cavities are existed in the majority of digits; the position and number vary according to the digit and style. There are 5 types of cavities: North, South, East, West and Central. the definition of four cardinal directions is required: North up, West left, etc… [16].

In this work, we have used as features, the number of cavity and the normalised surface of cavities (each cavities divised by the total cavities surface) "fig 4.".

Figure 4. The five cavities: East: red, West: green, North: blue, South: yellow, and Central: pink

5. Classification

The goal of classification is to make a model using features and class labels. Then the classifier used to find class labels of the new instances. This section describe the classifiers used in our study. The paper [33] present Detail description of various supervised machine learning classification techniques used in our work.

A. KNN

K-Nearest Neighbor algorithm has been used in great number of applications in the field of statistical pattern recognition, data mining and many others. KNN is a supervised method for classifying objects by a majority vote of its K (positive integer) neighbors [10]. The neighbors are taken using several distance measures such as: The Euclidean distance, Manhattan, Cityblock, Seuclidean, Minkowsky, Chebychev distance.

B. SVM

The support vector machine is capable of learning and to achieve good generalization performance. Different kernel functions SVM can transform, by finding the optimal separate hyper plane, a nonlinear separable problem into a linear separable problem. Initially, this method was suggested to solve two-class problems. Later, to extend this technique to multiclass classification problems, a few strategies were proposed [17.]. To solve the multiclass problems, two approaches are currently used: one-versus-all (building binary classifiers which distinguish between one of the labels and the rest) and one-versus-one (building binary classifiers which distinguish between every pair of classes).

C. Tree

The decision tree technic achieves the classification by a series of tests on the features. These all tests are organized as a tree (test is the node of this tree and class is the leaf). The response of

each test indicates what next test which must submit this object. A leaf of this tree denotes classes and node is associated to a test [9].

D. MLP

The multilayer perceptron is the most widely known feedback network, it is very powerful and complicated networks. Supervised Learning algorithms used is the most widely known the retro-propagation algorithm or back-propagation algorithm.

The accuracy in general of each classifier is shown in table2, with the speed of learning respecting the number of instance and the number of attributes, also speed of classification (the performance degrees are determined by execution (run) time and recognition rate).

Table 2. Classifiers Comparing (+ Represent Performance Degree)

Recently the new research field of Classifier combination has been investigated to improve recognition reliability by taking into consideration the complementarity between classifiers. in the literature, depending on the output type of information provided by each classifier: class, rank, measure, a many of combination rule has been proposed. you will find the detailed explanation in [28]. Now We briefly explain classifiers combination rules used in our work: Majority vote, is the popular way in the classifiers combination in type class, for each class count the votes and select the majority class. For the type of measure, we use Maximum rule, it selects the classifier producing the highest estimated confidence. from the rank type, we used Borda count: is based on the calculation of the total rank for each of the proposed classes, by calculating the sum of the numbers of classes placed below it by each of the classifiers.

6. Experiment Results and Discussion

We present experimental results in this section, for the MNIST handwritten digit dataset available at [8] described in section 3. The MNIST database has 60,000 images of digit for training, and 10,000 images of digit for testing. We note that the distribution of samples in trainclass and testclass into the ten class is not stratified as presented in table 3.

Table 3. Number and Percentage of Instances of Each Class

The experiment devised to two stages. In the first stage, the extracted features from cavity, Hu moment, Zernike moment and HOG are used in training and classification using five classifiers: KNN with Euclidean distance, decision tree, MLP, SVMOVO, SVMOVA. these features are compared in terms of recognition rate using these classifiers independently. Then, in the second stage, two types of combination are used. Features combination for each classifier, and classifiers combination by three combination rule: majority vote, Borda count and max.

Table 4 shows the recognition rate of each classifier with several features independently (on the use of single feature). We applied the test on the both training set and test set. The results illustrate that the best recognition rate varied between classifiers, with cavities features the best result is given by MLP classifier, Hu and Zernike moments by SVMOVA classifier, HOG feature by KNN classifier.

Table 4. Train and Test set Recognition Rate of Each Classifier

We note that the HOG feature vector is big (size vector is 1296), for this reason and with the capacity reason of our machine the MLP and SVMOVA not us. Test program is written with MATLAB R2012, the running platform is 64-bit Windows 7, the machine processor is Intel Core Intel (R) Core (TM) i7-2670QM CPU @ 2.20GHz, memory is 6GB,

Our SVMOVA achieved 86.97% of recognition rate with 47 Zernike moment which is invariant to geometrical transformation. KNN classifier achieved 96.57% with HOG feature which has a high dimension (of 1296 dimensional feature vector). Zernike moments are better than cavities and Hu moments, but are not so good as HOG features.

To improve the recognition rate another feature characteristic added is: the number of hole. Results obtained in table 5, indicate clearly that enriching the features with number of hole yielded better recognition rate with cavities features and Hu moments and excellent recognition rate with HOG using KNN 97.38%. Comparing to results obtained in [40] for Zernike moments 84.10%, our results illustrate that SVMOVA classifier achieved better recognition rate with Zernike moments 88.42% without any normalization.

Table 5. Train and Test Set Recognition Rate Enriched With Hole Number of Each Classifier

To increase recognition rate we adopt combining classifiers. The results of various experiments we conducted in our study to determine if classifier combination optimize the performance of classification in terms of recognition rate or no. We start with the Combination of classifiers by using different feature sets, in our work each classifier trained with four features types separately (all classifiers use the same features). Then the same classifiers type with various features separately combined (combining classifiers KNN with Cavity Hu moments Zernike moments HOG separately; combining classifiers Decision Tree with Cavity Hu moments Zernike moments HOG separately; combining classifiers MLP with Cavity Hu moments Zernike moments separately; combining classifiers SVMOVO with Cavity Hu moments Zernike moments separately; combining classifiers SVMOVA with Cavity Hu moments Zernike moments separately).

Three methods are used on combining classifiers: majority vote, Borda count and max rule. Results obtained in table 6 have shown that classifiers combining decrease the recognition rate of the best classifier and increase rate recognition of the lowest classifier except SVMOVO classifier where the recognition rate increases with the three methods of combination, especially with Borda count method.

Table 6. Recognition Rate of Each Classifier with Several Features Types

An experimental comparison of various classifier combination using same features type is shown in Table 7 (combining classifiers KNN Decision Tree MLP SVMOVO SVMOVA trained with Cavity feature; combining classifiers KNN Decision Tree MLP SVMOVO SVMOVA Trained with Hu moments feature; combining classifiers KNN Decision Tree MLP SVMOVO SVMOVA Trained with Zernike moments feature; combining classifiers KNN Decision Tree trained with HOG feature). Recognition rate Using cavities augment comparing to the individual classifiers except MLP in majority vote and except MLP decision tree in Borda count but with not big difference. With Hu moments results decrease comparing to MLP and SVMOVA classifiers. recognition rate is significantly improved with Zernike moments comparing to individual classifier especially KNN tree decision and SVMOVO, but comparing to SVMOVA results decrease with low difference. Concerning HOG, result not improved comparing to KNN classifier. Briefly, When the various classifiers with the same features type are combined, performance is only improved in case of the Zernike features. For the other features type the performances of combination are not significantly worse than the best individual classifier.

Table 7. recognition rate using various classifier combination with same features types

Finally, we use different classifiers with various feature set. according to the previous results we will choose the classifier which gives the great recognition rate for a type of characteristic parameter (from one feature set). so we obtain the following combination: KNN-HOG + SVMOVA-Zernike moments+ SVMOVA-Hu moments + MLP-cavities. Results of this combination and other shown in table 8. a recognition rate of 94.18% is obtained with KNN-HOG+SVMOVA-Zernike moments combination using Borda count rule, this value is between the highest recognition rate obtained by KNN-HOG 96.57%, and the recognition rate obtained by SVMOVA- Zernike moments 88.42%.

Due to higher classification performance of SVM, geometric transformation invariance of Zernike moment and because of the big dimension of HOG vector which is used for the rapid classifier KNN, we can say that this combination is very satisfied

Table 8. Recognition Rate Using Various Classifier Combination With Different Features Types

In comparison to other systems in table 9, our experience indicates that the above proposed works is good but not enough. so other improvement is requested such as enriching the feature extraction before giving them to classifiers, i.e. possibly if we add to these vectors other features such as cavity features or end points …etc., the results would be even better. also high recognition rate can achieve based on preprocessing stage (Noise removal, image size normalization -location of the digit-, …), for further processing and feature extraction.

Table 9. Other Results on MNIST Database

The KNN SVM MLP are the most efficient classifiers in the field of writing, so the factor causes the result to be less than 95%.is the features used. The projection histograms missing of the Discriminability property (the large inter-class variance) as between the number "2 and 3", "1 and 7", "0 and 8" i.e. the histograms of 2 and 3 are similar, the same thing for the histograms of 1 and 7 and the histograms of 0 and 8. concerning Improved triangle features which are based on or also known as multi-zoning does not verify the reliability property (low intra-class variance) as for digits "2 and 3 and 5", "1 and 7". regarding Zernike moments which take into account the invariance of geometric transformations this gives the non-separation between 6 and 9 (the rotation of 6 gives 9 and reciprocally).

Figure 5. misclassified samples

After analyzing digits images were misclassified (figure 5) by our model, we found that the Some possible difficulties contributing are can be categorized as follows :

• The variety of handwriting styles influence the appearance of numbers
• The similarity between numbers "1 and 7", "5 and 3", "3 and 8", "9 and 8", "3 and 9" etc.
• The broken digits may be due to poor writing conditions or degraded quality introduced by the scanning procedure and binarization processes.
• The degraded quality of images due to scanning process, writing instrument or size standardization. Parts of the image are broken or noise is generated
• The filled circle caused by the speed of the writer, like the lower circle of the number eight
• The ends touch each other like and form circles the number three

7. Conclusion

In this paper the work is composed of two parts: feature extraction and classification. The main objective of this research is to increase the recognition rates of handwritten digits using features extracted from Zernike moments, Hu moments, Cavities, HOG, with various classifiers KNN, SVM, MLP, Decision tree.

Our work achieved satisfactory results on a widely used database of handwritten digits MNIST. HOG features give the best results, the Hu moments and cavities looks slightly worse, but the Zernike moments are most promising considering that it realises invariance geometric transformation. Perhaps enriching these feature by combined them with other and using image enhancement techniques in preprocessing step will also improve features extraction performance and conducted to better recognition rate. this can also be used in future studies.

The classification part divided to tow stage, in the first stage we compare the performance of separately several classifiers. Using individual classifiers, HOG exhibited the highest recognition rate of 96.57% with KNN classifier and with SVMOVA the best results 86.97% is from Zernike moments.

Our contribution in This research mainly focus on improving classification on terms of recognition rate by combining classifiers. Three methods are used for the combination of classifiers: majority vote, Borda count, max rule. The experiments demonstrated that the combination by Borda count rule of KNN using HOG features with SVMOVA using Zernike moments, have improved significantly recognition rate from 86.97% of Zernike moments to 94.18% and decrease the recognition rate from 96.57% of HOG feature to 94.18%. This decreasing of 2% is not considered with the increasing by 8% and provide the property of transformation geometric invariance. Comparison to other recent work [40,32,46,50] this combination formed a good classifier pair and provided encouraging results.

8. References

• [1]. J. Schmidhuber, "Deep learning in neural networks: An overview", Neural Networks, vol. 61, pp. 85–117, 2015.
• [2]. D. Cireçan, U. Meier, J. Schmidhuber, "Multi-column deep neural networks for image classification", Technical Report No. IDSIA-04-12, 2012.
• [3]. M. D. McDonnell, T. Vladusich, "Enhanced image classification with a fast-learning shallow convolutional neural network", arXiv:1503.04596v3 [cs.NE], 15 Aug 2015.
• [4]. I. Sato, H Nishimura, K Yokoi, APAC, "Augmented pattern classification with neural networks, arXiv:1505.03229v1 [cs.CV], 13 May 2015.
• [5]. G. F. Fitriana, "Handwriting digit recognition using united moment invariant feature extraction and self-organizing maps", The Third ICT International Student Project Conference (ICT-ISPC2014), 2014.
• [6]. S. M. Mali, "Moment and density based handwritten Marathi numeral recognition", Indian Journal of Computer Science and Engineering (IJCSE), 2012.
• [7]. C.-Y. Lee, P. W. Gallagher, Z. Tu, "Generalizing pooling functions in convolutional neural networks: mixed, gated, and tree", arXiv:1509.08985v2 [stat.ML] 10 Oct 2015.
• [8]. Y. LeCun and C. Cortes. "Mnist handwritten digit database". Is Available at https://yann.lecun.com/exdb/mnist, 2018.
• [9]. A. Cornuéjols and L. Miclet. "Apprentissage artificiel,". Eyrolles, 2002.
• [10]. M. R. Teague, "Image analysis via the general theory of moments", Air Force Weapons Laboratory, Beam Control Systems Branch, Kirtland AFB, Albuquerque, New Mexico 87117, Received 1 November 1979.
• [11]. N. Benoudjit, "Reconnaissance de caractères manuscrits par la methode des moments", université catholique de Louvain, faculté des sciences appliqués, département D'électricité, laboratoire D'électronique et de microélectronique, 1999-2000
• [12]. S. Majumder, C. v. d. Malsburg, A. Richhariya, S. Bhanot, "Handwritten digit recognition by elastic matching", 2017.
• [13]. M. Mozafari, M. Ganjtabesh, A. Nowzari-Dalini, S. J. Thorpe, and T. Masquelier "Combining STDP and reward-modulated STDP in deep convolutional spiking neural networks for digit recognition", arXiv:1804.00227v1 [cs.CV] 31 Mar 2018.
• [14]. M. Sensoy, M. Kandemir, L. Kaplan, "Evidential deep learning to quantify classification uncertainty", arXiv:1806.01768v2 [cs.LG] 6 Jun 2018.
• [15]. Y. Shima, Y. Nakashima, M. Yasuda, "Classifying for a Mixture of Object Images and Character Patterns by Using CNN Pre-trained for Large-scale Object Image Dataset", 13th IEEE Conference on Industrial Electronics and Applications (ICIEA), 2018.
• [16]. G. Burel, "Introduction au traitement d'images, simulation sous MATLAB", Edition lavoisier, 2001
• [17]. X.-X. Niun, C. Y. Suen, "A novel hybrid CNN–SVM classifier for recognizing handwritten digits", Pattern Recognition 45, 1318–1325,2012.
• [18]. A. Jain, G.R. K. S. Subrahmanyam and D. Mishra, "rotation invariant digit recognition using convolutional neural network", Proceedings of 2nd International Conference on Computer Vision & Image Processing, Advances in Intelligent Systems and Computing 703, https://doi.org/10.1007/978-981-10-7895-8_8, 2018.
• [19]. Y. Shima,Y. Nakashima, M. Yasuda, "handwritten digits recognition by using CNN alexnet pre-trained for large-scale object image dataset" , 2018, https://doi.org/10.1145/3220162.3220163
• [20]. V. Manjunatha, S. Ramalingam, "Class subset selection for transfer learning using submodularity", arXiv:1804.00060v1 [cs.CV] 30 Mar 2018.
• [21]. S. R. Kulkarni, B. Rajendran, "Spiking neural networks for handwritten digit recognition supervised learning and network optimization", Neural Networks 103, pp 118–127, 2018, https://doi.org/10.1016/j.neunet.2018.03.019

• [22]. R. Salakhutdinov and G. Hinton, "Structure learning a nonlinear embedding by preserving class neighbourhood", proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, PMLR 2:412-419, 2007.
• [23]. P. Moacir, Jr. Ponti., "Combining classifiers: from the creation of ensembles to the decision fusion, DOI: 10.1109/SIBGRAPI-T.2011.9 · Source: dx.doi.org, 2011.
• [24]. J. Duong, R. Sabourin, H. Emptoz, "Proposition d'un classifieur en cascade, application à la reconnaissance de polices de caractères rares" ,2004.
• [25]. J. Ghosh, "Multi classifier systems: back to the future", In: Roli F., Kittler J. Multiple Classifier Systems. MCS, Lecture Notes in Computer Science, vol 2364, Springer, Berlin, Heidelberg, 2002.
• [26]. J. Kittler, M. hatef, R.P. W. Duin, J. matas, "On combining classifiers, IEEE transactions on pattern analysis and machine intelligence vol 20, no 3, march 1998.
• [27]. F. Roli, G. Giacinto, G. Vernazza, "Methods for designing multiple classifier systems". In: Kittler J., Roli F. Multiple Classifier Systems. MCS 2001. Lecture Notes in Computer Science, vol 2096. Springer, Berlin, Heidelberg, 2001.
• [28]. H. Zouari, L. heutte, Y, Lecourtier, A, Alimi, "Un panorama des methods de combinaison de classifieurs en reconnaissance de formes", An overview of classifier combination methods in pattern recognition, in Proc. RFIA, vol 2, pp499-508, 2002.
• [29]. Classification data sets results, http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html
• [30]. S. Nebti1, A. Boukerram1, "Use of Nature-inspired meta-heuristics for handwritten digits recognition", International Journal of Computational Linguistics Research, Volume 1 Number 1, March 2010.
• [31]. H. Zhang, A. C. Berg, M. Maire, J.Malik, "SVM-KNN: discriminative nearest neighbor classification for visual category recognition", 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06) , pp 2126-2136, 2006.
• [32]. E. Tuba, M. Tuba, D. Simian, "Handwritten digit recognition by support vector machine optimized by bat algorithm", 24th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision, (WSCG 2016), pp 369-376
• [33]. S. B. Kotsiantis, "Supervised machine learning: a review of classification techniques", Informatica, vol 31, 249-268, 2007.
• [34]. S. Zekovich, M. Tuba, "Hu moments based handwritten digits recognition algorithm", 12th International Conference on Artificial Intelligence, Knowledge Engineering and Data Bases (AIKED '13), pp. 98 - 103, Feb, 2013.
• [35]. D. Ruta and B. Gabrys, "An overview of classifier fusion methods", Computing and Information Systems, vol 7, pp 1-10, 2000
• [36]. H. Karimia, A. Esfahanimehrb, M. Moslehb, F. Mohammadian, j. ghadamc, S. Salehpourc, O. Medhatia, "Persian handwritten digit recognition using ensemble classifiers", The International Conference on Advanced Wireless, Information, and Communication Technologies, Procedia Computer Science, vol 73, 416 – 425, 2015.
• [37]. By S M Shamim, M. Badrul Alam Miah, A.Sarker, M.Rana, A. Al Jobair, "Handwritten Digit Recognition using Machine Learning Algorithms,Global", Journal of Computer Science and Technology: D Neural & Artificial Intelligence, vol 18, 2018.
• [38]. A. Ashiquzzaman, A. K. Tushar, "Handwritten arabic numeral recognition using deep learning neural networks", IEEE 978-1-5090-6004-7/17/$31.00, 2017.
• [39]. V. Moradi, F. Razzazi, A. Behrad, "Recognition of handwritten Persian two-digit numerals using a novel hybrid SVM/HMM algorithm", Majlesi Journal of Electrical Engineering and Informatics, Vol. 10, No. 3, September 2016.
• [40]. W. chemielnicki, K. stapor, Investigation of normalization techniques and their impact on a recognition rate in handwritten numeral recognition, schedae informaticae vol 19,2010.
• [41]. R. Vijaya Kumar Reddy, U. Ravi Babu*Efficient Handwritten Digit Classification using User-defined Classification Algorithm, international journal on advanced science engineering information technology, Vol.8, No. 3, ISSN: 2088-5334, 2018.

• [42]. A. K. Taqi, A. E. Korial Comparison between Feature Based and Deep Learning Recognition Systems for Handwriting Arabic Numbers, Computer Engineering and Intelligent Systems Vol.9, No.4, 2018.
• [43]. M. Z. Alom, P. Sidike, T. M. Taha, V. K. Asar Handwritten Bangla Digit, arXiv:1705.02680v1 [cs.CV] 7 May 2017.
• [44]. S. Thomas, "A study of representation learning for handwritten numeral recognition of multilingual data set", 2018.
• [45]. Singh P.K., Supratim Das, Ram Sarkar, Mita Nasipuri, 'Recognition of handwritten indic script numerals using mojette transform", In: Mandal J., Satapathy S., Sanyal M., Bhateja V., Proceedings of the First International Conference on Intelligent Computing and Communication. Advances in Intelligent Systems and Computing, vol 458, Springer, Singapore, 2017.
• [46]. N. A. Arbain, M. S. Azmi1, A. K. Muda1, N. A. Muda1, A. R. Radzid1. "Offline handwritten digit recognition using triangle geometry properties", International Journal of Computer Information Systems and Industrial Management Applications, vol 10pp 087- 097, 2018.
• [47]. S. Khorashadizadeh, A. Latif, "Arabic/Farsi handwritten digit recognition using histogram of oriented gradient and chain code histogram", The International Arab Journal of Information Technology, Vol. 13, No. 4, July 2016.
• [48]. P. K. Singh, R.Sarkar, and M. Nasipuri, "A study of moment based features on handwritten digit recognition", Applied Computational Intelligence and Soft Computing, Vol 2016, Article ID 2796863, 17 pages http://dx.doi.org/10.1155/2016/2796863.
• [49]. Prasad, B.K. & Sanyal, G., "Novel features and a cascaded classifier based Arabic numerals recognition system", Multidim Syst Sign Process, vol 29, pp 321–33, 2018.
• [50]. M. Ramzan, H. U. Khan, S. M. Awan, W. Akhtar, M. Ilyas, A.Mahmood , A. Zamir, "A survey on using neural network based algorithms for hand written digit recognition", International Journal of Advanced Computer Science and Applications, Vol. 9, No. 9, 2018.
• [51]. Ø. Due Trier, A. K. Jain and T. Taxt, "Feature extraction methods for character recognition a survey," Pattern Recognition, vol. 29, no. 4, pp. 641-662, 1996.
• [52]. M. K. Hu, "Visuel pattern recognition by moment invariants", IRE Transactions On Information Theory, pp 179-187, 1962.
• [53]. N. Dalal, and B. Triggs. "Histograms of oriented gradients for human detection", IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 1, pp. 886–893, June 2005.
• [54]. F. M. T. R. Kinasih, C. F. D. Saragih, C. Machbub, P. H. Rusmin, L. Yulianti and D. Andriana, " State Machine Implementation for Human Object Tracking using Combination of MobileNet, KCF Tracker, and HOG Features". International Journal on Electrical Engineering and Informatics – Vol. 11, No. 4. 697-712, 2019.
• [55]. R. Ebrahinpour, M. Amini, and F. Sharifizadehi. "Farsi Handwritten Recognition Using Combining Neural Networks Based on Stacked Generalization", International Journal on Electrical Engineering and Informatics – Vol. 3, No. 2. 146-164, 2011.

Khedidja Derdour was born in Batna, Algeria, in 1977. She received her B.Sc degree in computer science from Batna University in 2001 and her M.Sc degree in Industrial Engineering from the same university in 2010. The doctorat degree also in Industrial Engineering in 2021. She is currently a Professor in the Department of Electronic at M'sila university. Her research interests include machine learning, classification, pattern recognition and image processing. She is interested in improving the classification accuracy and developing new features.

Leila Hayet Mouss was born in Batna, Algeria, in 1954. She received the B.Sc. degree in Electrical Engineering, in 1979, from the National Polytechnic School of Algiers, Algeria; the M.Sc. degree in Electrical and Computer Engineering, in 1982, from the ENSERB, France; and finally the Ph.D. degree also in Electrical and Computer Engineering, in 1985, Bordeaux University, France. After graduation, she joined the University of Batna, Algeria, where she is an Associate Professor of Electrical and Computer Engineering. Pr. Mouss is a

member of New York Science Academy. She is the head of Automatic and Computer Integrated Manufacturing Laboratory. Pr. Mouss current research interests include industrial Diagnosis of production system using the artificial intelligence techniques in the LAP Lab (Laboratory of Automation and Manufacturing) at Batna, Algeria.

Rafik Bensaadi received the B.Sc. degree in Electronics and Control Systems and the M.Sc degree in Industrial Engineering, from the University of Batna, Batna, Algeria, in 1999 and 2006, respectively. He is currently working toward the Ph.D. degree at the University of Batna2, Algeria. His research interests include machine learning, control theory, microcontroller-based digital control design, DSP design, and hardware implementation of fault detection, diagnosis and decision-making scheme.