Wavelet based approach for facial expression recognition

In everyday life, people get in touch and communicate with others using verbal and non-verbal ways. One of non-verbal communications is emotions which are usually expressed through facial expressions. A facial expression is result of one or many movements or positions of facial muscles. By recognizing facial expressions, the emotion of people can be predicted. For instance, a smile expresses hospitality and affection, a lift of eyebrows shows confusion, a wince of forehead portrays fear and anxiety. According to Ekman [1], humans have universal facial expressions and the expressions categorized into six classes these are happiness, sadness, disgust, anger, surprise, and fear.


I. Introduction
In everyday life, people get in touch and communicate with others using verbal and non-verbal ways.One of non-verbal communications is emotions which are usually expressed through facial expressions.A facial expression is result of one or many movements or positions of facial muscles.By recognizing facial expressions, the emotion of people can be predicted.For instance, a smile expresses hospitality and affection, a lift of eyebrows shows confusion, a wince of forehead portrays fear and anxiety.According to Ekman [1], humans have universal facial expressions and the expressions categorized into six classes these are happiness, sadness, disgust, anger, surprise, and fear.
In recent years, researchers have been interested in studying facial expression recognition and over time, biometrics is widely used to recognize facial expression.It is applied using distinguishing traits.In general, these traits are divided into two, namely physical and behavioral characteristics.A facial expression is one of behavioural characteristics.The use of these characteristics allows us to recognize emotions of people.
Several methods for facial expression recognition have been developed and implemented.Long et al. [1] examined facial expression recognition using Pseudo Zernike Moment Invariant (PZMI) as a feature extraction from the global information of images and the Radial Basis Function (RBF) network was employed to be a classifier.In addition, Bashyal and Venayagamoorthy [2] applied Gabor wavelet and learning vector quantization (LVQ) in their study.Kulkarni [3] developed an applied smart system to recognize emotions using committee neural network.Meanwhile, Deng et al. [4] performed a research in comparing local Gabor filter bank using the approach of principal component analysis (PCA) and linear discriminant analysis (LDA).Furthermore, Ma and Khorasani [5] carried out facial expression recognition study using 2-D discrete cosine transform (DCT) and feedforward neural networks.Moreover, Abidin and Harjoko [6] analyzed facial expression recognition system using fisherface and BPNN approach.Most of facial expression recognition studies applied neural networks that were combined by different kind of methods.The key factors of neural network are able to conduct learning and Zaenal Abidin and Alamsyah (Wavelet based approach for facial expression recognition) generalization, non-linear mapping as well as parallel computation.However, neural networks sometimes trapped into local minima that lead to slow convergence level.According to Abiyev and Kaynak [7], to cope with neural network weaknesses, wavelet function applied in neural networks structure becomes an appropriate approach.
In this study, we developed a facial expression system using BPNN and wavelet based feature extraction which was expected to achieve similar or even better performance of facial expression recognition.In particular, the aims of this study are investigating the appropriate wavelet transform that can be used as feature extraction and analyzing the performance of the systems.

A. The Need for System Analysis
Six basic facial expressions such as happiness, sadness, anger, surprise, fear, and disgust, plus one neutral expression will be investigated to measure performance of proposed method.This study was performed using JAFFE database [21].It comprises 213 facial expression images from 10 Japanese women.Every subject posed three or four times to get six basic facial expressions and one neutral.The resolution of the image is 256×256 in *.tiff format with grayscale color mode.Images used for this study were split into two groups of training and testing data sets with 140 images and 73 images, respectively.The splitting image data follow the previous research performed by Abidin and Harjoko [6].The training data set was used for learning process meanwhile the testing data set was used for analyzing system performance.
Each image is represented in a matrix in which each element of the matrix corresponds to the intensity level of a pixel.The representation matrix of image usually has a big dimension which can lead to the process is computationally expensive.Therefore, reduction dimension is necessary.In this study, wavelet functions will be used to reduce the dimension of images.
The following steps describe the process of analyzing facial expression system using the proposed method: 1) Image pre-processing and feature extraction.
The step of image pre-processing is started by automatically face detection.There are several methods available for performing face detection.In this study, the integral projection method proposed by Abidin and Harjoko [6] is used.Once a face is detected, image pre-processing then can be employed.Firstly, the size of images will be normalized into 130×114 pixel.2) Learning process of neural networks for images in training data set.
In this step, all images in training data set were trained using a classifier that is BPNN.This neural network architectural design comprises fully weighted interconnected rows of nodes as processing units.Some nodes are organized into layers [6], [22].A typical neural network consist of an input layer, an output layer, and one or more hidden layers [3].The most popular learning algorithm for multi-layer perceptron is backpropagation.The backpropagation algorithm defines a systematic way to update the synaptic weights of multi-layer perceptron (MLP) networks.
The learning of backpropagation algorithm is performed in two phases: forward phase and backward phase [23].During the first phase, the free parameters of the network are determined, and then the input signal is propagated through the network layer by layer.In this phase, an error signal is calculated by ( 1): Where d ୧ is the desired response and y ୧ is the actual output produced by the network in response to the input x ୧ .
During this backward phase, the error signal e ୧ is propagated through the network in the backward direction.It is during this phase that adjustments are applied to the free parameters of the network in order to minimize the error e ୧ in a statistical sense.
In this study, backpropagation learning is implemented using pattern mode in which "weight updating is performed after the presentation of each training pattern" [24, p. 46].The outcome of learning process is stored to database and is used for testing analysis of the facial expression recognition system.
3) Testing the facial expression recognition system both images in training and testing data sets.
The last stage of evaluating the facial expression recognition system is analyzing the performance of the system by measuring the recognition rate.

B. System Design
In this sub section describes design of the facial expression recognition system.To investigate the performance of proposed method, an interface of facial expression recognition system has been developed using MATLAB.The interface consists of three subsystems, which are:

1) Subsystem image registration
This subsystem comprises some features, i.e. the storing tool, face detector, and feature extractor.Storing tool is used to record keeping images into the system.When an image is registered into the system, the face detector will capture the face region, and do image preprocessing.Integral projection method was employed to accomplish face detection.An integral projection is obtained through the sum of given set of pixels along a given direction (for details, see [6]).After image pre-processing, the second important task is feature extraction.The features are obtained by decomposing the images with wavelet transforms.In this study, Haar wavelet, Daubechies (4) wavelet, and Coiflet (1) wavelet are employed.Then the features are stored in a database.

2) Subsystem of neural network learning
Learning algorithm is built in this subsystem.BPNN's structure consists of three layers, namely input layer, hidden layer and output layer.The activation function for hidden layer is sigmoid bipolar while the activation function for output layer is sigmoid, hence the outcome of output layer is expected to be in the range [0, 1].The weights of BPNN learning result then stored to database and will be used to recognition processes.

3) Subsystem of facial expression recognition
This subsystem further classifies the facial expression into different categories based on the extracted data fed to it from a facial image.This subsystem can be seen in Fig. 3.

III. Experimental Results
In this study three experiments were conducted as trying to verify the effect of wavelet transforms on the performance of facial expression recognition system.The learning process was conducted by varying on the neural network parameters.The number of neurons in hidden layer was set by 5 while the learning rate was set by 0.25.Moreover, the minimum error was set by 0.001 and the maximum number of epoch was 1000.Facial expression recognition system has been tested by using 213 images from JAFFE database.The number of 140 images has been trained and the remaining 73 images were used for testing.Since the number of images in the JAFFE database is limited, we had performed the trial over 5 times to get the average classification rate.Table 1 shows the average of recognition rate of wavelet-based facial expression with the trial over 5 times.From Table 1 it can be seen that the recognition rate of data set with Coiflet (1) wavelet transform is lower than two others, however there is no significant difference in terms of recognition rate among wavelet transforms.In detail, the recognition rate of seven-class facial expressions for each Haar wavelet, Daubechies (4) wavelet and Coiflet (1) wavelet is depicted in Fig. 4, Fig. 5 and Fig. 6 respectively.The recognition rates for each facial expression from data set with Haar wavelet are over 90%.Similarly, the recognition rates from data set with Daubechies (4) wavelet and Coiflet (1) wavelet show good performances.
From three figures (Fig. 4, Fig. 5 and Fig. 6), it can be seen that fear expression is always in the lowest rate.Meanwhile, in Table 2, Table 3 and Table 4 show that some fear expression data are dominantly recognized as surprise expression.Similarly, sadness expression is dominantly identified as disgust and anger expression.Since facial expressions are combination of facial muscles movements, these may lead ambiguity [25].Although some expressions may be ambiguous, six expressions, namely anger, disgust, happiness, neutral, sadness, and surprise were well classified.The recognition rate of those expressions was that over 90% meanwhile fear expression was around 89.38% to 91.88%.Convolution matrices of Table 2, Table 3 and Table 4 give details information about facial expression recognition rate for each wavelet transform.
Overall, there are no significantly differences among those outcomes.In other words, Haar wavelet, Daubechies (4) wavelet or Coiflet (1) wavelet give a similar influence in classifying facial expression into seven group classes.
We have compared our proposed with the existing method which is method that uses the same data for training and testing, face detection method and also the same pre-processing of images.From Table 5 it can be seen that the proposed method shows better performance than method using fisherface + BPNN.

IV. Conclusions and Future Works
In this paper, a wavelet based facial expression recognition system is proposed.Three wavelet transforms namely Haar wavelet, Daubechies (4) wavelet, and Coiflet (1) wavelet were selected to perform feature extractions.Moreover, BPNN was chosen as a classifier.The proposed method has shown good performances in recognizing facial expression images of JAFFE database.However, the task of facial expression recognition is still challenging since the appearance of expressions can vary by person.Ambiguity of expression may occur and become hard to classify.
In future work the ambiguity of expression can be considered to improve better performance of facial expression recognition rate.It is also important to select different wavelet function to be employed in developing neural networks based facial expression recognition system as comparative study.

Fig. 1 .
Fig. 1.Illustration of 3-Level 2D Decomposition Secondly, image contrast is enlarged by applying histogram equalization and finally masking process is undertaken by covering each angle of image in order to decrease the variations appearing on them.After image pre-processing done, the next stage is feature extraction.In this study, three wavelet functions are used to extract the features, i.e.Haar wavelet, Daubechies (4) wavelet, and Coiflet (1) wavelet.The images are performed 3-level 2D decomposition and the dimension of images is changed into 17×15 pixel.

Table 1 .
Recognition Rate for Each Wavelet Transform

Table 2 .
Confusion Matrix of All Data Sets with Haar Wavelet (%)

Table 5 .
Comparative Results of the Classification Rate of Different Approaches