Biased support vector machine and weighted-SMOTE in handling class imbalance problem

Class imbalance problem occurs when classification results shows a class with much larger instances than the others. This raises an issue of majority versus minority [1]. Class imbalance can emerge if the size of dataset is considerably immense with high imbalance ratio [2]. Most of the traditional machine learning techniques deal with reasonably balanced class distributions in solving classification problems [3,4]. Class imbalance is major problem in machine learning, can affect prediction accuracy. Machine learning will provide more accurate predicted results for classes with larger number of instances, and the resulting accuracy will be lowered in the class with small number of instances [5]. This problem also increases the number of misclassification in machine learning [6]. In general, there are three approaches for solving class imbalance problems. First approach is Data-Level Solutions, second is Algorithm-Level Solutions, and last Cost-Sensitive Solutions [7]. The problem of class imbalance can be divided into two scenarios: binary and multi-class [8].


Introduction
Class imbalance problem occurs when classification results shows a class with much larger instances than the others.This raises an issue of majority versus minority [1].Class imbalance can emerge if the size of dataset is considerably immense with high imbalance ratio [2].Most of the traditional machine learning techniques deal with reasonably balanced class distributions in solving classification problems [3,4].Class imbalance is major problem in machine learning, can affect prediction accuracy.Machine learning will provide more accurate predicted results for classes with larger number of instances, and the resulting accuracy will be lowered in the class with small number of instances [5].This problem also increases the number of misclassification in machine learning [6].In general, there are three approaches for solving class imbalance problems.First approach is Data-Level Solutions, second is Algorithm-Level Solutions, and last Cost-Sensitive Solutions [7].The problem of class imbalance can be divided into two scenarios: binary and multi-class [8].
The representativeness of the sample problem is important due to three things: it must have a reduced size compared to the original source, contain the main information in the source, and low redundancy [9].SVM is robust and precise, but can be sensitive to missing values and difficult to train for larger and highly imbalanced datasets.To train large-scale data, it must be combined with another approach in Data Level-Solutions [10].A widely used method is Synthetic Minority Over-sampling Technique (SMOTE) method [11].SMOTE will irregularly initiate synthetic minority object to stabilize number of instances in the minority and majority classes [12].To control the sensitivity in Support Vector Machine, Veropoulos et al. [13] introduced Biased Support Vector Machine.Biased Support Vector Machine combined with SMOTE by Jian et al. [14] to handle class imbalance problem.

A R T I C L E I N F O A B S T R A C T
The SMOTE method will generally initialize a sample number of minorities and after that it will generate equal numbers of synthetic data, but this method should consider all the remaining minority data samples [15].A Weighted-SMOTE proposed by Prusty et al. [15] will oversampling each minority data sample and carried out based on the weight assigned to it.These weights are determined by using the Euclidean distance of a particular minority data sample with consideration to all remaining minority data samples.This paper introduced the implementation of Biased Support Vector Machine and Weighted-SMOTE in Handling Class Imbalance Problem.Biased Support Vector Machine will give better sensitivity control compare to Support Vector Machine.Biased Support Vector Machine works by providing different cost functions for positive samples (minority class) and negative samples (majority class) will group both positive samples and negative samples into Support Vector (SV) Sets and Non Support Vector (NSV) Sets.
Biased Support Vector Machine has been combined with SMOTEBoost in handling class imbalance problem to get the better accuracy.SMOTEBoost should consider all the minority data samples and for this reason, Weighted-SMOTE has been introduced.NSV Sets from negative samples and SV Sets from Positive Samples will then undergo a Weighted-SMOTE process.The traditional performance evaluation in imbalanced evaluation are accuracy and sensitivity [5].
On condition high imbalance ratio of 99%, which minority (positive) classes have only 1% instances as well as error rates may be reduced by learning algorithm through classifying all instances into majority class, until the result give a 1% error rate.Nonetheless, in data mining work, minority (positive) class usually a focus [7] and in case of class imbalance, minority classes frequently grow into the more important class [16].To be able to recognize unusual behavior differs from the general access patterns, the focus would be placed on the positive class and Biased Support Vector Machine (BSVM) became popular choice to solve the problem.The purpose of this research is to get a better sensitivity and accuracy using the implementation of BSVM and Weighted-SMOTE in handling class imbalance problem.

Biased Support Vector Machine and Weighted-SMOTE Algorithm
Traditional Support Vector Machine (SVM) cannot handling class imbalance problem with significant bias in data distribution.It is because the separation hyperplane learned by the SVM very close to the minority class which explains why SVM has a degrading performance on highly imbalanced datasets [17].A Geometric Mean SVM (GSVM) proposed by Gonzalez-Abril et al. [18] will get the ideal hyperplane by calculating the moving of the original bias in the SVM to improve the geometric mean between the true positive rate and the true negative rate.However, in some problem, focus of interest was given to the positive class since it may contain unusual behavior differs from the general access pattern and to deal with datasets where one class, the positive class, is considered more relevant than another class in binary classification problems, the Biased Support Vector Machine (BSVM) method [13] can be considered.BSVM designed for cases when it is non-critical to increase the true positive ratio in exchange for an increase in the false positive rate [19].BSVM achieve better performance in sensitivity, but have a significant reduction in accuracy [19].To overcome this issue, integration with Weighted-Smote Algorithm can be made.Biased Support Vector Machine combined with SMOTEBoost to handle class imbalance problem [14] but this method should consider all remaining minority data samples [15] and this is the reason why Weighted-SMOTE has an advantage to use in this research.
In Biased Support Vector Machine, minority samples are given with a larger cost function and the mathematical model of Biased Support Vector Machine is defined by (1) [13].
Vol. 4 Where w and b represent the normal vector and the intercept of hyperplane, respectively   ≥ 0 are slack variable.The detailed procedure of Biased Support Vector Machine and Weighted-SMOTE Algorithm in handling class imbalance is as follows.
Step 1, Take some data from minority class and majority class.Some data from the minority class are expressed as positive samples and some data from the majority class is expressed as negative samples.
Step 2, Determine the hyperplane combination of positive samples and negative samples.Where the combined value of positive samples is 1 and the combined value of negative samples is -1.
Step 3, Minimise the margin value of positive samples and negative samples to obtain Hyperplane using Biased Support Vector Machine which is done using equation ( 2) and (3) [14].
If the sign function gives a result greater than 0 then it is included in the minority class and if the sign function gives a result smaller than 0 then it is included in the majority class.Conduct training for minority class and majority class based on hyperplane equation obtained.The training process can be done using (4) Step 5, Obtain positive result for minority class and negative result for majority class.If the training result for an instance in the minority class gives a negative result then move the instance into the majority class and if the training result for an instance in the majority class gives a positive result then move that instance into the minority class.
Step 6, Take some data back from the minority class and make it as positive samples and retrieve some data back from majority class and make it as negative samples.Determine the hyperplane of each positive samples and negative samples using ( 2) and (3).
Step 7, Perform training process on minority class using (5).For the process of determining SV Sets and NSV Sets by using hyperplane from positive samples.If the calculation result gives  1 then it is categorized into SV Sets and otherwise categorized as NSV Sets.
  1 =      (. +  = 1)   Step 8, Do the training process on majority class using (6).For the process of determining SV Sets and NSV Sets by using hyperplane of negative samples.If the calculation result gives  -1 then it is categorized into SV Sets and otherwise categorized as NSV Sets.Step 9, SV Eliminate noise using Sets on the Minority Class and then process the Weighted-SMOTE to be combined with NSV Sets on the minority class to become a new minority class.The Weighted-SMOTE process is as follows [15]: (1) Calculate Euclidean Distance for each of T minority data samples using (7) [20].
(2) Normalized ED matrix using maximum of the ED (  ) and minimum of ED (  ) then called it as normalized ED (NED) that can be seen in (9).
(4) Calculated Weight Matrix for each minority of T samples considering the sum of RNED Matrix using (11).
(5) Using this Weight matrix to find the SMOTE generation matrix using (12).
Step 10, NSV Sets on Majority Class go through Weighted-SMOTE process and combined with SV Sets on Majority Class to become a new majority class.The Weighted-SMOTE process is similar with Step 9.

Research Design
The experiments conducted using Iris Dataset and Balanced Scale Weight & Distance Database using software deployed in R Language.Research Method can be seen in Fig. 1.This research calculates the Accuracy and Sensitivity from the result of implementation of Biased Support Vector Machine and Weighted-SMOTE using (13) [15] and ( 14) [13].Confusion Matrix [13] is used for determining the value of TN, TP, FN, and FP which can be seen in Table 1.

Positive Samples
True Positive (TP) False Negative (FN)

Negative Samples
False Positive (FP) True Negative (TN) majority and minority classes.In general, class imbalance issues will arise if there is misclassification of the placement of instances in the minority class to be placed in the majority class.
Result of testing using Balanced Scale Weight & Distance Database can be seen in Table 3.Similar results were obtained in experiments using the Balance Scale Weight & Distance Database which can be seen in Table 3.The results show that the Biased Support Vector Machine and Weighted-SMOTE can also correct the difference in the number of majority and minority that can be seen on the results before and after processing.This confirms that an increase in classification accuracy can provide an advantage in overcoming class imbalance problems.From Table 2 and Table 3, there is an improvement in the differentiation on number of majority and number of minority before and after processing with Biased Support Vector Machine and Weighted-SMOTE.The accuracy and sensitivity is satisfying.The high value of accuracy and sensitivity means that the implementation of Biased Support Vector Machine and Weighted-SMOTE can handle the class imbalance problem.The results also show that if focus will be given to positive samples or minority classes then Biased Support Vector Machine and Weighted-SMOTE can be considered.

Conclusion
The study implemented Biased Support Vector Machine and Weighted-SMOTE in handling class imbalance problem.Experiments of two datasets showed the satisfactory result that indicated in value of the accuracy and sensitivity with the average accuracy was 0.87 and average sensitivity was 0.86.From the study, it can be conclude that implementation of BSV and Weighted-SMOTE can handle class imbalance problem with the positive class in a higher priority.The proposed methods give the satisfactory result in accuracy and sensitivity.

Table 3 .
Result of Testing using Balanced Scale Weight & Distance Database