Support vector machine ensemble based on feature and hyperparameter variation

Abstract

The support vector machine (SVM) classifier is currently considered one of the most powerful pattern recognition based techniques for solving binary classification problems. To further increase the accuracy of an individual SVM, a well-established approach relies on using a SVM ensemble, which is a set of accurate, divergent SVMs. In this work we investigate composing an ensemble with SVMs that differ among themselves on the feature subset and also the hyperparameter value they use. We propose a three-stage method for building an SVM ensemble. First we use complementary Genetic Ensemble Feature Selection (GEFS) searches to globally investigate the feature space, aiming to produce a set of diverse feature subsets. Further, for each produced feature subset we build a SVM with tuned hyperparameters. Finally, we employ a local search to retain an optimized, reduced set of these SVMs to ultimately comprise the ensemble. Our experiments were performed in a context of real-world industrial machine fault diagnosis. We use 2000 examples of vibration signals obtained from motor pumps installed on oil platforms. The performed experiments show that the proposed SVM ensemble method achieved superior results in comparison to other well-established classification approaches.