Random Projection Optimal Trees Ensemble

Ensemble classifiers, formed by the combination of multiple weak learners, have been shown to outperform ordinary classification methods in that the former decrease bias, variance and/or improve predictions. These classifiers, however, can still result in low prediction performance when used with the wrong choice of their hyper-parameters values and/or when there are noisy features in the data. Thus, feature selection and fine tuning hyper-parameter could improve predictive accuracy of ensemble classifiers. This thesis first investigates the effect of feature selection on three methods: Random Forest (RF), Optimal Trees Ensemble (OTE) and Random Projection Ensembles (RP) in high dimensional settings. To this end, LASSO has been considered for selecting the most important features based on training data for dimension reduction. Additionally, the influence of various hyper-parameters regulating the three methods has also been assessed. Secondly, this thesis proposes a novel idea to use random projection method in conjunction with optimal tree selection to get an improved trees ensemble. This is achieved by randomly projecting the training data into lower dimension and classification trees are grown on bootstrap samples taken from the newly projected datasets. The best performing trees are selected based on out-of-bag error rate and combined to get the final Optimal Random Projection Trees Ensemble (ORPTE). The results of ORPTE are compared with those of Tree, RF, OTE, RP, k -NN, XGBoost and SVM. Analysis on several benchmark datasets is given to illustrate the effect of feature selection and hyper-parameter tuning on the methods and the efficiency of the proposed method. The results reveal that feature selection improves the predictive performance of the RP method in addition to reducing the computational burden on benchmark and example datasets. The performance of OTE and RF is less influenced by feature selection. Moreover, ORPTE has outperformed in terms of prediction accuracy in majority of the cases.