A Genetic Algorithm To Ensemble Feature Selection.pdf

A ic Algorithm to Ensemble Feature Selection











Ramon Armengol Garganté
Tutor: Elena Marchiori
Second reader: Wojtek Kowalczyk

Vrije Universiteit, Amsterdam
Faculty puter Sciences
May 2007
Index:

1 Introduction ………………………………………... 3

2 An ensemble of classifiers ………………………… 3
Ensemble feature selection ……………………. 4

3 GA for Sequential Ensemble Feature Selection ..... 5
Measures used in the fitness function ................ 8
Diversity: The fail/non-fail disagreement 8
measure.
Number of features .................................. 9
Fitness function .................................................. 10

4 Integration methods .................................................. 12
Simple Voting ………………………………… 12
Weighted Voting ……………………………… 12
Staking ………………………………………… 13
Dynamic Voting with Selection ………………. 14

5 Experimental Investigations .................................... 15
Experimental Settings …………………………... 15
Experimental results ……………………………. 16

6 Conclusions ................................................................ 28

7 Acknowledgements ………………………………... 30

8 Bibliography .............................................................. 31

Annex .............................................................................. 33

2
1. Introduction

It has been shown in several studies that an ensemble of diverse classifiers is
generally more accurate than a single model. One way to obtain an ensemble of
classifiers is selecting different feature subsets from the original dataset and creating for
each subset a base classifier. This approach is known as an ensemble feature selection.
Nowadays we may use large dataset where each instance can have more than
2000 features. To find the attributes to be selected in order to get the best subsets
collection we should try mor