Acta Oeconomica Pragensia 2008, 16(4):37-55 | DOI: 10.18267/j.aop.131

Identifying the most Informative Variables for Decision-Making Problems - a Survey of Recent Approaches and Accompanying Problems

Pavel Pudil, Petr Somol
Prof. Ing. Pavel Pudil, Dr.Sc.; Fakulta managementu, Vysoká škola ekonomická v Praze, ÚTIA Akademie věd ČR, pudil@fm.vse.cz.
RNDr. Petr Somol, Ph.D.; ÚTIA Akademie věd ČR, Fakulta managementu, Vysoká škola ekonomická v Praze, somol@utia.cas.cz.

We provide an overview of problems related to variable selection (also known as feature selection) techniques in decision-making problems based on machine learning with a particular emphasis on recent knowledge. Several popular methods are reviewed and assigned to a taxonomical context. Issues related to the generalization-versus-performance trade-off, inherent in currently used variable selection approaches, are addressed and illustrated on real-world examples.

Keywords: variable selection, feature selection, machine learning, decision rules, classification
JEL classification: C60, C80

Published: August 1, 2008  Show citation

ACS AIP APA ASA Harvard Chicago Chicago Notes IEEE ISO690 MLA NLM Turabian Vancouver
Pudil, P., & Somol, P. (2008). Identifying the most Informative Variables for Decision-Making Problems - a Survey of Recent Approaches and Accompanying Problems. Acta Oeconomica Pragensia, 16(4), 37-55. doi: 10.18267/j.aop.131
Download citation
  • BibTeX (.bib)
  • Bookends (.ris)
  • EasyBib (.ris)
  • EndNote (.enw)
  • EndNote 8 (.xml)
  • ISI WoS (.isi)
  • Medlars (.medlars)
  • Mendeley (.ris)
  • MODS (.xml)
  • Papers (.ris)
  • RefWorks (.txt)
  • RefManager (.ris)
  • RIS (.ris)
  • MS Word (.xml)
  • Zotero (.ris)
    • Open full article

    References

    1. ALEXE G.; ALEXE S.; HAMMER P. L.; VIZVARI, B. 2006. Pattern-basedfeature selection in genomics andproteomics. Annals of Operations Research, 2006, vol. 148, no. 1, pp. 189-201. Go to original source...
    2. BLUM A.; LANGLEY P. 1997. Selection ofRelevant Features and Examples in Machine Learning. Artificial Intelligence, 97 (1-2), 245-271, 1997. Go to original source...
    3. CHANG Ch.-Ch.; LIN Ch.-J. 2001. LIBSVM: a library for support vector machines. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm, 2001.
    4. DEVIJVER P. A.; KITTLER J. 1982. Pattern Recognition: A StatisticalApproach. Prentice-Hall, 1982.
    5. DUDA, R.O.; HART, P. E.; STORK, D. G. 2000. Pattern Classification. 2nd edition. Wiley-Inter-science, 2000.
    6. FERRI, F. J.; PUDIL, P.; HATEF, M.; KITTLER, J. 1994. Comparative Study of Techniques for Large-Scale Feature Selection. Gelsema, E. S.
    7. Kanal, L. N. (eds.) Pattern Recognition in Practice IV. Elsevier Science B. V., 1994, 403-413.
    8. FORMAN, G. 2003. An extensive empirical study of feature selection metrics for text classification. J. Mach, Learn. Res., 2003, vol. 3, pp. 1289-1305.
    9. FUKUNAGA, K. 1990. Introduction to Statistical Pattern Recognition. Academic Press, 1990. Go to original source...
    10. GUYON, I.; ELISSEEFF, A. 2003. An introduction to variable and feature selection. Journal of Machine Learning Research. 2003, 3, 1157-1182.
    11. HUSSEIN F.; WARD R.; KHARMA, N. 2001. Genetic algorithms forfeature selection and weighting, a review and study. Icdar, 2001, vol. 00, p. 1240.
    12. JAIN, A. K.; ZONGKER, D. 1997. Feature Selection: Evaluation, Application and Small Sample Performance. IEEE Transactions on PAMI 1997, 19 (2), 153-158. Go to original source...
    13. JAIN, A. K.; DUIN, R. P. W.; MAO, J. 2000. Statistical Pattern Recognition: A Review. IEEE Transactions on Pattern Analysis and Machine Intelligence 2000, 22 (1), 4-37. Go to original source...
    14. KOHAVI R.; JOHN G. H. 1997. Wrappers for Feature Subset Selection. Artificial Intelligence. 1997, 97 (1-2). : 273-324. Go to original source...
    15. KONONENKO, I. 1994. Estimating attributes: Analysis and extensions of Relief. In ECML-94: Proc. European Conf. on Machine Learning. Secaucus, NJ. USA : New York: Springer-Verlag Inc., 1994, pp. 171-182. Go to original source...
    16. KUDO, M.; SKLANSKY, J. 2000. Comparison of Algorithms that Select Features for Pattern Classifiers. Pattern Recognition. 2000, 33 (1), 25-41. Go to original source...
    17. LIU, H.; YU, L. 2005. Toward Integrating Feature Selection Algorithms for Classification and Clustering. IEEE Transactions on Knowledge and Data Engineering. 2005, 17 (4), 491-502. Go to original source...
    18. LOUGHREY, J.; CUNNINGHAM, P. 2004. Overfitting in Wrapper-Based Feature Subset Selection: The Harder You Try the Worse it Gets. 24th SGAI International Conference on Innovative Techniques and Applications of Artificial Intelligence, (AI-2004) Bramer, M., Coenen, F., T. Allen, 33-43, Springer, 2004. Go to original source...
    19. MARILL, T.; GREEN, D. 1963. On the effectiveness of receptors in recognition systems. IEEE Trans. on Information Theory, 1963, 9 (1), 11-17. Go to original source...
    20. MAYER, H. A.; SOMOL, P.; HUBER, R.; PUDIL, P.; 2000. Improving Statistical Measures of Feature Subsets by Conventional and Evolutionary Approaches. Proc. 3rd IAPR International Workshop on Statistical Techniques in Pattern Recognition (Alicante), 2000, 77-81. Go to original source...
    21. McLACHLAN, G. J. 1992. Discriminant Analysis and Statistical Pattern Recognition. New York : John Wiley & Sons, 1992. Go to original source...
    22. MURPHY, P. M.; AHA, D. W. 1994. UCI Repository of Machine Learning Databases [http://archive.ics.uci.edu/ml]. University of California, Department of Information and Computer Science, Irvine, CA, 1994.
    23. NARENDRA, P. M.; FUKUNAGA, K. 1977. A Branch and Bound Algorithm for Feature Subset Selection. IEEE Transactions on Computers, 1977, 26, 917-922. Go to original source...
    24. NOVOVICOVA, J.; PUDIL, P.; KITTLER, J. 1996. Divergence Based Feature Selection for Multimodal Class Densities. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1996, 18 (2), 218-223. Go to original source...
    25. NOVOVICOVA, J.; PUDIL, P. 1997. Feature selection and classification by modified model with latent structure. In Dealing With Complexity: Neural Network Approach, Springer Verlag, 1997, 126-140. Go to original source...
    26. PUDIL, P.; NOVOVICOVA, J.; KITTLER, J. 1994. Floating Search Methods in Feature Selection. Pattern Recognition Letters, 1994, 15 (11), 1119-1125. Go to original source...
    27. PUDIL, P.; NOVOVICOVA, J. 1998. Novel Methods for Subset Selection with Respect to Problem Knowledge. IEEE Transactions on Intelligent Systems; Special Issue on Feature Transformation and Subset Selection, 1998, 66-74. Go to original source...
    28. RAUDYS, S. 2006. Feature Over-Selection. Lecture Notes in Computer Science LNCS 4109, Springer, 2006, 622-631. Go to original source...
    29. RIPLEY, B. 1996. Pattern Recognition and Neural Networks. Cambridge, Massachusetts : Cambridge University Press, 1996. Go to original source...
    30. SALAPPA, A.; DOUMPOS, M.; ZOPOUNIDIS, C. 2007. Feature selection algorithms in classification problems: an experimental evaluation. Optimization Methods and Software, 2007, 22 (1), 199-214. Go to original source...
    31. SIEDLECKI, W.; SKLANSKY, J. 1988. On Automatic Feature Selection. International Journal of Pattern Recognition and Artificial Intelligence, 1988, 2(2):197-220. Go to original source...
    32. SOMOL, P.; BAESENS, B.; PUDIL, P.; VANTHIENEN J. 2005. Filter- versus wrapper-based feature selection for credit scoring. International Journal of Intelligent Systems, 2005, 20 (10), 985-999. Go to original source...
    33. SOMOL, P.; PUDIL, P.; NOVOVICOVA, J.; PACLiK, P. 1999. Adaptive Floating Search Methods in Feature Selection. Pattern Recognition Letters, 1999, 20 (11,12,13), 1157-1163. Go to original source...
    34. SOMOL, P.; PUDIL, P. 2000. Oscillating Search Algorithms For Feature Selection. Proc. 15th IAPR International Conference on Pattern Recognition, Barcelona, Spain, 2000, 406-409. Go to original source...
    35. SOMOL, P.; PUDIL P. 2002. Feature Selection Toolbox. Pattern Recognition, 2002, 35 (12), 2749-2759. Go to original source...
    36. SOMOL P.; PUDIL, P.; KITTLER, J. 2004. Fast Branch & Bound Algorithms for Optimal Feature Selection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004,26 (7), 900-912. Go to original source...
    37. STEARNS, S. D. 1976. On selecting features for pattern classifiers. In Proc. of the 3rd Int. IEEE Joint Conf. on Pattern Recognition, Coronado, CA, USA, 1976, 71-75.
    38. THEODORIDIS, S.; KOUTROUMBAS, K. 2003. Pattern Recognition. 2nd Ed., Academic Press, 2003.
    39. TSAMARDINOS, I.; ALIFERIS, C. 2003. Towards Principled Feature Selection: Relevancy, Filters, and Wrappers. Artificial Intelligence and Statistics, 2003.
    40. VAFAIE, H.; IMAM, I. 1994. Feature Selection Methods: Genetic Algorithms vs. Greedy-like Search. In Proceedings of the International Conference on Fuzzy and Intelligent Control Systems, 1994.
    41. WEBB, A.; 2002. Statistical Pattern Recognition. 2nd Ed., John Wiley &Sons, 2002. Go to original source...
    42. WHITNEY, A. W. 1971. A direct method of nonparametric measurement selection. IEEE Trans. Comput. 1971, 20 (9), 1100-1103. Go to original source...
    43. YANG, J.; HONAVAR, V. 1998. Feature Subset Selection Using a Genetic Algorithm. IEEE Intelligent Systems, 1998, 13, 44-49. Go to original source...

    This is an open access article distributed under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits use, distribution, and reproduction in any medium, provided the original publication is properly cited. No use, distribution or reproduction is permitted which does not comply with these terms.


    Return to the content