Abstract

We introduce a robust classification model designed for feature selection. Support vector machine (SVM) models continue to play a crucial role in binary classification, particularly with tabular data. Their robust variants are essential for developing classifiers that remain stable despite shifts in data distribution. Additionally, sparse classifiers are highly desirable, as they offer improved performance in classification tasks and help reduce overfitting, especially when the number of features exceeds the number of samples. In this context, penalty methods for feature selection are fundamental to the development of sparse optimization models. Despite its inherent nonlinearity and nonconvexity, the ℓ0\ell_0-norm remains a popular choice for this task and has been effectively applied in diverse fields, such as pattern recognition and signal processing. Our objective is to evaluate the performance of two classification approaches that integrate robust classification in SVM-type models with embedded feature selection. We propose a diagonal algorithm for numerically solving these optimization models, which we validate through numerical tests on benchmark datasets. The results are compared to those obtained using models based on ℓ2\ell_2-norm regularization.

Author information

Contact details are reproduced from the original publication and may be historical.

Miguel Carrasco

Facultad de Ingeniería y Ciencias Aplicadas, Universidad de los Andes, Santiago, Chile

macarrasco@miuandes.cl

Benjamin Ivorra

Interdisciplinary Mathematics Institute, Dept. of Applied Mathematics and Mathematical Analysis,
Universidad Complutense, Madrid, Spain

ivorra@ucm.es

Julio López

Facultad de Ingeniería y Ciencias, Universidad Diego Portales, Santiago, Chile

julio.lopez@udp.cl

Angel M. Ramos

Interdisciplinary Mathematics Institute, Dept. of Applied Mathematics and Mathematical Analysis,
Universidad Complutense, Madrid, Spain

angel@mat.ucm.es

Keywords

  • Feature selection
  • robust optimization
  • second-order cone programming
  • support vector machines

Mathematics Subject Classification

90C17, 90C26, 62H30, 65K05, 68T05

References

  1. A. Alizadeh et al.: Distinct types of diffuse large B-cell lymphoma identified by gene expression profiling, Nature 403 (2000) 503–511.https://doi.org/10.1038/35000501Google Scholar
  2. F. Alizadeh, D. Goldfarb: Second-order cone programming, Math. Program. Ser. B 95/1 (2003) 3–51.https://doi.org/10.1007/s10107-002-0339-5Google Scholar
  3. U. Alon, N. Barkai, D. A. Notterman, K. Gish, S. Ybarra, D. Mack, A. J. Levine: Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays, Proc. Natl. Acad. Sci. USA 96/12 (1999) 6745–6750.https://doi.org/10.1073/pnas.96.12.6745Google Scholar
  4. A. Beck, A. Ben-Tal, L. Tetruashvili: A sequential parametric convex approximation method with applications to non-convex truss topology design problems, J. Global Optim. 47/1 (2010) 29–51.https://doi.org/10.1007/s10898-009-9456-5Google Scholar
  5. P. Bradley, O. Mangasarian: Feature selection via concave minimization and support vector machines, in: Proc. 15th Int. Conf. on Machine Learning (ICML ’98), Morgan Kaufmann, San Francisco (1998) 82–90.Google Scholar
  6. A. Canelas, M. Carrasco, J. López: Application of the sequential parametric convex approximation method to the design of robust trusses, J. Global Optim. 68/1 (2017) 169–187.https://doi.org/10.1007/s10898-016-0460-2Google Scholar
  7. M. Carrasco, B. Ivorra, J. López, A. M. Ramos: Embedded feature selection for robust probability learning machines, Pattern Recogn. 159 (2025), art.no. 111157.https://doi.org/10.1016/j.patcog.2024.111157Google Scholar
  8. W. J. Chen, Y. J. Tian: Lp -norm proximal support vector machine and its applications, Procedia Comput. Sci. 1/1 (2010) 2417–2423.https://doi.org/10.1016/j.procs.2010.04.272Google Scholar
  9. A. Christou, A. Artemiou: Adaptive ℓ0 regularization for sparse support vector regression, Mathematics 11/13 (2023), art.no. 2808.https://doi.org/10.3390/math11132808Google Scholar
  10. C. Cortes, V. Vapnik: Support-vector networks, Mach. Learn. 20 (1995) 273–297.https://doi.org/10.1007/BF00994018Google Scholar
  11. M. A. Davenport, M. F. Duarte, Y. C. Eldar, G. Kutyniok: Introduction to compressed sensing, in: Compressed Sensing: Theory and Applications, Y. C. Eldar et al. (eds.), Cambridge Univ. Press, Cambridge (2012) 1–64.https://doi.org/10.1017/CBO9780511794308.002Google Scholar
  12. P. D. Tao, E. B. Souad: Algorithms for solving a class of non-convex optimization problems. Methods of subgradients, in: Fermat Days 85: Mathematics for Optimization, J.-B. Hiriart-Urruty (ed.), North-Holland Mathematics Studies Vol. 129, North-Holland, Amsterdam (1986) 249–271.https://doi.org/10.1016/S0304-0208(08)72402-2Google Scholar
  13. D. L. Donoho, M. Elad, V. N. Temlyakov: Stable recovery of sparse overcomplete representations in the presence of noise, IEEE Trans. Inf. Theory 52/1 (2006) 6–18.https://doi.org/10.1109/TIT.2005.860430Google Scholar
  14. R. O. Duda, P. E. Hart, D. G. Stork: Pattern Classification, Wiley, New York (2001).Google Scholar
  15. E. Gravier et al.: A prognostic DNA signature for T1/T2 node-negative breast cancer patients, Genes Chromosomes Cancer 49/12 (2010) 1125–1134.https://doi.org/10.1002/gcc.20820Google Scholar
  16. I. Guyon, J. Weston, S. Barnhill, V. Vapnik: Gene selection for cancer classification using support vector machines, Mach. Learn. 46 (2002) 389–422.https://doi.org/10.1023/A:1012487302797Google Scholar
  17. K. Huang, H. Yang, I. King, M. Lyu, L. Chan: The minimum error minimax probability machine, J. Mach. Learn. Res. 5 (2004) 1253–1286.Google Scholar
  18. Jayadeva, R. Khemchandani, S. Chandra: Twin support vector machines for pattern classification, IEEE Trans. Pattern Anal. Mach. Intell. 29/5 (2007) 905–910.https://doi.org/10.1109/TPAMI.2007.1068Google Scholar
  19. G. Lanckriet, L. El Ghaoui, C. Bhattacharyya, M. Jordan: A robust minimax approach to classification, J. Mach. Learn. Res. 3 (2003) 555–582.Google Scholar
  20. H. A. Le Thi, T. Pham Dinh, M. Thiao: Efficient approaches for l2 –l0 regularization and applications to feature selection in SVM, Appl. Intell. 45/2 (2016) 549–565.https://doi.org/10.1007/s10489-016-0778-yGoogle Scholar
  21. J. López, S. Maldonado, M. Carrasco: Double regularization methods for robust feature selection and SVM classification via DC programming, Inf. Sci. 429 (2018) 377–389.https://doi.org/10.1016/j.ins.2017.11.035Google Scholar
  22. S. Maldonado, M. Carrasco, J. López: Regularized minimax probability machine, Knowledge-Based Syst. 177 (2019) 127–135.https://doi.org/10.1016/j.knosys.2019.04.016Google Scholar
  23. S. Maldonado, J. López, M. Carrasco: The Cobb–Douglas learning machine, Pattern Recogn. 128 (2022), art.no. 108701.https://doi.org/10.1016/j.patcog.2022.108701Google Scholar
  24. O. L. Mangasarian: Exact 1-norm support vector machines via unconstrained convex differentiable minimization, J. Mach. Learn. Res. 7 (2006) 1517–1530.Google Scholar
  25. J. Neumann, C. Schnörr, G. Steidl: Combined SVM-based feature selection and classification, Mach. Learn. 61/1–3 (2005) 129–150.https://doi.org/10.1007/s10994-005-1505-9Google Scholar
  26. S. L. Pomeroy et al.: Prediction of central nervous system embryonal tumour outcome based on gene expression, Nature 415/6870 (2002) 436–442.https://doi.org/10.1038/415436aGoogle Scholar
  27. F. Rinaldi, F. Schoen, M. Sciandrone: Concave programming for minimizing the zero-norm over polyhedral sets, Comput. Optim. Appl. 46/3 (2010) 467–486.https://doi.org/10.1007/s10589-008-9202-9Google Scholar
  28. Y. H. Shao, W. J. Chen, N. Y. Deng: Nonparallel hyperplane support vector machine for binary classification problems, Inf. Sci. 263 (2014) 22–35.https://doi.org/10.1016/j.ins.2013.11.003Google Scholar
  29. M. A. Shipp et al.: Diffuse large B-cell lymphoma outcome prediction by geneexpression profiling and supervised machine learning, Nat. Med. 8/1 (2002) 68–74.Google Scholar
  30. J. Tan, Z. Zhang, L. Zhen, C. Zhang, N. Deng: Adaptive feature selection via a new version of support vector machine, Neural Comput. Appl. 23 (2013) 937–945.https://doi.org/10.1007/s00521-012-1018-yGoogle Scholar
  31. H. A. Le Thi, T. Pham Dinh, H. M. Le, X. T. Vo: DC approximation approaches for sparse optimization, Eur. J. Oper. Res. 244/1 (2015) 26–46.https://doi.org/10.1016/j.ejor.2014.11.031Google Scholar
  32. R. Tibshirani: Regression shrinkage and selection via the Lasso, J. R. Stat. Soc. B 58/1 (1996) 267–288.https://doi.org/10.1111/j.2517-6161.1996.tb02080.xGoogle Scholar
  33. V. N. Vapnik: Statistical Learning Theory, John Wiley & Sons, New York (1998).Google Scholar
  34. L. Wang, J. Zhu, H. Zou: The doubly regularized support vector machine, Stat. Sinica 16/2 (2006) 589–615.Google Scholar
  35. M. M. West et al.: Predicting the clinical status of human breast cancer by using gene expression profiles, Proc. Natl. Acad. Sci. USA 98/20 (2001) 11462–11467.https://doi.org/10.1073/pnas.201162998Google Scholar
  36. J. Weston, A. Elisseeff, B. Schölkopf, M. Tipping: The use of zero-norm with linear models and kernel methods, J. Mach. Learn. Res. 3 (2003) 1439–1461.Google Scholar
  37. L. Yang, R. Ju: A DC programming approach for feature selection in the minimax probability machine, Int. J. Comput. Intell. Syst. 7/1 (2014) 12–24.https://doi.org/10.1080/18756891.2013.864471Google Scholar
  38. J. Zhu, S. Rosset, R. Tibshirani, T. J. Hastie: 1-norm support vector machines, in: Proc. 16th Neural Information Processing Systems (NIPS’03), MIT Press, Cambridge (2003) 49–56.Google Scholar
  39. H. Zou, T. Hastie: Regularization and variable selection via the elastic net, J. R. Stat. Soc. B 67/2 (2005) 301–320.https://doi.org/10.1111/j.1467-9868.2005.00503.xGoogle Scholar

Suggested citation

M. Carrasco, B. Ivorra, J. López, A. M. Ramos. “Robust SVM Classification with ℓ_0-Norm Feature Selection.” Journal of Convex Analysis 33 (2026), No. 3&4, 781–800. https://doi.org/10.68381/jca33044

Download BibTeXPublished by Heldermann Verlag.

Published by Heldermann Verlag, 2026. Rights now held by Banach Press LLC.