There is a substantial demand for deep learning methods that can work with limited, high-dimensional, and noisy datasets. Nonetheless, current research mostly neglects this area, especially in the absence of prior expert knowledge or knowledge transfer. In this work, we bridge this gap by studying the performance of deep learning methods on the true data distribution in a limited, high-dimensional, and noisy data setting. To this end, we conduct a systematic evaluation that reduces the available training data while retaining the challenging properties mentioned above. Furthermore, we extensively search the space of hyperparameters and compare state-of-the-art architectures and models build and trained from scratch to advocate for the use of multi-objective tuning strategies. Our experiments highlight the lack of performative deep learning models in current literature and investigate the impact of training hyperparameters. We analyze the complexity of the models and demonstrate the advantage of choosing models tuned under multi-objective criteria in lower data regimes to reduce the likelihood to overfit. Lastly, we demonstrate the importance of selecting a proper inductive bias given a limited-sized dataset. Given our results, we conclude that tuning models using a multi-objective criterion results in simpler yet competitive models when reducing the number of data points.
Jaxy, S, Nowe, A & Libin, P 2024, 'Exploring the limits of deep learning in high-dimensional limited-sized data regimes using single- and multi-objective optimization strategies', Multi-Objective Decision Making Workshop at ECAI 2024, Santiago de Compostela, Spain, 20/10/24 - 20/10/24.
Jaxy, S., Nowe, A., & Libin, P. (2024). Exploring the limits of deep learning in high-dimensional limited-sized data regimes using single- and multi-objective optimization strategies. Poster session presented at Multi-Objective Decision Making Workshop at ECAI 2024, Santiago de Compostela, Spain.
@conference{50d36e015101450a9dee3acbbe2a9e20,
title = "Exploring the limits of deep learning in high-dimensional limited-sized data regimes using single- and multi-objective optimization strategies",
abstract = "There is a substantial demand for deep learning methods that can work with limited, high-dimensional, and noisy datasets. Nonetheless, current research mostly neglects this area, especially in the absence of prior expert knowledge or knowledge transfer. In this work, we bridge this gap by studying the performance of deep learning methods on the true data distribution in a limited, high-dimensional, and noisy data setting. To this end, we conduct a systematic evaluation that reduces the available training data while retaining the challenging properties mentioned above. Furthermore, we extensively search the space of hyperparameters and compare state-of-the-art architectures and models build and trained from scratch to advocate for the use of multi-objective tuning strategies. Our experiments highlight the lack of performative deep learning models in current literature and investigate the impact of training hyperparameters. We analyze the complexity of the models and demonstrate the advantage of choosing models tuned under multi-objective criteria in lower data regimes to reduce the likelihood to overfit. Lastly, we demonstrate the importance of selecting a proper inductive bias given a limited-sized dataset. Given our results, we conclude that tuning models using a multi-objective criterion results in simpler yet competitive models when reducing the number of data points.",
author = "Simon Jaxy and Ann Nowe and Pieter Libin",
year = "2024",
month = oct,
day = "18",
language = "English",
note = "Multi-Objective Decision Making Workshop at ECAI 2024, MODeM 2024 ; Conference date: 20-10-2024 Through 20-10-2024",
url = "https://modem2024.vub.ac.be",
}