Training deep Graph Neural Networks (GNNs) has proved to be a challenging task. A key goal of many new GNN architectures is to enable the depth scaling seen in other types of deep learning models. However, unlike deep learning methods in other domains, deep GNNs do not show significant performance boosts when compared to their shallow counterparts (resulting in a flat curve of performance over depth). In this work, we investigate some of the reasons why this goal of depth still eludes GNN researchers. We also question the effectiveness of current methods to train deep GNNs and show evidence of different types of pathological behavior in these networks. Our results suggest that current approaches hide the problems with deep GNNs rather than solve them, as current deep GNNs are only as discriminative as their respective shallow versions.
Sousa Gomes, D, Efthymiadis, K, Nowe, A & Vrancx, P 2024, 'Depth Scaling in Graph Neural Networks: Understanding the Flat Curve Behavior', Transactions on Machine Learning Research (TMLR). <https://openreview.net/pdf?id=fdyHzoGT8g>
Sousa Gomes, D., Efthymiadis, K., Nowe, A., & Vrancx, P. (2024). Depth Scaling in Graph Neural Networks: Understanding the Flat Curve Behavior. Transactions on Machine Learning Research (TMLR). https://openreview.net/pdf?id=fdyHzoGT8g
@article{6b5fa01fe0f24a81b8e355d9feac84b9,
title = "Depth Scaling in Graph Neural Networks: Understanding the Flat Curve Behavior",
abstract = "Training deep Graph Neural Networks (GNNs) has proved to be a challenging task. A key goal of many new GNN architectures is to enable the depth scaling seen in other types of deep learning models. However, unlike deep learning methods in other domains, deep GNNs do not show significant performance boosts when compared to their shallow counterparts (resulting in a flat curve of performance over depth). In this work, we investigate some of the reasons why this goal of depth still eludes GNN researchers. We also question the effectiveness of current methods to train deep GNNs and show evidence of different types of pathological behavior in these networks. Our results suggest that current approaches hide the problems with deep GNNs rather than solve them, as current deep GNNs are only as discriminative as their respective shallow versions.",
keywords = "graph neural networks, depth evaluation",
author = "{Sousa Gomes}, Diana and Kyriakos Efthymiadis and Ann Nowe and Peter Vrancx",
note = "Publisher Copyright: {\textcopyright} 2024, Transactions on Machine Learning Research. All rights reserved.",
year = "2024",
language = "English",
journal = "Transactions on Machine Learning Research (TMLR)",
issn = "2835-8856",
}