We prove that the terminal loss of a multilayer perceptron (MLP) under maximal update parametrization (μP) converges in probability to a deterministic continuous function, uniformly on every compact learning rate interval I ⊂ [0, ∞). The data and depth are fixed, and training consists of a fixed number of full-batch gradient descent steps. The activation is σ ∈ C\_b\textasciicircum{}2(ℝ), and only the hidden matrices are trained. If the limiting loss has a unique global minimizer in the interior of I, every measurable o\_ℙ(1)-approximate minimizer converges to it in probability. This uniform convergence also implies that transfer regret vanishes in probability as both source and target widths tend to infinity, provided the source optimization error is o\_ℙ(1). To obtain uniformity, we combine Tensor Program limits at fixed rates with a width-independent local H{\"o}lder estimate and a finite covering argument. We illustrate the loss curves and transfer regret across widths using one synthetic regression task.
MEKHZOUM, H 2026, 'Learning Rate Transfer via Uniform Convergence in Smooth Nonlinear μP MLPs', Transactions on Machine Learning Research.
MEKHZOUM, H. (2026). Learning Rate Transfer via Uniform Convergence in Smooth Nonlinear μP MLPs. Transactions on Machine Learning Research.
@article{ff261e4cf3984ea6927e6f3e48b57d45,
title = "Learning Rate Transfer via Uniform Convergence in Smooth Nonlinear μP MLPs",
abstract = "We prove that the terminal loss of a multilayer perceptron (MLP) under maximal update parametrization (μP) converges in probability to a deterministic continuous function, uniformly on every compact learning rate interval I ⊂ [0, ∞). The data and depth are fixed, and training consists of a fixed number of full-batch gradient descent steps. The activation is σ ∈ C\_b\textasciicircum{}2(ℝ), and only the hidden matrices are trained. If the limiting loss has a unique global minimizer in the interior of I, every measurable o\_ℙ(1)-approximate minimizer converges to it in probability. This uniform convergence also implies that transfer regret vanishes in probability as both source and target widths tend to infinity, provided the source optimization error is o\_ℙ(1). To obtain uniformity, we combine Tensor Program limits at fixed rates with a width-independent local H{\"o}lder estimate and a finite covering argument. We illustrate the loss curves and transfer regret across widths using one synthetic regression task.",
keywords = "Learning rate transfer, Uniform convergence, Maximal update parametrization, μP, Multilayer perceptrons, eural network optimization, Neural network scaling, Deep learning theory",
author = "Hamza MEKHZOUM",
note = "Published online in Transactions on Machine Learning Research (TMLR), November 2026. The journal follows a continuous publication model. The article is publicly available through OpenReview under identifier 0fyjoOFW9k. Official publication record: https://openreview.net/forum?id=0fyjoOFW9k",
year = "2026",
month = nov,
day = "5",
language = "English",
journal = "Transactions on Machine Learning Research",
issn = "2835-8856",
}