{"id":3368,"date":"2026-06-24T10:59:22","date_gmt":"2026-06-24T08:59:22","guid":{"rendered":"https:\/\/neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/"},"modified":"2026-08-27T21:22:18","modified_gmt":"2026-08-27T19:22:18","slug":"5_algorithms_to_train_a_neural_network","status":"publish","type":"blog","link":"https:\/\/www.neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/","title":{"rendered":"Neural Network Optimizers: From Gradient Descent to AdamW"},"content":{"rendered":"<style>\n.ndb{width:100vw;margin-left:calc(50% - 50vw);background:#eee;padding:22px 24px 18px;font-family:\"Outfit\",\"Roboto\",Arial,sans-serif;color:#1b2635}\n.ndb *{box-sizing:border-box}.ndb a{color:#2d799f;text-decoration:none}.ndb a:hover{text-decoration:underline}.ndb-wrap{width:min(100%,1120px);margin:0 auto}\n.ndb-lead{margin:0 0 34px;font-size:18px;line-height:1.65;color:#344654}.ndb-lead p{margin:0 0 14px}\n.ndb-toc{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:8px 36px;margin:0 0 48px;padding:20px 0;border-top:1px solid #cfdbe2;border-bottom:1px solid #cfdbe2;list-style:none}\n.ndb-toc a{display:block;padding:4px 0;color:#12354b;font-weight:600}\n.ndb-section{margin:0 0 54px;scroll-margin-top:90px}.ndb-section h2{position:relative;margin:0 0 20px;padding:0 0 12px;border-bottom:1px solid #cfdbe2;color:#001233;font-size:26px;line-height:1.25}.ndb-section h2:after{position:absolute;bottom:-1px;left:0;width:64px;height:3px;background:#56a1c8;content:\"\"}\n.ndb-section h3{margin:28px 0 12px;color:#12354b;font-size:20px}.ndb-section p{margin:0 0 15px;font-size:16.5px;line-height:1.65;color:#33424f}.ndb-section ul,.ndb-section ol{margin:0 0 18px;padding-left:24px}.ndb-section li{margin:7px 0;font-size:16px;line-height:1.58;color:#33424f}\n.ndb-equation{max-width:100%;margin:20px 0;overflow-x:auto;text-align:center}.ndb-figure{margin:26px auto;text-align:center}.ndb-figure img{display:block;width:auto;max-width:min(720px,100%);height:auto;margin:0 auto}.ndb-figure figcaption{margin-top:10px;color:#526674;font-size:14px;line-height:1.45}\n.ndb-table-scroll{max-width:100%;margin:24px 0;overflow-x:auto}.ndb table{width:100%;min-width:820px;border-collapse:collapse;background:transparent;font-size:14.5px}.ndb th,.ndb td{padding:12px 14px;border-bottom:1px solid #ccd9e1;text-align:left;vertical-align:top}.ndb thead th{background:#12354b;color:#fff;font-weight:700}.ndb tbody th{color:#12354b;font-weight:700}.ndb tbody tr:nth-child(even){background:rgba(255,255,255,.32)}\n.ndb-summary{margin:20px 0;padding-left:18px;border-left:4px solid #56a1c8}.ndb-summary p{margin:0}.ndb-references li{overflow-wrap:anywhere}\n@media(max-width:760px){.ndb{padding:14px 16px}.ndb-toc{grid-template-columns:1fr}.ndb-section h2{font-size:23px}.ndb-lead{font-size:17px}}\n<\/style>\n<div class=\"ndb\">\n<div class=\"ndb-wrap\">\n<div class=\"ndb-lead\">\n<p>An <strong>optimizer<\/strong> is the procedure that updates a neural network&#8217;s trainable parameters to reduce its loss. It is one part of the training strategy, together with the loss function, data batches, learning-rate schedule, regularization, and stopping criteria.<\/p>\n<p>There is no optimizer that is best for every neural network. Full-batch methods can exploit accurate gradient and curvature information on modest problems, while mini-batch methods make it possible to train networks with large datasets and millions or billions of parameters.<\/p>\n<p>This guide explains the main optimizer families, from gradient descent and curvature-aware methods to stochastic gradient descent, Adam, and AdamW. It compares their memory requirements, batch regimes, strengths, and limitations so that you can select an appropriate method for your problem.<\/p>\n<\/div>\n\n<ul class=\"ndb-toc\">\n<li><a href=\"#optimization-problem\">The optimization problem<\/a><\/li>\n<li><a href=\"#batch-regimes\">Full-batch and mini-batch training<\/a><\/li>\n<li><a href=\"#gradient-descent\">Gradient descent<\/a><\/li>\n<li><a href=\"#newton-method\">Newton&#8217;s method<\/a><\/li>\n<li><a href=\"#quasi-newton\">Quasi-Newton: BFGS and L-BFGS<\/a><\/li>\n<li><a href=\"#levenberg-marquardt\">Levenberg-Marquardt<\/a><\/li>\n<li><a href=\"#stochastic-gradient-descent\">Stochastic gradient descent<\/a><\/li>\n<li><a href=\"#adam-adamw\">Adam and AdamW<\/a><\/li>\n<li><a href=\"#optimizer-comparison\">Optimizer comparison<\/a><\/li>\n<li><a href=\"#choose-optimizer\">How to choose an optimizer<\/a><\/li>\n<li><a href=\"#emerging-optimizers\">Emerging optimizers<\/a><\/li>\n<li><a href=\"#conclusions\">Conclusions<\/a><\/li>\n<\/ul>\n\n<section class=\"ndb-section\" id=\"optimization-problem\">\n<h2>The optimization problem<\/h2>\n<p>Training searches for a parameter vector \\(\\boldsymbol{\\theta}^{*}\\) that minimizes a loss function. For a batch of \\(B\\) samples, a common formulation is<\/p>\n<div class=\"ndb-equation\">$$\nf(\\boldsymbol{\\theta})=\n\\frac{1}{B}\\sum_{b=1}^{B}\\ell_b(\\boldsymbol{\\theta})\n+\\lambda\\,\\Omega(\\boldsymbol{\\theta}),\n$$<\/div>\n<p>where \\(\\ell_b\\) is the error for sample \\(b\\), \\(\\Omega\\) is an optional regularization term, and \\(\\lambda\\) controls its contribution. Backpropagation computes the gradient<\/p>\n<div class=\"ndb-equation\">$$\n\\mathbf{g}=\\nabla f(\\boldsymbol{\\theta}),\n$$<\/div>\n<p>which indicates how the loss changes with each parameter. Curvature-aware methods also use the Hessian \\(\\mathbf{H}=\\nabla^2 f\\), an approximation to it, or a residual Jacobian.<\/p>\n<p>Neural-network losses are generally non-convex and high-dimensional. In practice, an optimizer aims for parameters that produce a low training loss and strong validation performance; it does not normally certify a global minimum.<\/p>\n<figure class=\"ndb-figure\">\n<img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/loss-function.svg\" alt=\"Loss function over neural network parameters\" \/>\n<figcaption>A neural-network optimizer searches the parameter space for a low-loss solution.<\/figcaption>\n<\/figure>\n<\/section>\n\n<section class=\"ndb-section\" id=\"batch-regimes\">\n<h2>Full-batch and mini-batch training<\/h2>\n<p>The most important distinction missing from many classical optimizer comparisons is how much data is used for each update.<\/p>\n<ul>\n<li><strong>Full-batch training<\/strong> evaluates the gradient using the complete training set. Its updates are deterministic, but each one becomes expensive when the dataset is large. Gradient descent, Newton, BFGS, L-BFGS, and Levenberg-Marquardt are commonly used in this regime.<\/li>\n<li><strong>Mini-batch training<\/strong> estimates the gradient from a subset of \\(B\\) samples. The estimate is noisy, but updates are cheaper and map efficiently to GPUs. SGD, Adam, and AdamW normally operate in this regime.<\/li>\n<li><strong>Online training<\/strong> is the limiting case \\(B=1\\). Its updates have maximum variance and are now less common than true mini-batches for hardware-efficient deep learning.<\/li>\n<\/ul>\n<p>One epoch is one complete pass through the training data. A full-batch method performs one update per epoch, whereas a mini-batch method performs approximately \\(Q\/B\\) updates, where \\(Q\\) is the number of training samples.<\/p>\n<p>Deterministic methods can use a line search to select a step size. Mini-batch methods usually use a prescribed learning rate and schedule because noisy batch losses make line minimization unreliable and expensive.<\/p>\n<\/section>\n\n<section class=\"ndb-section\" id=\"gradient-descent\">\n<h2>Gradient descent<\/h2>\n<p>Gradient descent is the simplest first-order optimizer. At iteration \\(i\\), it moves the parameters in the direction of the negative full-batch gradient:<\/p>\n<div class=\"ndb-equation\">$$\n\\boldsymbol{\\theta}^{(i+1)}=\n\\boldsymbol{\\theta}^{(i)}-\n\\eta^{(i)}\\mathbf{g}^{(i)},\n$$<\/div>\n<p>where \\(\\eta^{(i)}>0\\) is the learning rate. It can be fixed, scheduled, or selected by a line search when the loss is evaluated deterministically.<\/p>\n<p>Gradient descent requires little optimizer state, but it can converge slowly on poorly conditioned loss surfaces. In a long, narrow valley, successive gradients point across the valley and produce an inefficient zigzag path.<\/p>\n<figure class=\"ndb-figure\">\n<img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/gradient_descent_graph_big.webp\" alt=\"Gradient descent oscillating across a narrow loss valley\" \/>\n<figcaption>Gradient descent can oscillate across directions with high curvature and advance slowly along flatter directions.<\/figcaption>\n<\/figure>\n<p>Full-batch gradient descent is useful as a baseline and for modest deterministic problems. Large deep networks generally use its mini-batch descendants rather than plain gradient descent.<\/p>\n<\/section>\n\n<section class=\"ndb-section\" id=\"newton-method\">\n<h2>Newton&#8217;s method<\/h2>\n<p>Newton&#8217;s method uses both the gradient and Hessian to account for local curvature. Instead of explicitly calculating an inverse, a numerical implementation solves<\/p>\n<div class=\"ndb-equation\">$$\n\\mathbf{H}^{(i)}\\,\\boldsymbol{\\Delta}^{(i)}=-\\mathbf{g}^{(i)},\n\\qquad\n\\boldsymbol{\\theta}^{(i+1)}=\n\\boldsymbol{\\theta}^{(i)}+\n\\eta^{(i)}\\boldsymbol{\\Delta}^{(i)}.\n$$<\/div>\n<p>Near a well-behaved minimum, Newton&#8217;s method can converge in far fewer iterations than gradient descent. However, a network with \\(D\\) parameters has a \\(D\\times D\\) Hessian. Storing it requires \\(O(D^2)\\) memory, and a dense factorization typically requires \\(O(D^3)\\) work.<\/p>\n<p>The Hessian can also be indefinite away from a minimum, producing a direction that does not reduce the loss. Damping, trust regions, or a fallback descent direction are therefore needed in robust implementations.<\/p>\n<p>Exact Newton training is mainly a conceptual reference or a specialist option for very small networks. Quasi-Newton and Gauss-Newton methods retain some curvature information at a lower cost.<\/p>\n<\/section>\n\n<section class=\"ndb-section\" id=\"quasi-newton\">\n<h2>Quasi-Newton methods: BFGS and L-BFGS<\/h2>\n<p>Quasi-Newton methods estimate curvature from successive parameter and gradient differences. They avoid evaluating the exact Hessian while constructing a search direction of the form<\/p>\n<div class=\"ndb-equation\">$$\n\\mathbf{s}^{(i)}=-\\mathbf{G}^{(i)}\\mathbf{g}^{(i)},\n\\qquad\n\\boldsymbol{\\theta}^{(i+1)}=\n\\boldsymbol{\\theta}^{(i)}+\n\\eta^{(i)}\\mathbf{s}^{(i)},\n$$<\/div>\n<p>where \\(\\mathbf{G}^{(i)}\\) approximates the inverse Hessian. A line search, commonly using an Armijo or Wolfe condition, chooses a step that provides sufficient decrease.<\/p>\n<h3>BFGS<\/h3>\n<p>The Broyden-Fletcher-Goldfarb-Shanno method updates a dense inverse-Hessian approximation using the latest parameter difference \\(\\Delta\\boldsymbol{\\theta}\\) and gradient difference \\(\\Delta\\mathbf{g}\\). It often converges much faster than gradient descent on smooth, deterministic problems, but storing \\(\\mathbf{G}\\) requires \\(O(D^2)\\) memory.<\/p>\n<h3>L-BFGS<\/h3>\n<p>Limited-memory BFGS is the same quasi-Newton family with a different storage strategy. It retains only the most recent \\(m\\) parameter and gradient difference pairs and reconstructs their action on the gradient when needed. This reduces optimizer-state memory to approximately \\(O(mD)\\), with \\(m\\) usually much smaller than \\(D\\).<\/p>\n<p>BFGS is suitable for small-to-medium full-batch networks. L-BFGS extends the approach to larger deterministic parameter vectors, although both methods become less reliable when curvature pairs are estimated from unrelated noisy mini-batches.<\/p>\n<figure class=\"ndb-figure\">\n<img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/quasi-newton_algorithm_big.webp\" alt=\"Quasi-Newton neural network training process\" \/>\n<figcaption>BFGS and L-BFGS use the same quasi-Newton principle but store curvature information differently.<\/figcaption>\n<\/figure>\n<\/section>\n\n<section class=\"ndb-section\" id=\"levenberg-marquardt\">\n<h2>Levenberg-Marquardt algorithm<\/h2>\n<p>Levenberg-Marquardt is a specialized full-batch optimizer for smooth residual least-squares problems. Suppose a network produces residuals \\(\\mathbf{h}\\) with one component per sample and output, and define<\/p>\n<div class=\"ndb-equation\">$$\nf(\\boldsymbol{\\theta})=\n\\frac{1}{2BM}\\,\\mathbf{h}^{\\mathsf T}\\mathbf{h},\n$$<\/div>\n<p>where \\(B\\) is the number of training samples and \\(M\\) is the number of outputs. If \\(\\mathbf{J}\\) is the residual Jacobian, with shape \\((BM)\\times D\\), the damped Gauss-Newton step solves<\/p>\n<div class=\"ndb-equation\">$$\n\\left(\n\\frac{1}{BM}\\mathbf{J}^{\\mathsf T}\\mathbf{J}\n+\\lambda\\mathbf{I}\n\\right)\\boldsymbol{\\Delta}\n=-\frac{1}{BM}\\mathbf{J}^{\\mathsf T}\\mathbf{h}.\n$$<\/div>\n<p>For small damping \\(\\lambda\\), the update approaches Gauss-Newton. For large \\(\\lambda\\), it approaches a small gradient-descent step. Successful candidate steps reduce the damping; rejected steps increase it.<\/p>\n<p>Levenberg-Marquardt can converge rapidly for small, smooth regression networks. Its limitation is scale: explicitly storing \\(\\mathbf{J}\\) and the \\(D\\times D\\) Gauss-Newton matrix consumes substantial memory, and the dense linear solve is expensive.<\/p>\n<p>The method applies directly to residual least-squares objectives, not to cross-entropy. When applied to the same residuals without changing other objective terms, MSE, SSE, and RMSE have the same minimizer; however, the LM calculation is formulated through the underlying residual vector rather than through an arbitrary scalar loss transformation.<\/p>\n<figure class=\"ndb-figure\">\n<img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/levenberg_algorithm_big.webp\" alt=\"Levenberg-Marquardt neural network training process\" \/>\n<figcaption>Levenberg-Marquardt adapts its damping between gradient-like and Gauss-Newton-like behavior.<\/figcaption>\n<\/figure>\n<\/section>\n\n<section class=\"ndb-section\" id=\"stochastic-gradient-descent\">\n<h2>Stochastic gradient descent<\/h2>\n<p>In contemporary deep learning, stochastic gradient descent usually means <em>mini-batch<\/em> SGD. At each iteration, it calculates a gradient \\(\\mathbf{g}_{B}^{(i)}\\) from a batch of \\(B\\) samples:<\/p>\n<div class=\"ndb-equation\">$$\n\\boldsymbol{\\theta}^{(i+1)}=\n\\boldsymbol{\\theta}^{(i)}-\n\\eta^{(i)}\\mathbf{g}_{B}^{(i)}.\n$$<\/div>\n<p>A mini-batch gradient is noisy, but it is much cheaper than a full-dataset gradient and can be evaluated efficiently using parallel hardware. Batch size and learning rate must be chosen together.<\/p>\n<h3>Momentum<\/h3>\n<p>Momentum accumulates a velocity vector that smooths oscillations and reinforces directions that remain consistent across batches:<\/p>\n<div class=\"ndb-equation\">$$\n\\mathbf{v}^{(i)}=\n\\mu\\mathbf{v}^{(i-1)}-\n\\eta^{(i)}\\mathbf{g}_{B}^{(i)},\n\\qquad\n\\boldsymbol{\\theta}^{(i+1)}=\n\\boldsymbol{\\theta}^{(i)}+\\mathbf{v}^{(i)}.\n$$<\/div>\n<p>A momentum coefficient of \\(\\mu=0.9\\) is a common starting point, not a universal optimum. Momentum adds one state value per parameter.<\/p>\n<h3>Nesterov momentum<\/h3>\n<p>Nesterov momentum evaluates the gradient with respect to a look-ahead position, allowing the optimizer to correct its direction before completing the momentum step. Modern libraries often use an algebraically rearranged implementation that avoids a second gradient calculation.<\/p>\n<h3>Learning-rate schedules<\/h3>\n<p>SGD performance depends strongly on its learning-rate schedule. Step decay, exponential decay, cosine decay, and warm-up followed by decay are common choices. Line searches are rarely used because batch-to-batch noise makes them unreliable.<\/p>\n<p>SGD with momentum remains attractive for large networks when optimizer-state memory is constrained or when a carefully tuned SGD recipe provides strong validation performance. It generally needs more learning-rate tuning than Adam.<\/p>\n<\/section>\n\n<section class=\"ndb-section\" id=\"adam-adamw\">\n<h2>Adam and AdamW<\/h2>\n<p>Adam adapts the step for every parameter using exponential moving averages of the gradient and squared gradient. At iteration \\(i\\),<\/p>\n<div class=\"ndb-equation\">$$\n\\begin{aligned}\n\\mathbf{m}^{(i)}&#038;=\\beta_1\\mathbf{m}^{(i-1)}+(1-\\beta_1)\\mathbf{g}_{B}^{(i)},\\\\\n\\mathbf{v}^{(i)}&#038;=\\beta_2\\mathbf{v}^{(i-1)}+(1-\\beta_2)\\left(\\mathbf{g}_{B}^{(i)}\\odot\\mathbf{g}_{B}^{(i)}\\right).\n\\end{aligned}\n$$<\/div>\n<p>Because both averages start at zero, Adam applies bias correction:<\/p>\n<div class=\"ndb-equation\">$$\n\\widehat{\\mathbf{m}}^{(i)}=\n\\frac{\\mathbf{m}^{(i)}}{1-\\beta_1^i},\n\\qquad\n\\widehat{\\mathbf{v}}^{(i)}=\n\\frac{\\mathbf{v}^{(i)}}{1-\\beta_2^i}.\n$$<\/div>\n<p>The parameter update is<\/p>\n<div class=\"ndb-equation\">$$\n\\boldsymbol{\\theta}^{(i+1)}=\n\\boldsymbol{\\theta}^{(i)}-\n\\eta^{(i)}\n\\frac{\\widehat{\\mathbf{m}}^{(i)}}\n{\\sqrt{\\widehat{\\mathbf{v}}^{(i)}}+\\epsilon},\n$$<\/div>\n<p>where all divisions and square roots are element-wise. The original defaults \\(\\beta_1=0.9\\), \\(\\beta_2=0.999\\), \\(\\epsilon=10^{-8}\\), and \\(\\eta=10^{-3}\\) are useful starting values, but the learning rate and schedule still require validation.<\/p>\n<p>Adam is a robust mini-batch baseline for large, non-convex problems and can work well with noisy or sparse gradients. Its two moment estimates require two optimizer-state values per parameter, twice the state of momentum SGD.<\/p>\n<h3>AdamW and decoupled weight decay<\/h3>\n<p>For ordinary SGD, adding an L2 penalty to the loss is closely related to multiplying the parameters by a weight-decay factor. This equivalence does not hold for Adam because its adaptive denominator rescales the L2 contribution differently for every parameter.<\/p>\n<p>AdamW decouples weight decay from the adaptive gradient calculation:<\/p>\n<div class=\"ndb-equation\">$$\n\\boldsymbol{\\theta}^{(i+1)}=\n\\left(1-\\eta^{(i)}\\lambda_{\\mathrm{wd}}\\right)\n\\boldsymbol{\\theta}^{(i)}-\n\\eta^{(i)}\n\\frac{\\widehat{\\mathbf{m}}^{(i)}}\n{\\sqrt{\\widehat{\\mathbf{v}}^{(i)}}+\\epsilon}.\n$$<\/div>\n<p>This separation makes the learning rate and weight-decay coefficient easier to reason about independently. AdamW is a common choice for transformers and other modern deep networks when explicit weight decay is desired.<\/p>\n<\/section>\n\n<section class=\"ndb-section\" id=\"optimizer-comparison\">\n<h2>Optimizer comparison<\/h2>\n<p>The following table compares optimizer-state memory rather than total training memory. Parameters, gradients, activations, temporary tensors, and dataset storage are additional costs. Here, \\(D\\) is the number of trainable parameters, \\(m\\) is the L-BFGS history length, and \\(BM\\) is the number of residuals in a full LM batch.<\/p>\n<div class=\"ndb-table-scroll\">\n<table>\n<thead><tr><th>Optimizer family<\/th><th>Typical batch regime<\/th><th>Information used<\/th><th>Extra optimizer state<\/th><th>Best suited for<\/th><th>Main limitation<\/th><\/tr><\/thead>\n<tbody>\n<tr><th>Gradient descent<\/th><td>Full batch<\/td><td>Gradient<\/td><td>Minimal<\/td><td>Baselines and modest deterministic problems<\/td><td>Slow on poorly conditioned losses<\/td><\/tr>\n<tr><th>Newton<\/th><td>Full batch<\/td><td>Gradient and exact Hessian<\/td><td>\\(D^2\\)<\/td><td>Very small smooth problems<\/td><td>Hessian storage and dense solve<\/td><\/tr>\n<tr><th>Quasi-Newton<br \/>BFGS \/ L-BFGS<\/th><td>Full batch<\/td><td>Gradient and curvature pairs<\/td><td>\\(D^2\\) \/ approximately \\(2mD\\)<\/td><td>Small-to-medium smooth deterministic problems<\/td><td>Curvature estimates degrade with noisy batches<\/td><\/tr>\n<tr><th>Levenberg-Marquardt<\/th><td>Full batch<\/td><td>Residual Jacobian<\/td><td>\\(BMD+D^2\\)<\/td><td>Small residual least-squares networks<\/td><td>Specialized loss and high memory use<\/td><\/tr>\n<tr><th>SGD with momentum<\/th><td>Mini-batch<\/td><td>Batch gradient and velocity<\/td><td>\\(D\\)<\/td><td>Large networks; memory-conscious training<\/td><td>Sensitive to learning rate and schedule<\/td><\/tr>\n<tr><th>Adam \/ AdamW<\/th><td>Mini-batch<\/td><td>First and second gradient moments<\/td><td>\\(2D\\)<\/td><td>General deep-learning baseline; transformers<\/td><td>More optimizer state and regularization choices<\/td><\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>There is no universal ranking by \u201cspeed.\u201d Time per update, number of updates, hardware utilization, final validation performance, and hyperparameter budget can all change the result. Comparisons should report wall-clock time to a common validation target, peak memory, final test performance, and variability across repeated runs.<\/p>\n<p><strong>Product note:<\/strong> Neural Designer documents gradient descent, Newton, quasi-Newton, Levenberg-Marquardt, SGD, and Adam in its <a href=\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/\">training strategy guide<\/a>. Availability of individual variants such as L-BFGS or AdamW can depend on the installed product version.<\/p>\n<\/section>\n\n<section class=\"ndb-section\" id=\"choose-optimizer\">\n<h2>How to choose an optimizer<\/h2>\n<ol>\n<li><strong>Start with the loss and scale.<\/strong> If the objective is smooth residual least squares and both the network and dataset are modest, Levenberg-Marquardt is a strong candidate.<\/li>\n<li><strong>For a smooth general full-batch problem, try quasi-Newton.<\/strong> Use BFGS when its dense state fits comfortably in memory and L-BFGS when the full approximation is too large.<\/li>\n<li><strong>For large datasets or deep networks, use mini-batches.<\/strong> Adam is a reliable starting point; use AdamW when decoupled weight decay is available and appropriate.<\/li>\n<li><strong>Use SGD with momentum when memory matters or when the workload has a proven SGD recipe.<\/strong> Expect to tune the learning rate, momentum, batch size, and schedule together.<\/li>\n<li><strong>Reserve exact Newton for very small or specialist problems.<\/strong> It is valuable for understanding curvature, but its dense Hessian rarely scales to contemporary networks.<\/li>\n<\/ol>\n<p>Optimizer choice is only one part of training. Initialization, feature scaling, normalization, batch size, mixed precision, gradient clipping, regularization, learning-rate scheduling, and validation-based early stopping can matter as much as the optimizer name.<\/p>\n<div class=\"ndb-summary\"><p>A practical comparison keeps the model, data split, loss, stopping budget, and evaluation metric fixed, then tunes each optimizer fairly and compares validation performance, elapsed time, and peak memory.<\/p><\/div>\n<\/section>\n\n<section class=\"ndb-section\" id=\"emerging-optimizers\">\n<h2>Emerging optimizers<\/h2>\n<p>AdamW and momentum SGD remain strong reference methods, but optimizer research continues to address memory and curvature at scale:<\/p>\n<ul>\n<li><strong>Adafactor<\/strong> factors the second-moment accumulator for matrix parameters, reducing optimizer-state memory in large transformer models.<\/li>\n<li><strong>Shampoo<\/strong> uses tensor-structured preconditioners to capture more geometry than diagonal adaptive methods.<\/li>\n<li><strong>Lion<\/strong> uses sign-based momentum updates and stores one momentum state instead of Adam&#8217;s two moments.<\/li>\n<li><strong>Sophia<\/strong> applies a clipped update preconditioned by a lightweight diagonal Hessian estimate for language-model pretraining.<\/li>\n<li><strong>Muon<\/strong> orthogonalizes updates for matrix-valued hidden-layer parameters and is being studied as an alternative for large language-model training.<\/li>\n<\/ul>\n<p>These methods are promising but task-specific. They should be presented as research and engineering options, not as universal replacements for AdamW, SGD, quasi-Newton, or Levenberg-Marquardt.<\/p>\n<\/section>\n\n<section class=\"ndb-section\" id=\"conclusions\">\n<h2>Conclusions<\/h2>\n<p>Modern neural-network optimization spans two complementary regimes. Full-batch curvature-aware methods can converge rapidly when the model and dataset are modest. Mini-batch methods trade exact gradient information for scalable, hardware-efficient updates.<\/p>\n<ul>\n<li>Use <strong>Levenberg-Marquardt<\/strong> for small residual least-squares networks.<\/li>\n<li>Use <strong>BFGS or L-BFGS<\/strong> for smooth deterministic problems that benefit from curvature information.<\/li>\n<li>Use <strong>Adam or AdamW<\/strong> as practical starting points for large deep networks.<\/li>\n<li>Use <strong>SGD with momentum<\/strong> when its memory efficiency or task-specific generalization makes it preferable.<\/li>\n<\/ul>\n<p>The best choice is the optimizer that reaches the required validation quality within the available time and memory budget. That choice should be established by a fair experiment rather than by a universal ranking.<\/p>\n<aside class=\"ndb-cta\" aria-labelledby=\"optimizer-cta-title\" style=\"margin:2rem 0;padding:1.5rem;border:1px solid #d8e2ef;border-radius:12px;background:#f6f9fc;\">\n<h2 id=\"optimizer-cta-title\" style=\"margin-top:0;\">Compare optimizers on your own data<\/h2>\n<p>Use Neural Designer to train the same network with different optimization algorithms, compare validation results, and choose the best strategy without writing training code.<\/p>\n<p style=\"display:flex;flex-wrap:wrap;gap:0.75rem;margin-bottom:0;\">\n<a href=\"https:\/\/www.neuraldesigner.com\/downloads\/\" data-nd-cta=\"optimizer-article-download\" onclick=\"if(typeof gtag==='function'){gtag('event','cta_click',{cta_name:'optimizer_article_download',link_url:this.href});}\" style=\"display:inline-block;padding:0.75rem 1rem;border-radius:6px;background:#1261a0;color:#fff;text-decoration:none;font-weight:600;\">Download Neural Designer<\/a>\n<a href=\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/\" data-nd-cta=\"optimizer-article-tutorial\" onclick=\"if(typeof gtag==='function'){gtag('event','cta_click',{cta_name:'optimizer_article_tutorial',link_url:this.href});}\" style=\"display:inline-block;padding:0.75rem 1rem;border:1px solid #1261a0;border-radius:6px;color:#1261a0;text-decoration:none;font-weight:600;\">Follow the training tutorial<\/a>\n<\/p>\n<\/aside>\n<\/section>\n\n<section class=\"ndb-section\" id=\"references\">\n<h2>References<\/h2>\n<ol class=\"ndb-references\">\n<li>Sutskever, I. et al. (2013). <a href=\"https:\/\/proceedings.mlr.press\/v28\/sutskever13.html\">On the importance of initialization and momentum in deep learning<\/a>.<\/li>\n<li>Nocedal, J. (1980). <a href=\"https:\/\/doi.org\/10.1090\/S0025-5718-1980-0572855-7\">Updating quasi-Newton matrices with limited storage<\/a>.<\/li>\n<li>Marquardt, D. W. (1963). <a href=\"https:\/\/doi.org\/10.1137\/0111030\">An algorithm for least-squares estimation of nonlinear parameters<\/a>.<\/li>\n<li>Kingma, D. P. and Ba, J. (2015). <a href=\"https:\/\/arxiv.org\/abs\/1412.6980\">Adam: A method for stochastic optimization<\/a>.<\/li>\n<li>Loshchilov, I. and Hutter, F. (2019). <a href=\"https:\/\/openreview.net\/forum?id=Bkg6RiCqY7\">Decoupled weight decay regularization<\/a>.<\/li>\n<li>Shazeer, N. and Stern, M. (2018). <a href=\"https:\/\/proceedings.mlr.press\/v80\/shazeer18a.html\">Adafactor: Adaptive learning rates with sublinear memory cost<\/a>.<\/li>\n<li>Gupta, V. et al. (2018). <a href=\"https:\/\/proceedings.mlr.press\/v80\/gupta18a.html\">Shampoo: Preconditioned stochastic tensor optimization<\/a>.<\/li>\n<li>Chen, X. et al. (2023). <a href=\"https:\/\/arxiv.org\/abs\/2302.06675\">Symbolic discovery of optimization algorithms<\/a>.<\/li>\n<li>Liu, H. et al. (2023). <a href=\"https:\/\/arxiv.org\/abs\/2305.14342\">Sophia: A scalable stochastic second-order optimizer for language model pre-training<\/a>.<\/li>\n<li>Liu, J. et al. (2025). <a href=\"https:\/\/arxiv.org\/abs\/2502.16982\">Muon is scalable for LLM training<\/a>.<\/li>\n<\/ol>\n<\/section>\n<\/div>\n<\/div>\n","protected":false},"author":20,"featured_media":2664,"template":"","categories":[],"tags":[36],"class_list":["post-3368","blog","type-blog","status-publish","has-post-thumbnail","hentry","tag-tutorials"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.4 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Neural Network Optimizers: From Gradient Descent to AdamW<\/title>\n<meta name=\"description\" content=\"Compare gradient descent, Newton, BFGS, L-BFGS, Levenberg-Marquardt, SGD, Adam and AdamW, and learn which optimizer fits each neural network.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Neural Network Optimizers: From Gradient Descent to AdamW\" \/>\n<meta property=\"og:description\" content=\"A practical guide to full-batch and mini-batch neural network optimizers, including BFGS, L-BFGS, Levenberg-Marquardt, SGD, Adam and AdamW.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/\" \/>\n<meta property=\"og:site_name\" content=\"Neural Designer\" \/>\n<meta property=\"article:modified_time\" content=\"2026-08-27T19:22:18+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/algorithms-train-network.webp\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/webp\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:title\" content=\"Neural Network Optimizers: From Gradient Descent to AdamW\" \/>\n<meta name=\"twitter:description\" content=\"A practical guide to full-batch and mini-batch neural network optimizers, including BFGS, L-BFGS, Levenberg-Marquardt, SGD, Adam and AdamW.\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/algorithms-train-network.webp\" \/>\n<meta name=\"twitter:site\" content=\"@NeuralDesigner\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"12 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/\",\"url\":\"https:\/\/www.neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/\",\"name\":\"Neural Network Optimizers: From Gradient Descent to AdamW\",\"isPartOf\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/algorithms-train-network.webp\",\"datePublished\":\"2026-06-24T08:59:22+00:00\",\"dateModified\":\"2026-08-27T19:22:18+00:00\",\"description\":\"Compare gradient descent, Newton, BFGS, L-BFGS, Levenberg-Marquardt, SGD, Adam and AdamW, and learn which optimizer fits each neural network.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/#primaryimage\",\"url\":\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/algorithms-train-network.webp\",\"contentUrl\":\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/algorithms-train-network.webp\",\"width\":1200,\"height\":628,\"caption\":\"Algorithms to train a neural network\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.neuraldesigner.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Blog\",\"item\":\"https:\/\/www.neuraldesigner.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Neural Network Optimizers: From Gradient Descent to AdamW\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.neuraldesigner.com\/#website\",\"url\":\"https:\/\/www.neuraldesigner.com\/\",\"name\":\"Neural Designer\",\"description\":\"Explainable AI Platform\",\"publisher\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.neuraldesigner.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.neuraldesigner.com\/#organization\",\"name\":\"Neural Designer\",\"url\":\"https:\/\/www.neuraldesigner.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.neuraldesigner.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/05\/logo-neural-1.png\",\"contentUrl\":\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/05\/logo-neural-1.png\",\"width\":1024,\"height\":223,\"caption\":\"Neural Designer\"},\"image\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/NeuralDesigner\",\"https:\/\/es.linkedin.com\/showcase\/neuraldesigner\/\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Neural Network Optimizers: From Gradient Descent to AdamW","description":"Compare gradient descent, Newton, BFGS, L-BFGS, Levenberg-Marquardt, SGD, Adam and AdamW, and learn which optimizer fits each neural network.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/","og_locale":"en_US","og_type":"article","og_title":"Neural Network Optimizers: From Gradient Descent to AdamW","og_description":"A practical guide to full-batch and mini-batch neural network optimizers, including BFGS, L-BFGS, Levenberg-Marquardt, SGD, Adam and AdamW.","og_url":"https:\/\/www.neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/","og_site_name":"Neural Designer","article_modified_time":"2026-08-27T19:22:18+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/algorithms-train-network.webp","type":"image\/webp"}],"twitter_card":"summary_large_image","twitter_title":"Neural Network Optimizers: From Gradient Descent to AdamW","twitter_description":"A practical guide to full-batch and mini-batch neural network optimizers, including BFGS, L-BFGS, Levenberg-Marquardt, SGD, Adam and AdamW.","twitter_image":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/algorithms-train-network.webp","twitter_site":"@NeuralDesigner","twitter_misc":{"Est. reading time":"12 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/","url":"https:\/\/www.neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/","name":"Neural Network Optimizers: From Gradient Descent to AdamW","isPartOf":{"@id":"https:\/\/www.neuraldesigner.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/#primaryimage"},"image":{"@id":"https:\/\/www.neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/#primaryimage"},"thumbnailUrl":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/algorithms-train-network.webp","datePublished":"2026-06-24T08:59:22+00:00","dateModified":"2026-08-27T19:22:18+00:00","description":"Compare gradient descent, Newton, BFGS, L-BFGS, Levenberg-Marquardt, SGD, Adam and AdamW, and learn which optimizer fits each neural network.","breadcrumb":{"@id":"https:\/\/www.neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/#primaryimage","url":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/algorithms-train-network.webp","contentUrl":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/algorithms-train-network.webp","width":1200,"height":628,"caption":"Algorithms to train a neural network"},{"@type":"BreadcrumbList","@id":"https:\/\/www.neuraldesigner.com\/blog\/5_algorithms_to_train_a_neural_network\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.neuraldesigner.com\/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https:\/\/www.neuraldesigner.com\/blog\/"},{"@type":"ListItem","position":3,"name":"Neural Network Optimizers: From Gradient Descent to AdamW"}]},{"@type":"WebSite","@id":"https:\/\/www.neuraldesigner.com\/#website","url":"https:\/\/www.neuraldesigner.com\/","name":"Neural Designer","description":"Explainable AI Platform","publisher":{"@id":"https:\/\/www.neuraldesigner.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.neuraldesigner.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.neuraldesigner.com\/#organization","name":"Neural Designer","url":"https:\/\/www.neuraldesigner.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.neuraldesigner.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/05\/logo-neural-1.png","contentUrl":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/05\/logo-neural-1.png","width":1024,"height":223,"caption":"Neural Designer"},"image":{"@id":"https:\/\/www.neuraldesigner.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/NeuralDesigner","https:\/\/es.linkedin.com\/showcase\/neuraldesigner\/"]}]}},"_links":{"self":[{"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/blog\/3368","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/blog"}],"about":[{"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/types\/blog"}],"author":[{"embeddable":true,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/users\/20"}],"version-history":[{"count":14,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/blog\/3368\/revisions"}],"predecessor-version":[{"id":23789,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/blog\/3368\/revisions\/23789"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/media\/2664"}],"wp:attachment":[{"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/media?parent=3368"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/categories?post=3368"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/tags?post=3368"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}