A parameter that controls the magnitude of updates or movements in iterative processes and optimization algorithms.

Step Size

Step size, also known as the learning rate in some contexts, is a fundamental parameter that determines how far to move in a given direction during an iterative process. This concept plays a crucial role in various optimization techniques and algorithmic convergence.

Core Principles

The selection of an appropriate step size involves balancing two competing factors:

Speed of convergence
Stability of the process

Too large a step size can cause:

Overshooting the target
Oscillation around the optimal point
Divergence in extreme cases

Too small a step size can result in:

Slow convergence
Getting stuck in local minima
Computational inefficiency

Applications

Gradient Descent

In gradient descent algorithms, step size (η) determines how much to adjust parameters in response to the computed gradient:

θ_new = θ_old - η ∇f(θ)

Numerical Integration

When solving differential equations numerically, step size determines the granularity of the approximation in methods like:

Euler's method
Runge-Kutta methods
Adams-Bashforth methods

Adaptive Step Sizes

Modern algorithms often employ adaptive step sizes that automatically adjust based on:

Progress toward optimization
Local landscape characteristics
Convergence rate

Common adaptive methods include:

Adam optimizer
AdaGrad
RMSprop

Practical Considerations

When choosing a step size, practitioners should consider:

Problem characteristics
Algorithm stability requirements
Computational resources
Desired accuracy
Time constraints

Historical Development

The concept of step size emerged from:

Classical numerical analysis
Early optimization theory
Development of computer-based iterative methods

Understanding proper step size selection remains crucial in modern applications from machine learning to scientific computing.