Ranger

class pytorch_forecasting.optim.Ranger(params: Union[Iterable[torch.Tensor], Iterable[dict]], lr: float = 0.001, alpha: float = 0.5, k: int = 6, N_sma_threshhold: int = 5, betas: Tuple[float, float] = (0.95, 0.999), eps: float = 1e-05, weight_decay: float = 0)[source]

Bases: torch.optim.optimizer.Optimizer

Implements Ranger optimization algorithm (Lookahead with RAdam).

Implementation is modified version from pytorch-ranger package which build upon its original implementation. Ranger seems to be benefiting most models.

Parameters

  • params – iterable of parameters to optimize or dicts defining parameter groups

  • lr – learning rate (default: 1e-3)

  • alpha – linear interpolation factor. 1.0 recovers the inner optimizer. (default: 0.5)

  • k – number of lookahead steps (default: 6)

  • N_sma_threshhold – Maximum length of the simple moving average (SMA)

  • betas – coefficients used for computing running averages of gradient and its square (default: (0.95, 0))

  • eps – term added to the denominator to improve numerical stability (default: 1e-8)

  • weight_decay – weight decay (L2 penalty) (default: 0)

Example

>>> from pytorch_forecasting.optim import Ranger
>>> optimizer =  Ranger(model.parameters(), lr=0.1)
>>> optimizer.zero_grad()
>>> loss_fn(model(input), target).backward()
>>> scheduler = StepLR(optimizer, step_size=1, gamma=0.7)
>>> optimizer.step()
>>> scheduler.step()

Methods

step([closure])

Performs a single optimization step.
step(closure: Optional[Callable[], float]] = None) → Optional[float][source]

Performs a single optimization step. :param closure: A closure that reevaluates the model and returns the loss.

optim utils