Ranger¶
- class pytorch_forecasting.optim.Ranger(params: Union[Iterable[torch.Tensor], Iterable[dict]], lr: float = 0.001, alpha: float = 0.5, k: int = 6, N_sma_threshhold: int = 5, betas: Tuple[float, float] = (0.95, 0.999), eps: float = 1e-05, weight_decay: float = 0)[source]¶
Bases:
torch.optim.optimizer.Optimizer
Implements Ranger optimization algorithm (Lookahead with RAdam).
Implementation is modified version from
pytorch-ranger
package which build upon its original implementation. Ranger seems to be benefiting most models.- Parameters
params – iterable of parameters to optimize or dicts defining parameter groups
lr – learning rate (default: 1e-3)
alpha – linear interpolation factor. 1.0 recovers the inner optimizer. (default: 0.5)
k – number of lookahead steps (default: 6)
N_sma_threshhold – Maximum length of the simple moving average (SMA)
betas – coefficients used for computing running averages of gradient and its square (default: (0.95, 0))
eps – term added to the denominator to improve numerical stability (default: 1e-8)
weight_decay – weight decay (L2 penalty) (default: 0)
Example
>>> from pytorch_forecasting.optim import Ranger >>> optimizer = Ranger(model.parameters(), lr=0.1) >>> optimizer.zero_grad() >>> loss_fn(model(input), target).backward() >>> scheduler = StepLR(optimizer, step_size=1, gamma=0.7) >>> optimizer.step() >>> scheduler.step()
Methods
step
([closure])Performs a single optimization step.