Metrics

Try the API v2 pre-release!

A New API version 2 is in development. Try it out before release: v2 API Reference

Note: that metrics are SAME for both the versions.

Caution: v2 is WIP and unstable. Not yet production-ready.

Multiple metrics have been implemented to ease adaptation.

In particular, these metrics can be applied to the multi-horizon forecasting problem, i.e. can take tensors that are not only of shape n_samples but also n_samples x prediction_horizon or even n_samples x prediction_horizon x n_outputs, where n_outputs could be the number of forecasted quantiles.

Metrics can be easily combined by addition, e.g.

from pytorch_forecasting.metrics import SMAPE, MAE

composite_metric = SMAPE() + 1e-4 * MAE()

Such composite metrics are useful when training because they can reduce outliers in other metrics. In the example, SMAPE is mostly optimized, while large outliers in MAE are avoided.

Further, one can modify a loss metric to reduce a mean prediction bias, i.e. ensure that predictions add up. For example:

from pytorch_forecasting.metrics import MAE, AggregationMetric

composite_metric = MAE() + AggregationMetric(metric=MAE())

Here we add to MAE an additional loss. This additional loss is the MAE calculated on the mean predictions and actuals. We can also use other metrics such as SMAPE to ensure aggregated results are unbiased in that metric. One important point to keep in mind is that this metric is calculated across samples, i.e. it will vary depending on the batch size. In particular, errors tend to average out with increased batch sizes.

Details

See the API documentation for further details on available metrics:

metrics.quantile.QuantileLoss([quantiles])	Quantile loss, i.e. a quantile of q=0.5 will give half of the mean absolute error as it is calculated as.
metrics.point.CrossEntropy([reduction])	Cross entropy loss for classification.
metrics.point.PoissonLoss([reduction])	Poisson loss for count data.
metrics.point.SMAPE([reduction])	Symmetric mean absolute percentage.
metrics.point.MAPE([reduction])	Mean absolute percentage.
metrics.point.MAE([reduction])	Mean average absolute error.
metrics.point.RMSE([reduction])	Root mean square error.
metrics.point.MASE([reduction])	Mean absolute scaled error
metrics.point.TweedieLoss([reduction, p])	Tweedie loss.
metrics.distributions.NormalDistributionLoss([...])	Normal distribution loss.
metrics.distributions.MultivariateNormalDistributionLoss([...])	Multivariate low-rank normal distribution loss.
metrics.distributions.NegativeBinomialDistributionLoss([...])	Negative binomial loss, e.g. for count data.
metrics.distributions.LogNormalDistributionLoss([...])	Log-normal loss.
metrics.distributions.BetaDistributionLoss([...])	Beta distribution loss for unit interval data.
metrics.distributions.MQF2DistributionLoss(...)	Multivariate quantile loss based on the article Multivariate Quantile Function Forecaster.
metrics.distributions.ImplicitQuantileNetworkDistributionLoss([...])	Implicit Quantile Network Distribution Loss.