I am currently exploring different probabilistic time series forecasting models for car sales data and have planned to evaluate the probabilistic forecasts with the metrics rho-risk as described on page 1189 in Salinas et al. (2020). The formula of the evaluation metrics can be seen below.
Formula for rho risk from Salinas et al. (2020)
In general, I find that the values derived from rho risk make sense and somewhat logically describe a model's ability to make accurate probabilistic forecasts. However, in certain cases it is harder to understand why the rho risk values seem really good, even though one can visually see that the model's performance is bad.
I will describe my confusion by showing the results of two prediction models (Exponential Smoothing and NBEATS):
Results from Exponential Smoothing
The predictions from the exponential smoothing model give rather wide prediction intervals, but simultaneously almost always captures the actual values inside the prediction intervals. On the other hand, the predictions from the NBEATS models give tight prediction intervals that are piece-wise very accurate according to the actual values, but piece-wise the prediction intervals are very off from the actual values. Still, the rho risk values suggest that the NBEATS model would provide better probabilistic forecasts with respect to the 0.1, 0.25, 0.5, 0.75, 0.9 quantiles.
I am rather new at working with probabilistic forecasts and would be very thankful for any advice on how to explain these strange results.
Could also be worth mentioning that I am using the Python package DARTS for the entire modelling and evaluation.