8.5 Forecast evaluation

We can compute the forecast performance metrics as usual.

fit.ets <- forecast::ets(traindat, model="AAA")
fr <- forecast::forecast(fit.ets, h=12)

Look at the forecast so you know what years and months to include in your test data. Pull those 12 months out of your data using the window() function.

testdat <- window(chinookts, c(1996,1), c(1996,12))

Use accuracy() to get the forecast error metrics.

forecast::accuracy(fr, testdat)

##                         ME      RMSE       MAE       MPE     MAPE      MASE
## Training set -0.0001825075 0.5642326 0.4440532 -9.254074 25.40106 0.7364593
## Test set      0.3143200919 0.7518660 0.6077172 65.753096 81.38568 1.0078949
##                    ACF1 Theil's U
## Training set 0.07490341        NA
## Test set     0.05504107 0.4178409

We can do the same for the ARIMA model.

fit <- forecast::auto.arima(traindat)
fr <- forecast::forecast(fit, h=12)
forecast::accuracy(fr, testdat)

##                      ME      RMSE       MAE       MPE     MAPE      MASE
## Training set 0.01076412 0.5643352 0.3966735 -1.219729 26.91589 0.6578803
## Test set     0.79665978 0.9180939 0.7966598 19.587692 53.48599 1.3212549
##                     ACF1 Theil's U
## Training set -0.05991122        NA
## Test set     -0.12306276 0.5993699