5.2 Testing multiple models
For each model, you will do the same steps:
Fit the model
Create forecasts for a test data set or use cross-validation
Compute forecast accuracy metrics for the forecasts
Note when you compare models, you can use both ‘training data/test data’ and use time-series cross-validation, but report the metrics in separate columns. Example, ‘RMSE from tsCV’ and ‘RMSE from test data’.
5.2.1 Example candidate model set for anchovy
- Exponential smoothing model with trend
fit <- forecast::ets(traindat, model="AAN")
fr <- forecast::forecast(fit, h=1)
- Exponential smoothing model no trend
fit <- forecast::ets(traindat, model="ANN")
fr <- forecast::forecast(fit, h=1)
- ARIMA(0,1,1) with drift (best)
fit <- forecast::Arima(traindat, order(0,1,1), include.drift=TRUE)
fr <- forecast::forecast(fit, h=1)
- ARIMA(2,1,0) with drift (within 2 AIC of best)
fit <- forecast::Arima(traindat, order(2,1,0), include.drift=TRUE)
fr <- forecast::forecast(fr)
- Time-varying regression with linear time
traindat$t <- 1:24
fit <- lm(log.metric.tons ~ t, data=traindat)
fr <- forecast::forecast(fit, newdata=data.frame(t=25))
Null models
- Naive no trend
fit <- forecast::Arima(traindat, order(0,1,0))
fr <- forecast::forecast(fit, h=1)
# or simply
fr <- forecast::rwf(traindat)
- Naive with trend
fit <- forecast::Arima(traindat, order(0,1,0), include.drift=TRUE)
fr <- forecast::forecast(fit)
# or simply
fr <- forecast::rwf(traindat, drift=TRUE)
- Average or mean
fit <- forecast::Arima(traindat, order(0,0,0))
fr <- forecast::forecast(fit)