5.6 Models fit
Now we can use names() to see the models that we have fit. If we want to add more, we use the code above as a template.
## [1] "ETS w trend" "ETS no trend" "ARIMA(0,1,1) w drift"
## [4] "ARIMA(2,1,0) w drift" "TV linear regression" "Naive"
## [7] "Naive w trend" "Average"5.6.1 Metrics for each model
We will run the models and compute the forecast metrics for each and put in a table.
restab <- data.frame(model=modelnames, RMSE=NA, ME=NA, tsCV.RMSE=NA, AIC=NA, BIC=NA, stringsAsFactors = FALSE)
for(i in modelnames){
fit <- fit.list[[i]]
fr <- fr.list[[i]]
restab$RMSE[restab$model==i] <- accuracy(fr, testdat)["Test set","RMSE"]
restab$ME[restab$model==i] <- accuracy(fr, testdat)["Test set","ME"]
e <- tsCV(traindat, fun.list[[i]], h=1)
restab$tsCV.RMSE[restab$model==i] <- sqrt(mean(e^2, na.rm=TRUE))
restab$AIC[restab$model==i] <- AIC(fit)
restab$BIC[restab$model==i] <- BIC(fit)
}Add on \(\Delta\)AIC and \(\Delta\)BIC. Sort by \(\Delta\)AIC and format to have 3 digits.
restab$DeltaAIC <- restab$AIC-min(restab$AIC)
restab$DeltaBIC <- restab$BIC-min(restab$BIC)
restab <- restab[order(restab$DeltaAIC),]
resfor <- format(restab, digits=3, trim=TRUE)Bold the minimum values in each column so they are easy to spot.
for(i in colnames(resfor)){
if(class(restab[,i])=="character") next
if(i!="ME") testval <- restab[,i] else testval <- abs(restab[,i])
theminrow <- which(testval==min(testval))
resfor[theminrow, i] <- paste0("**",resfor[theminrow,i],"**")
}This is the table of FORECAST performance metrics. Not how well it fits the data, but how well it forecasts out of the data. RSME and ME are for the 2 data points in 1988 and 1989 that were held out for testing. tsCV.RMSE is the RSME for the time-series crossvalidation that makes a series of forecasts for each point in the data. AIC and BIC are information criteria, which are a measure of data support for each model.
| model | RMSE | ME | tsCV.RMSE | AIC | BIC | DeltaAIC | DeltaBIC | |
|---|---|---|---|---|---|---|---|---|
| 5 | TV linear regression | 0.195 | -0.114 | 0.247 | -7.00 | -3.4611 | 0.00000 | 0.123 |
| 3 | ARIMA(0,1,1) w drift | 0.370 | -0.332 | 0.231 | -6.99 | -3.5844 | 0.00443 | 0.000 |
| 4 | ARIMA(2,1,0) w drift | 0.381 | -0.347 | 0.224 | -6.08 | -1.5399 | 0.91340 | 2.044 |
| 7 | Naive w trend | 0.510 | -0.483 | 0.239 | -2.18 | 0.0883 | 4.81255 | 3.673 |
| 6 | Naive | 0.406 | -0.384 | 0.222 | -2.06 | -0.9247 | 4.93505 | 2.660 |
| 1 | ETS w trend | 0.538 | -0.500 | 0.251 | 3.67 | 9.5587 | 10.66374 | 13.143 |
| 2 | ETS no trend | 0.317 | -0.289 | 0.222 | 6.76 | 10.2988 | 13.75990 | 13.883 |
| 8 | Average | 0.656 | 0.643 | 0.476 | 33.04 | 35.3924 | 40.03162 | 38.977 |