5.6 Models fit

Now we can use names() to see the models that we have fit. If we want to add more, we use the code above as a template.

modelnames <- names(fit.list)
modelnames
## [1] "ETS w trend"          "ETS no trend"         "ARIMA(0,1,1) w drift"
## [4] "ARIMA(2,1,0) w drift" "TV linear regression" "Naive"               
## [7] "Naive w trend"        "Average"

5.6.1 Metrics for each model

We will run the models and compute the forecast metrics for each and put in a table.

restab <- data.frame(model=modelnames, RMSE=NA, ME=NA, tsCV.RMSE=NA, AIC=NA, BIC=NA, stringsAsFactors = FALSE)
for(i in modelnames){
  fit <- fit.list[[i]]
  fr <- fr.list[[i]]
  restab$RMSE[restab$model==i] <- accuracy(fr, testdat)["Test set","RMSE"]
  restab$ME[restab$model==i] <- accuracy(fr, testdat)["Test set","ME"]
  e <- tsCV(traindat, fun.list[[i]], h=1)
  restab$tsCV.RMSE[restab$model==i] <- sqrt(mean(e^2, na.rm=TRUE))
  restab$AIC[restab$model==i] <- AIC(fit)
  restab$BIC[restab$model==i] <- BIC(fit)
}

Add on \(\Delta\)AIC and \(\Delta\)BIC. Sort by \(\Delta\)AIC and format to have 3 digits.

restab$DeltaAIC <- restab$AIC-min(restab$AIC)
restab$DeltaBIC <- restab$BIC-min(restab$BIC)
restab <- restab[order(restab$DeltaAIC),]
resfor <- format(restab, digits=3, trim=TRUE)

Bold the minimum values in each column so they are easy to spot.

for(i in colnames(resfor)){
  if(class(restab[,i])=="character") next
  if(i!="ME") testval <- restab[,i] else testval <- abs(restab[,i])
  theminrow <- which(testval==min(testval))
  resfor[theminrow, i] <- paste0("**",resfor[theminrow,i],"**")
}

This is the table of FORECAST performance metrics. Not how well it fits the data, but how well it forecasts out of the data. RSME and ME are for the 2 data points in 1988 and 1989 that were held out for testing. tsCV.RMSE is the RSME for the time-series crossvalidation that makes a series of forecasts for each point in the data. AIC and BIC are information criteria, which are a measure of data support for each model.

knitr::kable(resfor)
model RMSE ME tsCV.RMSE AIC BIC DeltaAIC DeltaBIC
5 TV linear regression 0.195 -0.114 0.247 -7.00 -3.4611 0.00000 0.123
3 ARIMA(0,1,1) w drift 0.370 -0.332 0.231 -6.99 -3.5844 0.00443 0.000
4 ARIMA(2,1,0) w drift 0.381 -0.347 0.224 -6.08 -1.5399 0.91340 2.044
7 Naive w trend 0.510 -0.483 0.239 -2.18 0.0883 4.81255 3.673
6 Naive 0.406 -0.384 0.222 -2.06 -0.9247 4.93505 2.660
1 ETS w trend 0.538 -0.500 0.251 3.67 9.5587 10.66374 13.143
2 ETS no trend 0.317 -0.289 0.222 6.76 10.2988 13.75990 13.883
8 Average 0.656 0.643 0.476 33.04 35.3924 40.03162 38.977