7. UNIVARIATE MODEL FORECASTING › 7.2 Usage Guidelines

7.2 Usage Guidelines

This section discusses three methods for evaluating the fit
of regression models:

o  The coefficient of determination.  This value, which is
   commonly called "r-squared," indicates the percent of the
   variability in the historical data that is explained by
   the model.

o  The F value for testing that all of the parameters of the
   model are zero except for the intercept.

o  The development of confidence limits for the model.
   Confidence limits which are based on the standard error
   for the model allow you to determine confidence bounds
   (that is, upper and lower limits) for the forecasts that
   are produced with a model.

Each of these methods is based on statistical tests of
certainty using the regression model.  A tutorial on
regression models is presented in Section 7.4.  If you are
not familiar with regression models, we recommend that you
review the tutorial in conjunction with this section on usage
guidelines.

COEFFICIENT OF DETERMINATION

Perhaps the best technique for evaluating the fit of a
proposed model is the value of r-squared.  This parameter,
called the coefficient of determination, is calculated using
the following equation:
                                     _
              b * SY + m * SXY - n * Y ** 2
       2
      r  =  ---------------------------------         (Eqn 1)
                               _
                     SXY - n * Y ** 2

Basically, r-squared indicates the percentage of the
variability in the historical data that is explained by the
model (SAS82).  If the model is evaluated and the resulting
r-squared is 0.98, the proposed model explains 98% of the
variability of the historical data.  For most forecasting
applications, an r-squared value of 0.70 or greater is more
than suitable for evaluating models (SAR79).

NULL HYPOTHESIS

The second method for evaluating models is to test the "null
hypothesis" using the F value for the model.  The null
hypothesis states that the historical data is as well
represented by the mean value (that is, the Y intercept) as
by the proposed model.  The F value is computed using the
ratio of the sum of the squares values calculated for the
model. (Details of this calculation are beyond the scope of
this tutorial.)  You may compare the F value to standard
statistical tables to determine the probability that the
proposed model would have arisen under the null hypothesis.
This probability is given by the value of p.

A "good model" is indicated by a large F value and a very
small p probability.  Although most statistics texts
recommend testing the null hypothesis to a level of 0.001, a
value of 0.01, investigators studying short series of
computer measurement data (SAR79) suggest a value of 0.01. An
F statistic can be misleading when there are too few degrees
of freedom (that is, less than 15 or 20).  Therefore, the F
statistic should not be your primary consideration in
evaluating models developed for short series of historical
observations.

CALCULATION OF CONFIDENCE LIMITS

The third method for evaluating regression models is the
calculation of confidence limits.  If the proposed linear
model does not fit the historical data perfectly, you can
expect that errors will also be present in the values
predicted by the model.  You can bound these errors by
confidence limits.  You can determine the average confidence
limit for a regression model using the standard error value
calculated for the model.  Equation 2 shows the equation for
calculating the standard error.

                _                          _
               |                            | ** 1/2
               |   SSY - b * SY - m * SXY   |
          s  = |  ------------------------  |         (Eqn 2)
           e   |            n-2             |
               |_                          _|


By solving the equation, you can determine the standard
error, and using this value, you can estimate the confidence
limits for the forecast.  For example, if a 99% confidence
limit is calculated for a forecast, then you can be 99% sure
that all future values will fall between the upper and lower
confidence limits.

For large samples (that is, n greater than 30), a 68%
confidence limit is approximated by plus or minus one times
the standard error; a 95.5% confidence limit is approximated
by two times the standard error; and a 99.7% confidence limit
(that is, 99.7% of actual future values should fall within
plus or minus three standard errors of their predicted value)
is approximated by three times the standard error.  For
smaller samples, the number of standard errors required to
approximate the confidence limits is determined using a "t
statistic".  (The details of this calculation are beyond the
scope of this tutorial.)  Note that the width of the
confidence interval (that is, the difference between the
exact upper and lower confidence limits) increases with the
length of the forecast.  That is, the width of the confidence
limit increases monotonically with the length of the
forecast.

The SAS REG procedure calculates an exact 95% confidence
limit for each point in the historical data series and for
each point estimated by the model (SAS81).  These
calculations are based on the standard error and are similar
in nature to the equations presented in this section.  (The
details of these calculations are beyond the scope of this
tutorial.)  Confidence limits allow you to evaluate the range
of potential errors that may exist in the forecasts produced
by the program.