14. WORKLOAD FORECASTING › 14.2 Usage Guidelines › 14.2.3 Evaluating the Model

14.2.3 Evaluating the Model

The second step in developing a Workload Forecasting Analysis
model is evaluating how well the data fits the model.  For
this purpose, you can use the model summary report produced
after the training phase.  The statistics are the ANOVA
R-squared, Average Model Error and the Average Model RMS
Error.

The statistics produced by the model are interpreted as
follows:

     o  ANOVA R-squared is the coefficient of determination
        and termed the "goodness of fit".  This value
        indicates what percent of the variability in the
        historical (training) data is explained by the
        proposed model.  You would typically be seeking
        r-squared values greater than or equal to 0.70.

     o  Average Model Error is the mean value of the
        differences between the predicted and actual data
        values occurring within the Forecast data and
        observed by the model, expressed as a ratio of the
        Forecast data values. A value of less than .10 should
        be considered acceptable.

     o  Average RMS Error is the mean value of squared
        differences between the predicted and actual data
        values occurring within the Forecast data and
        observed by the model.  This value is expressed in
        the same units as the Forecast element value.

As with statistical modeling methods, the analyst should
review the Neugents technology modeling output statistics to
determine how effective the model trained and how accurately
it represents the historical data values before proceeding
with any forecasting operations.

The following steps should be taken to ensure model accuracy
and provide confidence in modeled results.


Step 1 - Examine the ANOVA R-squared, Model Error and RMS
         Model Error statistics.

      We recommend an ANOVA R-squared value greater than
      0.70.  This value is an estimate of how many of the
      actual Forecast element values were accurately
      predicted during the training phase. Thus, an ANOVA
      value of .70 indicates that 70 percent of the Forecast
      element values were accurately predicted by the model.
      A value less than .70 would indicate that the input
      data for the training phase did not properly define the
      forecasting problem and the modeling data needs to be
      revised.  A well defined model should produce an ANOVA
      R-squared value of at least .85, with higher values
      typically observed.

      The Model Error calculated by the Workload Forecasting
      application is expressed as a ratio, and as such should
      be less than .10. Many models can produce model error
      measurements of less than .05, so an error value of
      more than .10 would likely indicate that improper
      choices were made when selecting training elements and
      that the model requires revision.

      The RMS Error is the square root of the mean of the
      squared actual model errors. While no specific
      recommendation is currently available, the value here
      should be "small" with respect to the Forecast element
      values and should track the average model error.

      The ANOVA R-squared, Model Error and RMS Error values
      are printed on the Model Error Summary Report.


Step 2 - Visually examine the data.

      Visual examination of the actual and predicted data is
      very important and can sometimes show that a model
      whose ANOVA R-square, or other statistics look bad may
      be significantly improved by the exclusion of a small
      number of outliers from the data.  While models built
      using Neugents technology are generally less affected
      by "outliers" and other "noise" in the input data, they
      are not completely immune, and the Workload Forecasting
      Analysis application generates both descriptive
      statistics as well as reports and graphs to help
      evaluate model performance and to identify problem data
      values.


Step 3 - Apply the common sense test.

      The common sense test asks, "Does this model make any
      sense?"  To execute and depend on various forecasting
      models you should understand the data well. For
      example, you can load up a year's worth of resource
      usage data, construct and train a model, check the
      statistics and decide the model is valid and can be
      used for forecasting, when actually it is not valid.
      If you are using weekly data for planning purposes and
      you are a bank that does a lot of credit card
      processing, examining the data visualization graphs and
      reports might erroneously cause you to decide the
      fourth week of November is an outlier. In fact, it is
      probably the single most important week of the year.
      The reason is that Thanksgiving week and the Friday
      after Thanksgiving is typically the single largest
      shopping day of the year, with credit cards melting
      down all over the country.  The obvious motto is: know
      your business and know your data.

      Any forecasting technique needs validation, and
      Neugents technology is no different.  Execute your
      models and save the forecasts.  Once the new planning
      data is available, compare your previous forecasts to
      the actual planning data. Are you tracking well?  If
      not, investigate and find out why.  You may need to
      include more workloads into your model, or switch to
      the MANUAL option and "grow" your workloads more
      aggressively.  Or you may need to get creative and
      define a special workload that represents planned new
      business and use this "dummy" workload to document the
      need for additional capacity.  A good deal of an
      analysts work is in understanding the business and
      ensuring that his plans properly address these needs.
      Tools such as Neugents technology can help create
      better forecasts, but no tool can substitute for the
      knowledge the analyst possesses.