Agencies are increasingly employing forecasting models in their strategic planning efforts. Analytic tools such as regression models can be used to forecast future revenues from taxation & licensing, demand for public services & infrastructure, and agency staffing needs with reasonable accuracy. However, there are a number of common pitfalls that undermine the usefulness of forecasting models as planning tools.
Here are a few to avoid:
1. Don’t try to model everything—keep it simple!
Put simply, forecasting models rely on a number of “input variables” to predict outcomes, and as the number of variables increase, so too does the complexity of the forecast and the risk for errors. A forecasting model attempts to predict the value of an outcome variable from the values of one or more input variables. As decision makers, we have to accept that no model will be able to account for all the variation in our outcome variable, no matter how much data we collect and how many input variables we add to the model. The goal should not be to precisely predict some future outcome; instead, the goal should be to produce a forecast that informs an organization’s strategic planning efforts with reasonable accuracy.
For example, we might be interested in estimating the number of individuals applying for a business license in the next quarter. To develop a reliable model, we would build a few different model specifications, accounting for different variables. We would then apply these competing models to historical data and evaluate them based on how well each captures the variation that we observe in historical business license volume data. The model with results that are most closely aligned with observed trends can be reasonably relied upon. Attempting to include every possible variable—a kitchen sink model—is counterproductive as it can introduce a variety of modeling errors that render the results meaningless. Once a reasonably accurate model is developed, resources should be focused on interpreting the results.
2. Forecast ranges, not single points
The actual forecasting and modeling in most organizations is performed by staff with technical expertise and then presented to decision makers who determine how best to act on that information. In an effort to distill a large volume of complex information into something actionable, decision makers are often presented with a single forecasting number—the dollar value of collections from tax or licensing activities, ridership for public transit services, or the number of inbound customer calls.
In addition to predicting a single point, forecasting models also provide a measure of uncertainty—taken together, these two pieces of information can be used to construct a range of possible values known as a prediction interval. A prediction interval says that for a given level of confidence, X, as chosen by the modeler, the interval is likely to contain the actual value of the parameter we are forecasting. More simply, a prediction interval allows for statements such as the following:
“The model estimates we will receive 1,250 customer calls in the next month. Based on this forecast, we are 90 percent confident that the number of customer calls next month will range between 850 and 1,650.”
Prediction intervals quantify the range of likely outcomes, creating scenarios that aid in strategic planning efforts. Armed with a prediction range, decision makers responsible for staffing a call center can develop a staffing plan that both meets the predicted volume and has contingencies for when call volumes are higher or lower than expected.
3. Understand how forecast reliability changes over time
Forecasting models use historical data to predict future events, and in doing so assume that the underlying relationships between inputs and outputs remains unchanged over time. Both history and experience teach us otherwise, and we have seen events like the Great Recession fundamentally alter the human behavior that drives much of what we attempt to forecast.
The confidence we place in forecasts should decline as we project further out from known data. Projections about short- and medium-term events are likely to be based on actual observed data or on projections of model inputs based on recent experience. Forecasts of next years’ sales volumes, for example, are likely to be made using the previous year’s sales numbers and a forecast of economic activity for the next year. In order to forecast sales tax revenues twenty years from today, however, we have to make a host of assumptions and predictions about forecasting model inputs—effectively, we are forecasting model inputs in order to forecast model outputs.
When evaluating forecasting models, decision makers should treat short- and medium-term forecasts as predictions: likely outcomes given the current state of the world. Long-term forecasts should be treated as projections: possible outcomes given the current trajectory of the organization. As time progresses, projections should be updated and compared to actual performance; in doing so, previous long-term projections become short- and medium-term predictions.
4. Iterate on the process
At the end of each forecasting cycle, technical staff and key decision makers should take time to evaluate forecasted versus actual volumes. When forecasts diverge significantly from actual experience, it is important to determine the possible causes. The forecasting model may have been misspecified, failing to capture the key factors that account for the variation in the outcome variable. The data used in forecasting could have been transcribed or entered incorrectly, causing the model to produce inaccurate estimates. Forecasts can also diverge from actual experience due to real-world events that no forecasting model can anticipate.
Organizations that fail iterate on the forecasting process run the risk of propagating future forecasts with data and modeling errors, leading to a significant divergence between expectations and actual experiences and undermining the usefulness of forecasting as a planning tool.
