How Macroeconomic Forecasters Adjusted During the COVID-19 Pandemic

The COVID-19 pandemic has posed substantial challenges for macroeconomic forecasting. In the absence of a recent directly comparable episode, forecasters have modified their models or sought additional information in data. We survey these forecasting approaches and highlight the importance of transparency and flexibility of assumptions. With the benefit of data over the course of the pandemic, we can now see how different assumptions led to forecast errors that might have been predictable at the time of the forecast.

Macroeconomic forecasting during the COVID-19 pandemic has been especially challenging. The pandemic was a once-in-a-century health crisis that generated an unprecedented impact on the economy. For instance, GDP contracted by an annualized rate of 33 percent in the second quarter of 2020, more than four times as much as any quarterly drop since 1948.

Furthermore, we do not have recent examples of recessions driven by public health crises, so forecasters had no directly comparable past data and experience that they could draw from. As a result, not only did point forecasts become more uncertain, but quantifying this uncertainty was unusually difficult.

How Forecasters Adjusted to the New Environment

Faced with unique circumstances, forecasters had to acknowledge the difference without completely ignoring lessons from previous business cycles:

• Should one view the COVID-19 pandemic as simply a period of high volatility?
• Would economic variables comove differently than previous recessions?
• Would the effects of the pandemic propagate and persist as other drivers of business cycles do?

These questions mattered for forecasts but did not have precise answers with the scarce amount of data available, especially in the early stages of the pandemic. How could forecasters tackle such questions as the pandemic unfolded, acknowledge the level of confidence in their answers and express how these questions influence their point forecasts and associated uncertainty?

In a recent working paper, "Forecasting in the Absence of Precedent," I discuss two broad approaches to dealing with the lack of precedent. First, forecasters used subjective judgment or prior knowledge — typically from economic theory — to adapt their models. Such model adjustments are most fruitful when their underlying assumptions are transparent and acknowledge the lack of certainty.

Alternatively, forecasters found new sources of information, typically by incorporating new data into forecasting models. For example, epidemiological and high-frequency data were of special interest during the pandemic. However, forecasters need to know how these new variables comove with variables of interest, which once more raises the question of model specification and the choice of assumptions. This Economic Brief focuses on several representative papers for each approach.

Adapting Models During Periods of High Uncertainty

When forecasters observe large swings in the economy, they need to translate their interpretations of the data into model assumptions. While it is impossible to perfectly model the economy, carefully crafted assumptions can allow model forecasts to be useful even during unusual episodes such as the pandemic.

First, assumptions should be easy to communicate so audiences can put forecasts into proper context. Specifically, even those who disagree with the model's assumptions can infer how those assumptions might influence the forecast, allowing them to learn from the model despite their disagreement. During the COVID-19 pandemic, such transparency has been especially important given the high level of disagreement among economists and policymakers.

Second, where possible, assumptions should be imposed based on probability. Rather than insist that a feature of the economy is "definitely true," one can model that feature as being "probably true." Uncertainty is especially relevant during an event in which the structure of the economy is less certain. Acknowledging the uncertainty in the model translates to forecast error bands that more accurately express the forecaster's own level of confidence.

One view of the large economic fluctuations during the pandemic is that they arose from a sequence of large disturbances to the economy. This was featured in the 2020 working paper "How to Estimate a VAR after March 2020" by Michele Lenza and Giorgio E. Primiceri. This approach has several strengths:

• First, it is easy to communicate, allowing the forecasts to be informative even if one did not fully agree with the model.
• Second, it widened the forecast error bands during the early stages of the pandemic, which better reflected the forecast uncertainty at the time.

A limitation of the approach is that it keeps the rest of the structure of the economy — that is, the type of shocks and propagation of these shocks — unchanged, an assumption that was questionable even in the initial stages of the pandemic.

In particular, there was discussion of a swifter recovery and of different sectors in the economy being hit as compared to past recessions. Indeed, the results from the paper suggest that the model was unable to predict the relatively rapid decline in unemployment after the start of the pandemic.

As an alternative, the 2020 working paper "Macroeconomic Forecasting in the Time of COVID-19" by Primiceri and Andrea Tambalotti acknowledges the uniqueness of the pandemic-driven recession by introducing a new shock to the model.

However, it makes strong assumptions about the behavior of that shock resembling previous drivers of business cycles. As a result, this paper also fails to predict the rapid decline in unemployment after the initial sharp rise in March and April 2020. A more flexible structure that acknowledges the uncertainty about the COVID-19 shock would potentially have improved forecasts.

Additional Information in New and Old Data Sources

While the methods above focused on modifying the model to capture changes in the economy, there were also attempts to find additional information in new or existing data sources. These attempts continue to rely on well-designed models to avoid bias or a false sense of precision in forecasts.

One source that gained renewed attention during the pandemic was high-frequency data, which provided updated snapshots of the economy before quarterly data were released. In their working paper "Real-Time Forecasting with a (Standard) Mixed-Frequency VAR During a Pandemic," Frank Schorfheide and Dongho Song formally incorporate these data into their forecasts.

They have mixed success predicting second quarter 2020 values for quarterly variables, even though they use monthly data available on June 30, 2020.

While their forecast for GDP performed well (median forecast of 12 percent below the fourth quarter 2019 level, compared to an actual decline of 10 percent), their model predicted a decline in investment of about double what was realized (median forecast of 22 percent below the fourth quarter 2019 level, compared to an actual decline of 9 percent).

Given the expectation that the COVID-19 recession could be shorter than typical recessions, economic theory would have suggested that investment should have fallen less than usual. Incorporating this knowledge into the model could have improved forecasts.

Instead of assuming that the economy behaved normally during the pandemic, a paper by Claudia Foroni, Massimiliano Marcellino and Dalibor Stevanovic, "Forecasting the COVID-19 Recession and Recovery: Lessons from the Financial Crisis," aligns forecasts with how data behaved during the Great Recession. They postulate that the behavior of economy during the COVID-19 recession looks more like the Great Recession than normal times. In particular, they modify the model parameter estimates by giving greater weight to Great Recession observations and adjust the model forecasts based on forecast errors made by the model during the Great Recession.

The paper predicts a decline in investment that is relatively close to the data (forecast of an initial annualized decline of 25 percent, compared to an actual decline of 27 percent) but substantially underpredicts the decline in GDP (forecast of an annualized decline of 10 percent, compared to an actual decline of 33 percent). As argued above, the overprediction of the decline in investment relative to GDP could have been inferred by economic theory.

Looking Back at the Validity of Model Assumptions

Having discussed how various forecasting methods performed, it is useful to look back now that we have observed more months of data. The 2021 paper "Modeling Macroeconomic Variations After COVID-19" by Serena Ng conducts such an exercise using data through December 2020, providing a retrospective view of how various assumptions in the forecasting models held up.

She first estimates her model using macroeconomic and financial data through February 2020. She then extends the data series to December 2020 and incorporates data on COVID-19 cases and hospitalizations. Comparing the two estimation exercises provides insight into what changed in the economy during the pandemic.

After accounting for COVID-19 data, Ng finds that the comovement across variables did not change substantially and volatility in the economy reached approximately the same peaks seen in 1973-74, 1981-82 and 2007-09. In other words, the economy appeared to have been hit by a single shock that could be captured by the COVID-19 data, while the typical drivers of the economy behaved similar to how they had in the past, with variances matching previous large recessions.

These results contrast with the assumption in Lenza and Primiceri's paper that there was an unprecedented increase in volatility. Their estimated levels of volatility are arguably a way for the model to capture the unique COVID-19 shock, rather than a reflection of existing shocks simultaneously becoming more volatile.

In addition, the longer time series allows Ng to observe a proxy for the new COVID-19 shock proposed by Primiceri and Tambalotti and to estimate its impact on macroeconomic variables. Finally, the COVID-19 data captures the differences in the pandemic era structure of the economy, which the Schorfheide and Song paper and the Foroni, Marcellino and Stevanovic paper both omit from their forecast models.

It is striking that the COVID-19 data capture many of the changes in the economy. While these data were not available in the initial stages of the pandemic, there was a wide array of epidemiological forecasts for the path of the pandemic. However, there remained uncertainty in the initial months of the pandemic as to how the COVID-19 variables related to macroeconomic variables of interest. It is only with hindsight that we can estimate how they comove.

Conclusion

During an episode like the COVID-19 pandemic with no recent precedent, information outside typical models takes on an increased importance. The successes and failures of various forecasting methods highlight how acknowledging the pandemic's particular circumstances could have led to more accurate forecasts, both in terms of point forecasts being close to actual results and error bands realistically capturing the underlying uncertainty.

The lessons go beyond the specific context of model forecasts during the COVID-19 pandemic. Model specification is similarly important for forecasters during normal times. While we have more data to discipline the model parameters, assumptions can mask uncertainty and bias forecasts. Introducing additional data only improves forecasts when paired with an appropriate model. Only models with sufficient flexibility can fully capture the uncertainty in forecasts.

People making use of model forecasts — such as policymakers or business managers — also need to be cognizant of these issues. Model forecasts are often examined to form personal forecasts that incorporate other knowledge or beliefs one may have. Understanding the assumptions underlying any forecast allows one to see potential sources of forecasting mistakes. In addition, when using narratives or other sources of information outside the model, how exactly these should influence forecasts should be considered.

None of this is straightforward. Economists have had lengthy debates about how to model the changes that led to the Great Moderation and Great Recession. There is a vast amount of ongoing work understanding both the short- and long-term impact of the COVID-19 pandemic on the economy. Forecasters should reflect these challenges by modeling the underlying uncertainty and communicating the inevitable assumptions clearly.

Paul Ho is an economist in the Research Department at the Federal Reserve Bank of Richmond.

This article may be photocopied or reprinted in its entirety. Please credit the author, source, and the Federal Reserve Bank of Richmond and include the italicized statement below.

Views expressed in this article are those of the author and not necessarily those of the Federal Reserve Bank of Richmond or the Federal Reserve System.