Do Regional Fed Surveys Reflect National Manufacturing Conditions?

Surveys play an important role in how research staff across the Federal Reserve System understand changing economic conditions. Several Reserve Banks conduct manufacturing surveys to gauge the health of the manufacturing sector in their districts. Since these surveys are published before national data, policymakers and market watchers often use them for a signal of where U.S. manufacturing activity is heading. In October 2014, the Dallas Fed published an Economic Letter in which they found that regional Fed manufacturing surveys improved forecasts for several key national indicators. In this post, we reexamine the Dallas Fed's analysis to understand if regional manufacturing data published by the New York, Philadelphia, Richmond, Dallas, and Kansas City Reserve Banks — 10 years later — still track leading national economic indicators.

How Well Do Regional Fed Surveys Track National Data?

To assess how each Bank's survey tracks national data, we look at the monthly "topline" number published from each Bank between January 2004 through December 2023. There are differences in several aspects of each Bank's survey that could affect overall performance against national measures, such as how the published topline number is derived. For example, New York, Philadelphia, and Dallas Reserve Banks publish a general business activity index (GBAI) based off a single question, whereas Richmond and Kansas City Reserve Banks publish a composite index (CI) derived from several questions. Richmond's CI represents a weighted average of the shipments (33 percent), new orders (40 percent) and employment (27 percent) indexes. Kansas City's CI is an average of its survey's production, new orders, employment, supplier delivery time, and raw materials inventory indexes.

The Institute of Supply Management Manufacturing Purchasing Managers Index (ISM PMI) is a monthly index that gauges economic activity in the manufacturing sector. This survey is seen as a reliable leading indicator of the U.S. manufacturing industry. Our analysis found that all Reserve Bank surveys correlate strongly with the ISM PMI: New York had that strongest overall correlation (.834), followed by Richmond (.808). Kansas City had the lowest correlation, although still strong (.714).

Interestingly, the strength of each Bank's correlation has not been consistent over time. During the COVID-19 period (2020-2023), correlation between the ISM PMI and surveys from Philadelphia and Kansas City strengthened, whereas the correlation from New York weakened.

Another measure of assessing national manufacturing activity is the monthly Industrial Production (IP) index produced by the Federal Reserve Board of Governors. This index measures real output in the manufacturing, mining, electric, and gas industries, relative to a base year. Overall, there is a moderate level of correlation with topline metrics and the FED's IP index, ranging between .353 (Kansas City) and .406 (Richmond).

When we looked at correlations across several economic periods, we found that the strength of the correlation declined post-Great Recession for all Banks. For example, the Richmond Fed index had a correlation of .6 during the Great Recession but dropped to less than .3 during the 2015-2019 recovery. However, starting in 2020 (COVID-19), the strength of the correlations for most Banks, except the Kansas City Fed, improved.

Which Regional Fed Survey Indexes Reflect National Manufacturing Conditions?

While correlations provide a useful gauge of the comovement of regional Fed surveys with national manufacturing indicators, it is also helpful to test their statistical power. In this section, we run multiple regressions of key national manufacturing indexes on their lags and each of the contemporaneous values of five selected regional Fed survey indexes.

When coefficients in the models are statistically significant, we have more confidence that a variable can provide a signal about national manufacturing indexes. The table below captures results from two multiple regressions used to assess the predictive power of each of the regional Fed surveys on the ISM PMI and industrial production growth. The "Yes" fields show which regional Fed indexes were statistically reliable in explaining month to month changes in the national manufacturing variables.

The first column shows that the Kansas City Fed's CI, Philadelphia Fed's GBAI, and the Richmond Fed's CI provided statistically significant predictive power for the ISM PMI. However, the second column shows that only the Richmond Fed's CI provides explanatory power for industrial production growth.

How Useful Have Regional Fed Surveys Been in Forecasting National Manufacturing Activity Over the Past Three Years?

We assess the ability of each of the regional Fed indexes to aid out-of-sample forecasting of national manufacturing indicators from January 2021 through December 2023. In the table below, we compare the forecasting performance of a baseline model (that includes only the three most recent months of each of the national manufacturing indicator) to a model that also includes the current value of the regional Fed survey. The relative root mean squared forecast error (RMSFE) assesses how well each regional Fed survey index aids the forecasting accuracy of national manufacturing activity compared to the baseline model. When the relative RMSFE is lower than 1, the regional Fed survey improved the forecasting accuracy compared to the baseline model. We also report results on the statistical significance between the forecast errors of the baseline versus each regional Fed model.

The results from our forecasting exercise suggest that only the Philadelphia Fed's GBAI significantly improved the forecasting accuracy of the ISM PMI compared to a baseline model, and the New York Fed's GBAI significantly weakened forecasting performance. None of the regional Fed manufacturing indexes significantly improve or weaken the forecasting performance of industrial production growth compared to a baseline model.

Since we focus only on contemporaneous relationships between the national and regional surveys (comparisons within the same month), our analysis may be obscuring more dynamic relationships between the series. For example, an article by Santiago Pinto and Nika Lazaryan contends that dynamic forecasting models of the ISM index using the Richmond Fed survey series tend to be more accurate when they allow for different numbers of lags for each variables, since many of the series exhibit a high degree of persistence with differing correlations depending on the time horizon. Additionally, the ability of a regional survey to forecast national variables may ultimately depend on how representative the region (or the survey's sample) is of the nation's manufacturing firms.

Views expressed are those of the author(s) and do not necessarily reflect those of the Federal Reserve Bank of Richmond or the Federal Reserve System.