On the difficulties of using macroeconomic data for policy advice

The economics blogosphere and now the mainstream financial press are full of discussion about the flaws in widely cited research done by Professors Ken Rogoff and Carmen Reinhart. These authors produced an excellent and path-breaking book This Time Is Different which surveyed in detail several centuries of financial crises. This book, which suggested, unfortunately correctly, that the aftermath of financial crises is a long, slow economic recovery, is NOT being criticised. It stands as a first class piece of scholarship.

Does high debt cause slower growth?

What is being criticised is a paper published in the American Economic Review (Papers and Proceedings) in 2010 “Growth in a time of debt“. (Note that “papers and proceedings” are not peer reviewed). The article found that “whereas the link between growth and debt seems relatively weak at “normal” debt levels, median growth rates for countries with public debt over roughly 90 percent of GDP are about one percentage point lower than otherwise.” Bear in mind when reading what follows that this statement remains intact. The next statement, that mean growth rates were much lower for higher debt countries is no longer valid based on the new criticism of the paper. And note, critically, that the statement says nothing about causality – it talks of a link, a “systematic relationship” but the paper is “decidedly empirical”. But in the conclusion the authors extend their comments to suggest that the causation might be from high debt to low growth, through high interest rates associated with what an earlier Rogoff paper called “debt intolerance.”

The statistical association and the implication of the authors’ conclusion that the causation ran from high debt to low growth naturally led many commentators to read the paper as a justification for so-called “austerity” fiscal policies, or more accurately contractionary fiscal policy that aimed to cut cyclically adjusted budget deficits in the middle of a recession. This is the policy of the UK and the Eurozone (forced upon periphery nations by the conditions of actual or potential bailouts, chosen freely by Germany and unhappily accepted by the French government). The US by contrast pursued a somewhat expansionary fiscal policy until the current year when the “sequester” kicked in to reduce automatically some areas of government spending. Unlike Europe though, the US economy shows signs of recovery.

Many economists supported the contractionary policies, which did not depend for their justification on Reinhart and Rogoff’s empirical work. Many other economists argued that the policies were wrong, not grounded in macroeconomic principles or financial history and were likely to at best cause prolonged economic stagnation and at worse make the debt/GDP ratio worse because any improvement in the numerator was more than offset by a fall in the denominator. There was also a lively debate about the size and sign of the fiscal multiplier, the ratio of an increase in government spending to the subsequent change in GDP. The IMF, traditionally a stern opponent of any fiscal excess, initially seemed supportive of fiscal austerity, but gradually shifted its view, informed by new research that suggested that multipliers were positive and significantly above one when interest rates were at the lower bound, namely that they couldn’t be cut any more (sometimes called the “liquidity trap”). In it’s most recent World Economic Outlook, published a few days ago, the IMF is clearly urging the pro-austerity UK government to curtail or even reverse its policies.

Correlation doesn’t imply causation

Perhaps the most important lesson students should learn from doing elementary statistics is that correlation doesn’t imply causation. Two variables may be correlated but far from X causing Y, they may be either randomly correlated or Y may actually cause X, or they may both be caused by a third variable Z. In the monetarism debate of the 198os, crude proponents of purely monetary solutions to inflation argued from the correlation of money supply and price inflation that M caused P so controlling M would curtail changes in P. Both practical experience and a wealth of statistical analysis destroyed this argument and the monetarism of the 1980s has vanished. M and P are correlated, like so many macroeconomic time series, because they all tend to rise over time. But so does cumulative rainfall, as Prof David Hendry of Oxford University once pointed out. Better macroeconometric techniques are now available which reduce the risk of spurious correlation and a much more sophisticated argument about independent central banks targetting inflation succeeded in theory and practice, with very successful results.

Phase one of criticism of Reinhart and Rogoff centred on their implied argument for causation, though they didn’t technically say this in their paper. The new criticism is about internal errors in the research itself, summarised in an FT article by the authors of the academic paper, Robert Pollin and Michael Ash of the University of Massachusetts. Their criticisms are as follows.

1. A coding error in the Excel spreadsheet meant some data were left out of the analysis. This is fully accepted by Reinhart and Rogoff.

2. The authors omitted other data from the analysis in an apparently selective way. The implication is that Reinhart and Rogoff “rigged” their conclusions. They dispute this totally, arguing that some of the data had not been fully validated and it WAS included in a later paper published in the Journal of Economic Perspectives in 2012, which reached very similar conclusions to their 2010 paper. This paper more clearly argued for causation, directly contesting the argument that slow growth causes high debt rather than, as they authors argue, high debt being an “overhang” that damages GDP growth.

3. The authors used an inappropriate way of weighting the different country data, specifically an episode when highly indebted New Zealand suffered a 7.6% fall in GDP. Reinhart and Rogoff, in their reply, don’t specifically address this point, preferring to emphasise that the overall findings of Pollin, Ash and Herndon (the other co-author of the Amherst paper) broadly agree with Reinhart and Rogoff’s findings of median growth and debt relationships.

Reinhart and Rogoff acknowledge a technical error in their work. Their refusal for some time to make their data available and other authors’ inability to replicate their exact results did raise some suspicion and that error may appear to vindicate the critics. I should note that Reinhart and Rogoff are highly distinguished academics. Rogoff was also the Chief Economist at the IMF for several years and is co-author of a leading textbook in advanced international macroeconomics. He is a highly sophisticated author and communicator.

Other research has cast some doubt on their findings (that is, other researchers disagree, which is common enough in empirical macroeconomics – see this very thoughtful blog by Mark Thoma). Some authors have concentrated on the example of the US economy right after World War II. Emerging with very high levels of debt but facing a period of massive demobilisation of military resources, there is a definite combination of high debt and low growth but it is obviously not the debt causing the slow growth. Historically rich countries got into very high debt only because of wars and the end of wars often leads to temporary depression. But if you exclude these cases you shrink the sample size appreciably and there is disagreement about excluding “special cases” since history is a long sequence of unique events. 

Conclusions – economists must be very careful

What can we conclude from all of this? The rather gleeful reaction to the critique of Reinhart and Rogoff is a little overdone, in my view. The core of their statistical findings remains intact: countries with high debt typically have lower growth of around one percentage point, which may not sound a lot but cumulatively over several years is significant. The causation, though, remains much more unclear and likely works in both directions at different moments of history. It’s a little unfair to blame mass unemployment on the error in their work, though this blog, which sort of does that, is still worth reading. Reinhart and Rogoff’s book remains a landmark in research.

Politicians such as Paul Ryan, who passes in the US for an intelligent and sophisticated thought leader, made much of the 90% threshold, without giving due weight to the lack of evidence of causation. But he is, after all, a politician. The continued support by the European Central Bank, European Commission (particularly Olli Rehn – see Paul Krugman’s critique here) and many German politicians for austerity policies, does not depend on Reinhart and Rogoff, though it was clearly convenient to have their apparent support. The continued destruction of the Iberian economy will continue and the EU project remains in jeopardy. (A general survey of the fading case for austerity economics was just published in The New Yorker here).

The economic controversies of recent years have brought out some very unedifying, poorly justified arguments from several American economists who were seemingly motivated by politics (and in some cases the hope of being appointed Chairmen of the Fed after Bernanke). Some on the left have also been sloppy in their thinking too. Academics have to be very careful in how they express their research findings in a subject like economics that can lend support, even unwittingly, to politicians who lack the intellectual or in some cases moral integrity to do justice to what are highly provisional, uncertain and incomplete research findings. This all reminds me once again of one of the most famous quotations from John Maynard Keynes.

The ideas of economists and political philosophers, both when they are right and when they are wrong are more powerful than is commonly understood. Indeed, the world is ruled by little else. Practical men, who believe themselves to be quite exempt from any intellectual influences, are usually slaves of some defunct economist.” (The General Theory of Employment, Money and Interest, 1936)

A wider reading list on this topic is available here.


Now that the primary data are available to other researchers, we should see further work on this subject. This blog post shows some preliminary work that suggests the main causality is from low growth to high debt, in other words the opposite of the effect that proponents of austerity economics have inferred from Reinhart and Rogoff’s work.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.