Rigour in the academic world

Recently one of my most distinguished colleagues said that the defining feature of academic research is rigour. He didn’t define it but I think his is a widely accepted view. I agree but “rigour” needs to be considered in context.

I have an excellent book from my graduate economics days, when I was struggling with mathematical economics, called “Logic, Sets and Numbers” by the LSE professor Ken Binmore. On page 4 there is an excerpt from a book on metamathematics, showing a proof that “a = a”. Since that was not one of the author’s axioms, it had to be formally proven. The proof takes 17 lines and starts with:  a=b→(a=c→b=c). This is certainly rigorous but isn’t useful in most contexts. We can, on the whole, take it as reasonably given that a does indeed equal a.

Economists like rigorous statements because it can be easy to waffle in this subject, with the risk that a questionable or ill-founded argument slips by. But the hygiene of rigorous statement can turn compulsive. At Cambridge, in the notoriously difficult second year microeconomics course taught by Frank Hahn (a giant of general equilibrium theory, though admittedly not a household name), I watched entranced as he proved beyond any logical doubt that, given the axioms of consumer choice, compensated demand curves have non-positive slope. This statement is slightly less impressive when rephrased as: if you hold a person’s income constant, they will never buy more of something when its price goes up. That conclusion follows rigorously from the assumptions economists make about consumer choice (which amount to saying that people know what they want in all circumstances and choose consistently). But it’s actually not always right in practice. We know there are (relatively rare) examples when people do buy more of something of which the price is going up, either because of a bandwagon effect or speculative bubbles or perhaps because the price is taken to be a symbol of quality (fine wine). Professor Hahn would have said that theory is there to help us organise our thoughts and not simply to be tested by empirical accuracy.

At graduate economics level the maths got more complex. The proofs moved away from calculus (which requires nice, differentiable functions, a very small subset of the possible range) in favour of topology. Proofs involved convex hulls, hyperplanes and cluster points, which can be illustrated in three dimensions but when generalised to n-dimensional space are hard for me to visualise, explaining why I’m not a mathematician.

I think Keynes somewhere argued it’s better to be approximately right than precisely wrong. But at least there is a definite meaning to the sort of rigour cherished by economists. Other social sciences simulate rigour in the form of opaque jargon or conceptual complexity that you can’t help suspecting is there to make banal statements seem more remarkable. In A-level human geography I discovered that I got high marks for describing a house as a “residential locational unit”. Once I’d grasped this trick I was able to write all sorts of pretensious nonsense and get top grades. Perhaps the reason I’m so dismissive of management jargon and this sort of ugly writing is that I’m quite good at it, a sort of self-loathing sets in.

A lot of research in management studies uses the method of ethnography, a technique of observation and interpretation invented in anthropology. This is distinguished from journalism in its ambition of rigour, in particular trying to be as clear and honest as possible about potential distortions and bias that inevitably arise because the observer (or journalist) cannot possibly be entirely independent, objective or disinterested. It is a well known result in social science that the mere act of observing something may change it.  A classic early research project done in the 1920s at General Electric’s factory in Hawthorne, Illinois was shown later to have been contaminated by the fact that the workers seemed to work harder because there were researchers taking an interest in them. This became known as the Hawthorne Effect. One can try very hard to design experimental research to minimise its effect but it’s hard to take the observer completely out of the picture.

Sometimes social scientists like to refer instead to the Heisenberg uncertainty principle, which is that you can know with precision the position or the momentum of a particle but not both. In other words, trying to measure one variable reduces your knowledge of the other. Very loosely this gives you something like the Hawthorne effect, that in an attempt to study a social phenomenon you distort or influence it. I think mentions of the Heisenberg principle in social science often arise from an attempt to invoke the much higher standards of physical science. Social science, though, is simply harder because you cannot, for the most part, do controlled experiments. We social scientists should just accept that as a limitation rather than pretend to be able to copy natural science.

[Actually, Esther Duflo, another brilliant young economist like Daron Acemoglu (“Why Nations Fail”), has shown that you can do blind experiments in social and economic policy which yield very real and valuable results for the better design of development policy. There is an inspiring lecture on this theme at TED].

My colleague Mark de Rond is so rigorous in his attempt to minimise (but not remove) his own influence from his ethnographic research that he keeps a diary of his dreams, in an attempt to distinguish what are his thoughts from those of the people he is observing (most recently in his remarkable work with military surgeons in Afghanistan, as described here). Mark makes clear in the video that we can never completely take ourselves out of the picture.

It’s hard in practice to really nail down the difference between ethnography and first rate journalism. The goals may be different, but journalists can bring enormous knowledge from their work. Some famous examples are George Orwell’s writings on life as a down-and-out and Barbara Ehrenreich’s under-cover work among low paid US workers, Nickel and Dimed. (Polly Toynbee of the Guardian wrote a similar, brilliantly revealing piece about low pay in the UK).  Orwell was not trying to be neutral, he was an avowed political writer, but he did try to capture the reality of being poor. There is a great deal of first class foreign correspondent work too(*). This work has all the value of more academically rigorous research. Yet I have often heard the term “journalistic” used to disparage a piece of academic research.

Rigour is more about the intellectual honesty of your argument. One of the most gifted and inspirational teachers I had was the late Dr. Iain Macpherson of Gonville and Caius College, Cambridge. He taught us economic history, a subject that is perfect for learning how to think and argue logically because it combines qualitative and quantitative evidence, well defined and interesting questions (did the standard of living increase or decrease in the early years of the nineteenth century in industrialising England?) with all the challenges of history – missing data, conflicting anecdotal evidence and an unavoidable ideological taint to the debate. Dr. Macpherson was rigorous in requiring us to be clear about the question, the meaning of the words (standard of living how defined, relative or absolute etc) and weighing the evidence according to its provenance and sample size. By comparison, rigour in logic or maths is relatively straightforward. In economic history there is a multitude of traps where your argument may turn out to be false or overstated. As a lesson in thinking, this was without parallel in my education. Long after I’ve forgotten whether the standard of living fell or not, I hope I’ve retained the thinking skill. As Einstein said, “education is what remains after one has forgotten what one learned at school”.

As I was rude to lawyers in a previous post I should say that is sort of rigorous reasoning is equally applicable to some law. By way of amends, let me end by recommending a brilliant, short, beautifully clear book by the late Tom Bingham, The Rule of Law. Interpreting the law is seldom, if ever, a matter of pure formal logic, but requires the sort of judgement and intellectual breadth needed in economic history. Lord Bingham (who studied history at Oxford, then become a barrister and later Lord Chief Justice of England and Wales – Scotland has a separate legal system) makes the idea of the rule of law wonderfully lucid and argues, rigorously, for the need to protect it.

(*) But clearly not all of it is first class. A classic book on being a foreign correspondent (a now fast declining breed because of budget cuts at newspapers squeezed by web-based news) is called “Anyone Here Been Raped and Speaks English?”