Why Many Scientific Articles are Wrong

 Suppose you are a professional academic who wants to publish a journal article in order to improve your chance of getting an offer, getting tenure, getting a raise. One way to do so is to produce and write up research that provides support for a novel theory. One problem is that, if the theory is true, it is quite likely that someone else in your field, over the past century or so, has already discovered it and published it, making your result not novel, hence likely to be rejected by the journal you submit it to.

If, on the other hand, your theory is false, the odds are much better that nobody else will have come up with it, found evidence to support it, and published. So if you can produce what looks like good evidence for a false theory, the odds that it will be novel, hence publishable, hence will contribute to your career, is much higher than for a true theory.

How do you produce evidence good enough to be publishable for a result that is not true? 

One solution is a specification search, aka p-hacking. Your theory is that eating onions reduces the risk of Alzheimers disease. To test it, you find a sample of old people who have been tested for symptoms of cognitive decline and survey them on their dietary habits. As a first crude test, you run a regression with degree of cognitive decline as the dependent variable, estimated previous onion consumption as the independent variable.

Unfortunately, that doesn't work—there is no significant correlation between the two. You rerun the regression, this time doing it separately for men and women. Then separately by race. Then by race and gender. Then limited to people over 80. Then to people over 90. Then making your independent variable not estimated onion consumption but whether they report eating onions frequently, occasionally, or not at all. Then do that version for all your racial and gender categories. Then ...

When you are done, you have run a hundred different regressions, testing different variants of the theory that onions are protective against Alzheimers. You are gratified to discover that three of them are significant at the .05 level, with the right sign. You pick the best one and publish it: "Cognitive Effect of Self-Reported Onion Consumption on Elderly Afro-American Women."

The fact that a regression result is significant at the .05 level means that, if your theory is not true, the chance that the evidence in favor of it will, by pure chance, be as good as what you found is only .05. It follows that, if your theory is false, a hundred separate experiments can be expected to produce about five that support it at the .05 level.

In this version of the story, the researcher is deliberately trying multiple experiments and only reporting the results that support his theory. The same effect could occur via multiple experiments by multiple researchers. If a hundred different researchers produce one experiment each, all for false theories, about five will show evidence for the theory at the .05 level. 

And get published.

Project I: Why do States Spend Money?

I have a file drawer full of research projects that I at some point started and then abandoned. Since I am unlikely to ever get back to them I thought it would be worth describing some here in the hope that someone else, earlier in his career, would be interested in reviving them, perhaps for a PhD thesis or a book. This is the first such post.

At a crude level, there are two theories of government expenditure. One is that it is determined by the need for government services. The other is that it is determined by the ability of the government to get money. It would be interesting to know the relative importance of the two in the real world.

One obvious test is the behavior of government finance before, during, and after a war. The war sharply increases the need for government spending. Its end eliminates most of the need but leaves standing the higher taxes used to pay part of the cost of the war. On a pure need theory, one would expect government expenditure to fall back to something close to its pre-war level. On a pure revenue theory, one would expect expenditure to remain high—not as high as during the war, when it was financed in part by borrowing and/or inflation, but at least as high as the taxes enacted during war time would support.

That is one approach to the question, but I thought of another. Different U.S. states have different sources of revenue, different tax structures. Different taxes respond differently to exogenous changes such as inflation. If a state is financed by a progressive income tax, inflation pushes taxpayers into higher brackets and so raises real revenue; if prices and incomes double, tax revenue should more than double. If, at the other extreme, a state is financed by regressive income taxes or by property taxes in a system where reassessment of property values is infrequent, inflation should reduce real revenue; if prices and incomes double, tax revenue should less than double. Keeping tax laws is easier than expanding them, so, if expenditure is driven by the availability of revenue, we would expect the first sort of states to have real expenditure increased, the second sort to have it decreased, by inflation. One could do a similar analysis for other changes in the economic environment that could be expected to change, in one direction or another, real revenues, and then see how the expenditure of states was affected by those changes.

To test the alternative, need, hypothesis, you need changes that affect needs for revenue. The largest expenditure of state and local governments is schooling. The cost of schooling largely depends on the number of school age children, which changes over time. On a need theory, when the fraction of the state’s population that is school age goes up, as it did for (I think) all states as the baby boom reached school age, state expenditures should go up. When it goes down, as it did in the years when the baby boom was coming out of the schools and onto the labor market, it should go down. While this effect would apply to all states, its strength should depend on how large a fraction of state expenditure goes to schooling. And other changes that affect the need for state expenditure may vary more across states.

For both hypotheses, the best evidence would be differences in what happened in different states, since that eliminates causes you are missing that affect all states equally, such as changes in technology that  make schooling or tax collection more or less costly. But you would also want to look at changes that affected all states similarly, such as demographic changes that affected the fraction of the population of school age.

That’s the project. Calculate, for each state, how real revenue would be expected to respond to changes in its environment. Calculate, for each state, how the demand for government services would be expected to respond to changes in its environment. See which plays how large a role in predicting what actually happened.

I have described the project from a U.S. point of view but it could also be done for Canadian provinces, Indian or Australian or Brazilian states, or as an international comparison—perhaps with the price of oil as an important exogenous variable.

How to Milk an Almond, Stuff an Egg, and Armor a Turnip now out as a Kindle

Some time back, my wife and I published our medieval cookbook, How to Milk an Almond, Stuff an Egg, and Armor a Turnip. In addition to making it available as a paperback on Amazon, I also put a pdf of it up on my web page for anyone who wanted to download.

It recently occurred to me that it would get much more visibility on Amazon, and that some people would find a kindle more convenient than a pdf. I put the question to people on Facebook, and quite a lot of them said they would like it as a kindle. 

So I converted my Word document to a kindle, using Calibre, a free conversion program. It's a nice program, and their technical support is fast and helpful. Despite that I ran into a lot of different problems, with the result that the conversion took something like ten to twenty hours, about ten times as long as I had expected. But it's now done and up on Amazon, and has sold thirteen copies in the last two days.

For those not familiar with it, the book consists of all of the cooking material from the Miscellany that Elizabeth and I have self-published for a very long time. It contains over three hundred medieval and renaissance recipes, in each case with the original (or a translation) and our worked out version. In addition there are articles on topics related to period cooking. I assume that most but not all of our customers will be fellow SCA members.

McCabe and the 25th Amendment: Two Puzzles

According to news stories based on statements by Andrew McCabe, he tried to arrange to remove Trump via the 25th Amendment. This raises two puzzles:

1. Under the 25th Amendment, the VP plus a majority of the cabinet can temporarily suspend the power of the President. But the next step is for the President to inform both houses of Congress that he is able to function. He then gets back into power unless both houses vote against it by a two-thirds majority

If two thirds of both houses wanted to get rid of Trump they wouldn't need to use the 25th Amendment, they could just impeach and convict. McCabe's tactic only makes sense if he hadn't read part 4 of the amendment the tactic was based on, which seems unlikely. Am I missing something?

2. "You know those paranoid ideas Trump had that the Deep State was out to get him? Well, we were." That's what the story comes down to. It's hard to see how that doesn't help Trump—which, on McCabe's account, is just what he shouldn't want to do. 

So why did he tell it? The least implausible answer I can come up with is that he wanted to sell his book—even if doing so resulted in four more years of Trump. 

Comments welcome.

The Machinery of Freedom as an Audiobook

I am in the process of recording my first book, The Machinery of Freedom, which I plan to make available on Audible and iTunes. So far I have recorded parts I-III, which include all of the contents of the first edition, and the recordings are now on my web page for comments. If you notice any mistakes, let me know.

One problem in doing an audiobook is footnotes. I have mostly left them out, aside from ones that could be, and needed to be, incorporated in the text. Suggestions on that subject are welcome. Possibilities include:

What I have done, perhaps incorporating more

Inserting all footnotes in the recording, probably as "footnote: ..."

Giving the URL of the webbed second edition in my introductory comments and suggesting that anyone who wants to see the footnotes for parts I-IV can find them there.

The webbed recordings are lower quality than the ones I will use for the final audiobook, 16 kbps instead of 192 kbps, in order to keep down file sizes so as to make downloading easier.

Fairfax v Kavanaugh

One point I have not seen discussed in comparisons between Vanessa Tyson's accusation of Justin Fairfax and Christine Ford's of Brett Kavanaugh is the reason that the more recent accusation is also much more likely to be true. 

For any given woman to invent such a story is quite unlikely. That out of a thousand women with both opportunity and motive at least one would do so is not. There were hundreds, probably thousands, of women who could have told the same story that Christine Blasey Ford did tell—any who, in high school or college, lived close enough to Kavanaugh to have gone out with him or attended a party at which he was present. Any of them who were politically left of center had a reason to invent such a story, since even before the accusation Kavanaugh was being ferociously attacked for his predicted effect on the court. 

Fairfax has admitted a sexual encounter with Vanessa Tyson, the first of his accusers. That drastically reduces the number of women in an equally good position to make such an accusation. Further, there is no obvious reason why Tyson, or anyone else in a similar position, would want to invent such a story—Fairfax is not a conservative Supreme Court candidate or anything similar. Instead of hundreds or thousands of potential accusers with both opportunity and motive, we have perhaps none, perhaps two or three. That makes the odds that the story is an invented one a great deal lower.

Assortative Mating and Increasing Inequality

A thought on inequality, based in part on a point in The Bell Curve.

The authors argue that one effect of a meritocratic system is an increase in assortative mating. It occurs to me that the same effect would be expected from any change that increased the range over which individuals sought mates. The girl in your village who makes the best fit with you is likely to fit less well than the girl in your city who makes the best fit. As population becomes more concentrated, transport and communication better, the result should be a greater pairing of like with like.

That assumes, in the context of intelligence, that smart men want to marry smart women and vice versa. I have just been listening to an audiobook of Heinlen’s Podkayne of Mars, in which it is assumed, by the viewpoint character and presumably the author, that men don’t want to marry smart women, hence that smart women find it prudent to conceal their intelligence. If true, that might reduce or eliminate the effect.

If my line of argument is correct, it provides an explanation of increasing economic inequality, since assortative mating should result in widening the spread of whatever characteristics are being sorted on, and some, such as IQ, are relevant to income.

The Flynn effect is a gradual increase in mean IQ. Has anyone looked at whether variance is also increasing?