A recent and widely reported news story, headlined "Kids of lesbians have fewer behavioral problems, study suggests," reported that "A nearly 25-year study concluded that children raised in lesbian households were psychologically well-adjusted and had fewer behavioral problems than their peers."
The conclusion does not strike me as particularly surprising. In our society and many others, mothers play a larger role in child rearing than fathers, so it would not be surprising if children with two mothers did, on average, better than children with a mother and a father. Reading the story, however, I concluded that it did not actually give me much reason to believe in its conclusion—for two related reasons.
The first was a quote from the lead researcher that appeared in some versions of the story: "Gartrell can't say with certainly whether the findings would apply to gay fathers. It's ''highly likely," she says."
Gartrell's study was limited to lesbian couples. On theoretical grounds, one could take her result as evidence not that homosexuals are good at child rearing but that women are, in which case it would imply the precise opposite of what she suggests. Insofar as the quote is evidence of anything, it is evidence of bias on the part of the researcher. As anyone familiar with statistical work knows, there are a lot of ways in which a researcher can tweak the design of a study, deliberately or not, to produce the result the researcher wants.
One way might be by the choice of the set of heterosexual couples to which the lesbian couples were being compared. The two groups might differ in important ways other than their sexual preferences. Most obviously, since the lesbian parents had conceived via artificial insemination, their pregnancies were all planned and all desired. If the comparison group contained a significant number of children from unplanned and unwanted pregnancies, that might explain why more of them had behavioral problems. One could imagine a variety of other possible explanations as well—and the news stories did not provide enough information to confirm or reject them.
Fortunately, nowadays, one is not limited to news stories. A web search quickly turned up the actual text of the article. Reading it, I discovered:
1. The two groups were not closely matched, due to data limitations, a problem that the authors noted. They differed strikingly in geographic location, since the lesbian couples were all recruited in the Boston, D.C., and San Francisco meteropolitan areas, while the data on children of heterosexual couples, coming from another researcher's work, was based on a wider distribution of locations. They were not matched racially—14% of the heterosexual couples were black, 3% of the lesbian couples were. They were not matched socio-economically—on average, the heterosexual couples were of higher SES than the lesbian couples.
The statistical analysis on which the story was based contained no controls for those differences, any one of which might have affected the conclusion. The authors could have compared the outcome of child rearing by white lesbian couples to the outcome of child rearing by white heterosexual couples, by high SES lesbian couples to high SES heterosexual couples, ... . They did none of that, although it is possible that in future work they will.
2. There was a second problem that had not occurred to me, possibly because I had not read the news story carefully enough. Questionaires went, at various points in the study, to both mothers and children. But the conclusion about how well adjusted the children were was based entirely on the reports of ther mothers. A more accurate, if less punchy, headline would have read: "Lesbian Mothers Think Better of Their Kids than Heterosexual Mothers Do."
My point in this post is not to criticize the authors of the study. There may have been good reasons why they used the data they did, despite its limitations, and they may be planning a more informative analysis of their results for later work. It is easy for an outside observer to suggest things scholars ought to have done, not always as easy for the scholars to follow the advice.
My point is rather about the information available to ordinary readers through newspapers, the webbed equivalent, radio and television, and similar sources. The implication of the news stories was that the study provided strong evidence that lesbian parents did, on average, a better job of child rearing than heterosexual parents. A reader who went to the trouble of locating and reading the published report of the study, as few would, would discover that that implication was false. The study provided some evidence for the conclusion, but not very much. It could as easily be interpreted as evidence that richer people do a worse job of child-rearing than poorer people, blacks worse than whites, parents from flyover country worse than the inhabitants of the coastal metropolises—or that lesbian parents are even more strongly biased in favor of their own children than other parents.
The only thing that the headlines provided clear evidence of was what their authors wanted their readers to believe.
The conclusion does not strike me as particularly surprising. In our society and many others, mothers play a larger role in child rearing than fathers, so it would not be surprising if children with two mothers did, on average, better than children with a mother and a father. Reading the story, however, I concluded that it did not actually give me much reason to believe in its conclusion—for two related reasons.
The first was a quote from the lead researcher that appeared in some versions of the story: "Gartrell can't say with certainly whether the findings would apply to gay fathers. It's ''highly likely," she says."
Gartrell's study was limited to lesbian couples. On theoretical grounds, one could take her result as evidence not that homosexuals are good at child rearing but that women are, in which case it would imply the precise opposite of what she suggests. Insofar as the quote is evidence of anything, it is evidence of bias on the part of the researcher. As anyone familiar with statistical work knows, there are a lot of ways in which a researcher can tweak the design of a study, deliberately or not, to produce the result the researcher wants.
One way might be by the choice of the set of heterosexual couples to which the lesbian couples were being compared. The two groups might differ in important ways other than their sexual preferences. Most obviously, since the lesbian parents had conceived via artificial insemination, their pregnancies were all planned and all desired. If the comparison group contained a significant number of children from unplanned and unwanted pregnancies, that might explain why more of them had behavioral problems. One could imagine a variety of other possible explanations as well—and the news stories did not provide enough information to confirm or reject them.
Fortunately, nowadays, one is not limited to news stories. A web search quickly turned up the actual text of the article. Reading it, I discovered:
1. The two groups were not closely matched, due to data limitations, a problem that the authors noted. They differed strikingly in geographic location, since the lesbian couples were all recruited in the Boston, D.C., and San Francisco meteropolitan areas, while the data on children of heterosexual couples, coming from another researcher's work, was based on a wider distribution of locations. They were not matched racially—14% of the heterosexual couples were black, 3% of the lesbian couples were. They were not matched socio-economically—on average, the heterosexual couples were of higher SES than the lesbian couples.
The statistical analysis on which the story was based contained no controls for those differences, any one of which might have affected the conclusion. The authors could have compared the outcome of child rearing by white lesbian couples to the outcome of child rearing by white heterosexual couples, by high SES lesbian couples to high SES heterosexual couples, ... . They did none of that, although it is possible that in future work they will.
2. There was a second problem that had not occurred to me, possibly because I had not read the news story carefully enough. Questionaires went, at various points in the study, to both mothers and children. But the conclusion about how well adjusted the children were was based entirely on the reports of ther mothers. A more accurate, if less punchy, headline would have read: "Lesbian Mothers Think Better of Their Kids than Heterosexual Mothers Do."
My point in this post is not to criticize the authors of the study. There may have been good reasons why they used the data they did, despite its limitations, and they may be planning a more informative analysis of their results for later work. It is easy for an outside observer to suggest things scholars ought to have done, not always as easy for the scholars to follow the advice.
My point is rather about the information available to ordinary readers through newspapers, the webbed equivalent, radio and television, and similar sources. The implication of the news stories was that the study provided strong evidence that lesbian parents did, on average, a better job of child rearing than heterosexual parents. A reader who went to the trouble of locating and reading the published report of the study, as few would, would discover that that implication was false. The study provided some evidence for the conclusion, but not very much. It could as easily be interpreted as evidence that richer people do a worse job of child-rearing than poorer people, blacks worse than whites, parents from flyover country worse than the inhabitants of the coastal metropolises—or that lesbian parents are even more strongly biased in favor of their own children than other parents.
The only thing that the headlines provided clear evidence of was what their authors wanted their readers to believe.
13 comments:
The problem here is surely the newspapers. Time was when newspapers did that sort of fact checking for you.
It is an unfortunate thing that it is simply impossible to believe most "studies", unless you do some pretty deep research, simply because "studies" tend to be extremely sensitive to the control levers you mention. This is especially so of highly controversial matters such as the subject of this study. It is very easy for researchers, even well intentioned ones, to allow their biases to creep in.
A good quality study of this kind not only needs to control these variables, but needs to be double blind, and it needs the control variables defined by an independent group. With a controversial study, you probably need to get people from each side of the controversy to define the controls they would like to see in place before the study even takes place.
This speaks to the importance of three techniques:
Finding high quality sources that report on studies only after digging through them and routing out these statistical, contextual, and cognitive biases.
Having a strong skeptical "critical thinking" foundation through which to look at every new study that comes along, filtered through a good source or (especially) not.
Having a strong bias towards wanting to know "the truth" as opposed to any other reason to hold a belief. So much so that it creates psychological pain when you feel you could be in error.
All of these techniques are hard, require a solid background in critical thinking, take time, and constantly require adjustment and fine tuning.
The first commentator to this post writes: "Time was when newspapers did that sort of fact checking for you."
Was that the time of William Randolph Hearst or the time of Joseph Pulitzer?
This very article is good evidence that mainstream media is largely pointless and blogs are where the real journalism is done.
David, have you seen this cartoon?
http://xkcd.com/552/
The mere fact that they dedicated their work to this research means that:
- They understand the weaknesses of their study more than any of us
- Given more funding they'd be happy to refine their work, or they would be willing to help others with similar studies
By doing actual research they show far more interest than random bloggers on the internet.
Researchers >> Bloggers > Media
I believe it would be interesting to compare children conceived by one of the gay parents with children adopted, to which the adoptive parents are unrelated.
When we study children who have been conceived by the same people who raise them, we are analysing not only the quality of the rearing, but also their genetic makeup, which, some economists, like Sacedote, have pointed as important to socioeconomic outcomes.
I really have to commend you for this . I personally think the government should allow every couple to marry if it allows any couple to marry so I'm usually disgusted by what I read from NOM. But this was well-written, looked to original sources, and made some completely valid points. So I have to say, without sarcasm, that I am really surprised and impressed. It makes me think that perhaps someday we can have an actual debate on this issue.
Does anyone mention the fact that lesbian mothers have been inseminated by high quality sperm, chosen for high motility and health... qualities associated with high IQ. Donors are usually screened for health and IQ as well. this is likely the primary cause of the effect.
I disagree with both of your critiques.
The first critique is that much of the difference in the quality of parents can be explained by the simple fact that same-sex couples do not procreate accidentally. Agreed, but so what? This supports the proposition that on the whole, children of same-sex couples will do better than children of opposite-sex couples. To the extent that this bears on the question of whether procreation by same-sex couples is a good thing or a bad thing, it suggests that it is a good thing. In other words, it directly refutes the proposition that children need -- or are even better off with -- a mother and a father. Same-sex parents make for better families, if your theory is correct and wholly determinative, precisely BECAUSE they don't periodically procreate by accident.
Your second critique is that because (according to you) the study relied mostly or entirely on parents' voluntary responses, the results prove only that lesbians think more of their children's achievements than opposite-sex parents do. First, if this is true, it's still a pretty flattering depiction of same-sex parents. And second, this is an unfounded conclusion to make without greater familiarity with the methods. We don't know precisely what the questionnaires asked, but it's entirely possible that the conclusions relied on purely objective questions. For example: has your child been arrested or cited? What is his high school GPA? Is he in the National Honor Society? Does he play a sport, and if so, at what level? In which of the following categories of extracurriculars does he participate? Did he go to college, and if so, where? What were his SAT scores? Has he attempted suicide or been hospitalized for mental health reasons? Has he been expelled or otherwise disciplined in school? Does he have a job? What is his salary? Did he go to graduate school? Is he married?
It's entirely possible for a survey to collect data about a child's development by sticking to objective questions. It's also unclear to me that subjective questions are an invalid tool for research into children's development.
"First, if this is true, it's still a pretty flattering depiction of same-sex parents."
How do you come to that conclusion? Lesbian parents are surely aware that they are being watched - why not fabricate evidence of how great things are?
"entirely possible that the conclusions relied on purely objective questions."
Let's see those objective questions.
The only study I would trust is one that compares objective evidence (health, STD rates, teen pregnancy, use of drugs, depression rates, etc.). Not, "how are you doing?"
"It is very easy for researchers, even well intentioned ones, to allow their biases to creep in."
And it is even easier for passive commenters, with no expertise, no evidence and no scientific data, to disbelieve whatever is in conflict with their preconceptions, simply because they choose to do so. This is called "prejudice", hallmark of bigotry.
If you choose to disbelieve science, then do so, but be honest about the fact that this is what you're doing. If you think you can dispute the facts with science, then follow the rules: do the research and have it published in quality peer-reviewed journals. Otherwise your pontificating is of value only to those who share your prejudices.
To Andrew:
Since you interpret my response as bigotry, explain which part of my critique of the study you disagree with. Do you think judging how successful parents have been in bringing up their children entirely by the parents' report gives reliable information? Do you think that comparing the outcomes from two quite different groups without controlling for the differences, and reporting the result as entirely due to one difference, is a competent statistical procedure?
I can't tell from your post whether you even know what a multiple regression is, let alone how they are normally employed in analyzing data. What is clear is that the bias in this case is yours. You don't want to believe what I am saying, and instead of actually reading the study and seeing if my criticisms are valid you prefer to attack the messenger.
And you didn't even read my post carefully enough to note that I started by saying that I didn't find the conclusion of the study surprising.
Post a Comment