Thought for the Day--May 14, 2011: Watching the Watchdog? A Call for Consumers Union Action: A Plea for Transparent Research Protocols

I think it's a good thing that there are bloggers out there watching very closely and holding people accountable. Everyone in the news should be able to hold up to that kind of scrutiny. I'm for as much transparency in the news gathering process as possible.

--Anderson Cooper

For my May 12 post Shame on You, Consumer Reports..., I was surprised at how difficult it was to track down Consumer Reports' primary research source for its diet rankings in the June 2011 issue ("Pick Your Ideal Diet"). I cite two reasons:

1. Consumer Reports omitted the name of the author and title of their primary source, the very source used to "edge" Jenny Craig into First Place.

(Cheryl Rock, et al; "Effect of a Free Prepared Meal and Incentivized Weight Loss Program on Weight Loss and Weight Loss Maintenance in Obese and Overweight Women." (1803-1810)

2. The CR source appeared in the October 27, 2011, issue of JAMA, which published 2 articles and 2 opinion pieces on diet studies and trials:

Two Original articles (Primary Sources):
--"Effects of Diet and Physical Activity Interventions on Weight Loss and Cardiometabolic Risk Factors in Severely Obese Adults: A Randomized Trial," by Bret H. Goodpaster, et al

--"Effect of a Free Prepared Meal and Incentivized Weight Loss Program in Weight Loss and Weight Loss Maintenance in Obese and Overweight Women: A Randomized Controlled Trial," by Cheryl L. Rock, et al

Two Editorials (Secondary Sources):
--"The State of Obesity and Obesity Research," by Donna H. Ryan and Robert Kushner

--"Treatment Options for Obesity: Do Commercial Weight Loss Programs Have a Role?" by Rena R. Wing (this article is an editorial response to the Rock, et al, article, not an original article; an editorial is almost always considered a "secondary source.")

At first, I assumed that the Wing article was the source that CR had used for deciding to crown Jenny Craig as the weight-loss program winner. The title was logical for what CR was trying to accomplish. However, after I received the full text, I realized it was an editorial about the Rock article.

So I now knew that I needed to read the Rock article and, as I noted in the May 12 post, I was stunned at what I discovered.

Was Consumer Reports' failure to include the title and author in its ranking of diet programs intentionally deceptive or just plain sloppy research?

I don't know, but does it really matter? The effect was the same: CR handed its readers a shoddy bill of goods, readers who trust CR to offer them the best possible information.

Consumers Union, as the largest non-profit "...independent consumer testing organization." (Consumer Reports, June 2011, page 4) has a responsibility to ensure that their findings are based on solid research and facts.

Consumer Reports should be transparent with their research sources and protocols:

--When referring to outside research (as opposed to in-house testing), use only unbiased research. At the very least, never cite a study or trial that was financially supported by the product or service being tested or ranked, as was the case with Jenny Craig.

--Publish a full reference list at the end of each article and/or on the CR website.

--Reveal any possible bias of the study/trial, especially if it might not be evident to the reader, for example, size of sample, type, demographics, potential author bias, sponsors, funding sources, protocols, objective of study, outcomes, perks given to research subjects, etc.

--Disclose whether the study or trial was blind. The Cheryl L. Rock trial was not blind; Jenny Craig staff knew which members were part of the trial (thus, non-paying and receiving free pre-packaged food), which may have skewed the results in favor of the sponsor--in this case, Jenny Craig.

--Author bylines should be attached to all Consumer Reports articles and rankings. That way, authors will be more accountable for their work, and, thus, more invested in conducting careful research.

My trust in Consumer Reports has been shattered, and I will consider any future product tests and rankings with great skepticism and scrutiny--regaining my trust will take a long time.

However, Consumer Reports can begin right now by taking a good look at their current protocols and then improving upon them.