If we knew what it was we were doing, it would not be called research, would it?The critics are finally discovering what I posted about last month about Consumer Reports' ranking of diet programs.
--Albert EinsteinAdvertising people who ignore research are as dangerous as generals who ignore decodes of enemy signals.
Meanwhile, the media machine has already spread the "best" program "news" like a virus. The Google search term "Jenny Craig Best Program" (with quotes) has returned (as of June 7) about 47,000 results.
Very few "critical" entries of the Consumer Reports diet program rankings. No exact results for the terms "critics of diet ratings" or "critics of diet rating." Nothing for "critics of Jenny Craig diet rating." Zip for "critics of Consumer Reports diet rating" and zero for "critics of Consumer Reports diet ratings."
But buried within the rah, rah positive reviews, one can find a lame response to the critics who have posted their disdain on news and personal blogs from Consumers Reports itself. Also, some inklings of dissent are finally emerging in the media.
In "Critics Chew Over Our Diet Ratings" on the Consumer Reports Health blog, Nancy Metcalf, Senior Scientific Editor said,
We should have prominently mentioned that participants in the 2010 study, which had a remarkably low dropout rate, didn’t have to pay for the prepackaged meals, snacks, and desserts that are the backbone of Jenny Craig’s program--a freebie that the investigators valued at $6,240 over the course of the two-year study. The monetary value of the free food and counseling in the Jenny Craig trial is indeed a lot higher than, say, the monetary value of the Slim-Fast products, or the cost of membership to Weight Watchers, which were also free to participants in clinical trials of those diets...We didn’t delve into all this in our three-page article because freebies are standard for clinical trials.Unfortunately, the Consumer Reports audience consists of ordinary people who may not know that "freebies" are standard for clinical trials. Even so, the "freebie factor" is a major impediment to study non-bias, along with the fact (disclosed in the study notes) that the author of the study had sat, at one time, on the board of Jenny Craig and that the study was not conducted as a "blind" trial. In other words, Jenny Craig knew which of their members were enrolled in the study and getting a complete free ride. In addition, the study was small--442 people--and all were women.
By Metcalf's own admission, Consumer Reports ignored a large Jenny Craig study:
We did exclude this study, because it’s not a clinical trial like the ones we considered for the other plans, but a tracking study that observed what people naturally did over time. But it’s interesting nevertheless.It would be interesting to compare the Jenny Craig "stalwarts" with the Weight Watchers long-term members or even the stubborn Slim-Fast camp. I'd bet $50 that the stats would be similar; after all, long term dieters, no matter what program they choose, are simply more motivated than the 73% who quit after one month and the 93.4% who quit within a year.
The study, which was published in 2007 in the International Journal of Obesity, reviewed the records of 60,164 people who voluntarily signed up for Jenny Craig over a one-year period and paid for everything themselves.
The dropout rate was high: 27 percent had quit after a month, and only 6.6 percent lasted a full year. But among those stalwarts, weight loss after a year was significant: The women lost an average of about 16 percent of their starting weight, around 34 pounds. And men lost about 13 percent of their weight, or 37 pounds.
Oh, wait. Those other studies don't exist. My bad.
Also, Metcalf does not explain why the author and title of this "important" study--the one CR says "edged" out other programs--was not disclosed in the rankings:
"Effect of a Free Prepared Meal and Incentivized Weight Loss Program on Weight Loss and Weight Loss Maintenance in Obese and Overweight Women," by Cheryl L. Rock, PhD, RD, et al, and published in JAMA online on Oct. 9, 2010.Why do you think that this information was conveniently omitted? Could there have been an undisclosed bias toward Jenny Craig, for example, one or more CR researchers involved in the ratings article active or past Jenny Craig members? If so, then CR needs to disclose this fact.
Also, if Consumer Reports used, for comparison and contrast purposes, a study offering "free" Jenny Craig program materials, then shouldn't they also have used studies incorporating "free" Weight Watchers and Slim Fast materials? Wouldn't that at least come closer to comparing apples to apples?
Metcalf insisted that there are no definitive studies out there that compares and contrasts different programs, so this begs the question: why would Consumer Reports do any comparison rankings of diet programs if there are no studies that compare and contrast diet programs?
Sloppy, sloppy, sloppy research methods.
In this case, a consumer survey conducted by Consumer Reports would have been more accurate. Consumer Reports surveys its readership about automobiles, so why not for diet programs?
Otherwise, refrain from ranking diet programs until the right studies and trials have been conducted.
Metcalf's "Bottom Line":
No matter how we look at the numbers, Jenny Craig is still our winner, and still not for everyone. All of our research continues to point to the fact that the best diet for you is the one that you can follow for months or even years.Okay. I see how this works. The study in question was flawed and does not really measure what Consumer Reports is trying to measure, but Jenny Craig is still the winner.
On what basis?
Metcalf's staff worked very hard on this article and deserves an "A" for effort.
It's all about how hard you work at manipulating the numbers, making faulty comparisons, and omitting important information.
The sad part of it all: the mainstream media, like lemmings, have jumped on the bandwagon without doing serious fact checking, instead depending on Consumer Reports' "sterling" reputation, which goes to show how bloated, stupid, and lazy the media have become.
To give some due to the media, Jennifer LaRue Huget, in "Is that right? Jenny Craig’s the top diet?" (The Washington Post's Lifestyle Blog), has questioned the science behind the rankings:
...when you set out to rate diets (or dishwashers, or whatever), you have to establish some system for objectively [my emphasis] evaluating them and stick with the results it produces.Weight Watchers is, understandably upset over the ratings, raising many of the same issues I raised in my May 12 post Shame on You, Consumer Reports...
Metcalf's response to Weight Watchers:
I’m sorry that Weight Watchers seems to be so upset. I do have a high opinion of their science. But I can’t invent data that I don’t have.So, Ms. Metcalf, you would rather use faulty research which fails to measure what you set out to measure and not fully reveal your sources to your readers and then accuse Weight Watchers of whining because they had the audacity to call you out on your half truths and omissions?
--quoted in The Washington Post
(Shake head here.)
You and your "science" writers have just tanked Consumer Reports' credibility even deeper into the toilet.
As one might expect, Jenny Craig is using this ranking in their TV ads and on their website, perhaps drawing in new customers into an expensive program that might not be right for them.
Shame on Consumer Reports and Jenny Craig.
DISCLOSURE: I am a Weight Watchers' member, and I gave a copy of my May 12 posting (along with a copy of the Consumer Reports diet program rankings article) to my Weight Watchers' leader.
I do not pretend to be unbiased; I love Weight Watchers and know very little about Jenny Craig. However, in this case, I care more about accuracy in reporting the correct facts than I am in diet program rankings.