Thought for the Day--May 12, 2011: Shame on You, Consumer Reports and How About a Refresher Course in Research Methodology 101?

There are three kinds of lies: lies, damned lies, and statistics.

--Attributed to Sir Charles Wentworth Dilke (1843–1911)

What gave [Jenny Craig] the edge over the big names we assessed--stalwarts such as Atkins, Ornish, and Weight Watchers--was a 332-person [actually, it was a 442-person study], two-year study of the program published in the Oct. 27, 2010, Journal of the American Medical Association. [Title of article, which was omitted from CR's ratings or its accompanying article: "Effect of a Free Prepared Meal and Incentivized Weight Loss Program on Weight Loss and Weight Loss Maintenance in Obese and Overweight Women," by Cheryl L. Rock, PhD, RD, et al, and published online on Oct. 9, 2010.]

...Jenny Craig's prepared food was decent, though not great, as we noted in "Diet Taste-off" in our February 2011 issue.

--"Pick your ideal diet," Consumer Reports, June 2011, page 14, after selecting Jenny Craig as the number 1 diet (Score: 85) in the diet wars.

Seriously?

I see a conflict here. The message seems to be, "While the food isn't great, Jenny Craig is the best commercial diet."

Huh?

From the beginning, I smelled something "off" about your selection of Jenny Craig as the First Place winner in the diet war rankings, raising many red flags and questions:

--Why on earth would a dieter select a program that requires its members to eat mediocre food, often for long term?

--How can such a diet be sustainable for life?

--What about life-time support for maintenance?

--Will Jenny Craig's members be eating its mediocre food for the rest of their lives?

--If not, how will members transition successfully into a maintenance program?

So I did a little digging of my own, and my suspicions were right.

First off, your readers need to know upfront that Jenny Craig itself was involved in the genesis of this particular study and participated (albeit minimally) in the design and program protocols for the study. Moreover,

Jenny Craig provided, free of charge, all pre-packaged food and all center, counseling, and phone support to about two-thirds of the participants within the study.

(A control group did not receive any Jenny Craig foods or support.)

Other research costs were assumed by The School of Medicine, University of California, San Diego.

Jenny Craig had no role in data collection, analysis, and publication of the article.

These facts are explained clearly in the research notes at the end of "Effect of a Free Prepared Meal and Incentivized Weight Loss Program on Weight Loss and Weight Loss Maintenance in Obese and Overweight Women," on page 1810.

In other words, Consumer Reports "edged" out other diet programs in favor of Jenny Craig, basing its assessment on a study that was proposed and supported, at least in part, by Jenny Craig, Inc.

Shame on you, Consumer Reports. You are capable of so much better than this bogus ranking of diet programs. At best, your research methods are shoddy; at worst, they are deceptive and will fool many of your readers into believing that Jenny Craig is the Holy Grail of diets when, in fact, it may just be another diet that requires its members to buy and eat pre-packaged foods.

Aside from the issue of Jenny Craig's corporate support, Consumer Reports also failed to disclose that the authors' primary purpose was not to rank diet programs per se or to establish Jenny Craig as the "best" program, but to accomplish the following objective:

To test whether a free [My emphasis] prepared meal and incentivized structured weight loss program promotes greater weight loss and weight maintenance at two years compared with usual care. (page 1803)

Also, the authors may be asking the following underlying question, which may reveal the real purpose of the study:

If insurance companies covered both the cost of diet programs and the pre-packaged diet food, would clients be more likely to stick with the program long term?

Indeed, in their conclusion, the authors suggest that based on strong compelling evidence, insurance companies should consider covering all diet program costs (and not necessarily just Jenny Craig), including pre-packaged diet foods. (1810)

The results were short of amazing: For groups 1 (Jenny Craig diet center support) and 2 (Jenny Craig telephone support), in which the participants were given a total free ride (both food and full program support), adherence was high. After two years, Group 1 had maintained an average loss of 16.28 pounds (7.4 kg), and Group 2, an average of 13.64 pounds (6.2 kg). On the other hand, Group 3 (the control group, which received "usual care"--handouts, diet plans, exercise advice, etc.--and no freebies from Jenny Craig), had maintained an average of only 2 pounds (2 kg) (page 1807)

While Group 1 and 2's astounding adherence results seem to accomplish, in the affirmative, the goals of the researchers' stated objective, this particular study does not support Consumer Reports' rationale for crowning Jenny Craig as "the best" weight loss program, for the study's objective does not match Consumer Reports' stated objective: ranking diet programs.

A researcher might have achieved similar results with Second Place Slim-Fast (Consumer Reports score: 63) and Third Place Weight Watchers (Score: 57).

The article also concedes some limitations of this study, for example,

The [strong] results may be related in part to the economic benefits to the participants of providing food, as well as reimbursement for participating in clinic visits, and the low dropout rate in this study contrasts with the high attrition rates reported among weight loss program cohorts. (1809)

I don't know about you, but if my insurance company or a researcher, over a two-year period, paid for my food and program and other support costs, you're damn right I'd adhere to the program.

Have you checked out the cost of food lately?

In addition, paying for program support is not cheap (as I have discovered in the past 4 months), so a client has to have some financial means to afford the weekly fees, often for months or even years, no matter what commercial program one selects.

In short, by using this particular study as evidence for Jenny Craig's superiority in the diet wars, Consumer Reports compares apples to oranges, playing sleight of hand with the facts.

While the CR source seems sound on its own merits--that is, independent of how CR used it--I have some other concerns regarding the study's objectivity:

--Dr. Cheryl L. Rock (Moores UCSD Cancer Center, La Jolla, CA) and lead author, had, during 2003-2004, served on the advisory board for Jenny Craig. Indeed, Dr. Rock had been approached by Jenny Craig about the possibility of such a study. (1810; TheHeart.org )

--Obviously, Jenny Craig's financial support suggests that this was a "corporate study," not necessarily a negative thing, for corporations develop studies and trials for myriad reasons, such as testing products and services and marketing purposes. But Jenny Craig's hand in the funding aspect does bring into question a probable bias toward Jenny Craig.

--This was not a blind study. In other words, the staff at Jenny Craig knew which members were participating in the program as research subjects, although the staffers were directed to treat the participants as paying customers. (page 1805) However, one has to wonder if the research participants were treated better than ordinary customers because of the stakes involved in the trial outcome. For example, were the research subjects called or e-mailed more often when they didn't show up for meetings or didn't call in for counseling sessions? We don't know, of course, but a blind study would have offered more credibility.

--The sample (442 women) was relatively small, and all participants were women.

Finley et al (2007) conducted a 2001-2002 study, involving 60,164 commercial diet-center participants; this larger study might offer more realistic results: 73% adherence after 4 weeks, 22% after 26 weeks, and only 7% after 52 weeks, a far cry from the 92% after 24 months reported by the Rock et al trial. (Rena R. Wing, PhD, "Editorial: Treatment Options for Obesity: Do Commercial Weight Loss Programs Have a Role?" Journal of the American Medical Association (JAMA), Vol 304, No. 16, Oct. 27, 2010, page 1837)

--The study included only one commercial diet. Dr. Rena Wing (Department of Psychiatry and Human Behavior, Warren Alpert Medical School, Brown University, and Weight Control and Diabetes Research Center, Miriam Hospital) believes that the Rock study offers "a best-case scenario" and that a larger study, involving several commercial weight loss programs, should be conducted "to examine whether providing these programs free of charge to participants would be a cost-effective approach" to the obesity epidemic. (page 1838)

Wing said nothing about Jenny Craig being a superior program.

--It is unclear if a maintenance plan was included in the 2-year study.

The authors' conclusion suggests "no":

Compared with usual care, this structured weight loss program resulted in greater weight loss over 2 years. (Rock et al, page 1803)

Finally, let's consider Second Place Slim-Fast and Third Place Weight Watchers and try to figure out why they were ranked as such.

According to the Slim-Fast chart, it should rank very low in the ratings: very little program support (other than a website) and, long-term, a huge dropout rate. However, one can buy Slim-Fast products at the grocery store and access to the website is free. Therefore, the financial stakes are fairly low. ("Pick your ideal diet," Consumer Reports, June 2011, page 14)

Weight Watchers was nudged into third because the sample menu included soup, which is high sodium. (page 14)

Really?

Anyone who has been a Weight Watchers member knows that a dieter is not limited to one sample menu, that soup is just one weapon in the arsenal. Also, for those who are sodium sensitive, there are lower sodium soups on the market, so one can have her soup and eat it. Also, one is not required or pressured to buy Weight Watchers' products, most of which can be purchased at the grocery store, and, with coupons and sales, can be purchased for reasonable prices. However, the fact that the Weight Watchers program is somewhat expensive may be a major strike against it. As a member, I tend to agree.

My main beef is not with the Consumer Reports rankings per se--we can agree to disagree--but I question CR's use of an inappropriate source to rank a program higher than it should be ranked, thus misleading readers into believing that Jenny Craig is offering a superior service, when, in fact, it may just be another program that has a poor long-term prognosis for success.

Consumer Reports' diet program ratings source = FAIL
____________________

References
Rock, Cheryl L., et al. "Effect of a Free Prepared Meal and Incentivized Weight Loss Program on Weight Loss and Weight Loss Maintenance in Obese and Overweight Women," Journal of the American Medical Associlation (JAMA), Vol. 304, No. 16 (Oct. 27, 2010, first published online on Oct. 9, 2010), pages 1803-1811. Downloaded on May 11, 2011 from jama.ama-assn.org at York College of Pennsylvania.

Wing, R.R., "Treatment Options for Obesity: Do Commercial Weight Loss Programs Have a Role?" Journal of the American Medical Associlation (JAMA), Vol. 304, No. 16 (Oct. 27, 2010, first published online on Oct. 9, 2010), pages 1837-1838. Downloaded on May 11, 2011 from jama.ama-assn.org at York College of Pennsylvania.

Wood, Shelley. "Free Food, Some Oversight, and a Plan: Popular Commercial Diet Works in Randomize Trial." TheHeart.org. October 11, 2010. Accessed on May 11, 2011.

NOTE: To access these articles, you may have to pay a fee. However, you may be able to access them through your school or public library.

____________________

DISCLOSURE: I am a Weight Watchers member, who believes strongly in the efficacy of the program. However, I am NOT affiliated with the corporate aspect of Weight Watchers.

In addition, this article is not meant to disparage the Jenny Craig Program, which may be a fine program; I have never been a Jenny Craig member. My issue is with Consumer Reports and its research source for its rating of Jenny Craig.

In the interest of getting this response to the CR ratings (which I first read on May 9, 2011) posted as soon as possible, I am publishing this in draft form, which I will rework and revise later.
____________________

Due to Blogger's major outage on May 12-13, 2011, this is an edited repost of an earlier post, which was wiped out (except as an incomplete draft). Fortunately, I had saved most of it offline.

Lesson here: Always save your blog posts offline, including your html codes.