It is important to know the different kinds of studies that we rely on to know what works best. If you don’t understand this, you can easily be misled by books, articles and advertisements.
The weakest form of evidence is the testimonial. A testimonial is similar to endorsements you see on the cover of a book, where someone says something like, “This is a really great book.” These kinds of statements are common in advertisements for weight loss products. You may see comments such as, “I tried this product and lost 30 pounds without even dieting,” or, “I haven’t felt better in years since I started taking this stuff.” The problem with testimonials is that you can always find someone who seems to do better with any product. That is why placebos are used in clinical trials. Depending upon the outcomes measured by a trial, about 5% to 30% of people feel better when taking a placebo (while a similar number may feel worse due to side-effects such as upset stomach or tiredness). If I were to begin selling a new product that I just invented (let’s call it Slimy Joes Snake Oil Remedy), all I would need to do is find the 5% to 30% of people who try my useless product and have a positive effect and ask them to write a testimonial for me about how my snake oil remedy has made their lives so much better. I will have to ignore the much larger group of people who felt no benefit, or actually did worse with my totally useless product. But that is not a problem, because I get to choose which comments to put on my advertisements. So if you see a product endorsed with testimonials, run away. Though the people writing the testimonials may be completely sincere, it means that there is no real evidence that the product is effective.
Another type of study is the observational study. This type of study looks at large groups of people who have a condition, such as obesity, and compares them with people who don’t have the condition. These studies are very useful to generate ideas for future clinical trials, but are not useful for finding the cause of something. Observational studies can tell us that those who spend more time watching television or who drink more soft drinks are more likely to be overweight. But to say for certainty that they cause obesity, we would need to do a randomized controlled trial where we randomly divide a large number of normal weight people into two groups, where one group must watch a lot of television and the other group may not watch any. Then follow these groups for several years and see if one group gains more weight than the other. It is easy to do this with laboratory animals, but not with real people.
The trial that gives us the best evidence is the prospective, double blind, placebo controlled, randomized controlled trial (RCT). Prospective means that we decide in advance what we will be testing, before we do the experiment. Looking backward at what has already happened can give us ideas of what to test, but cannot assure us that it did not happen just by chance. For example, if I happen to notice that two of the five cars passing me on my way to work have New York license plates, I could wrongly conclude that my town in Colorado has been overrun by New Yorkers. But if I decide in advance (prospectively) that I am going to look for New York license plates on my way to work, I’m not likely to see any. Double blind means that the people in the trial and the scientists observing the outcome do not know who is receiving the trial intervention and who is receiving the placebo. This helps to reduce bias that can occur when we anticipate who will or won’t have a good response. Placebo controlled means that some people receive the intervention and some do not. Randomization means people are randomly assigned to be in either the test group or the control group. This is necessary to try to eliminate differences between the two groups, so that the only difference between them is the intervention being studied. In general, larger RCTs with more people enrolled are better, because the outcomes of the trial are less likely to have occurred only by chance.
Systematic reviews and meta-analyses are when researchers look at all of the studies done for a specific intervention and try to draw conclusions based upon all of the data. In this way, the results of several small studies can be combined to simulate a larger trial.
There are many ways that clinical trials can be done badly. Just because there has been a clinical trial does not mean that anything has been clinically “proven”. For practice in reviewing levels of evidence, let’s analyze the clinical trial that I found on the web site of a common weight loss herb. Most of the evidence that the site gives is in the form of testimonials, such as “I notice I was eating less at meal time and I wasn’t snacking in between meals.” Or, “I didn’t think about food and I did not feel deprived.” But there is one clinical trial mentioned on the web site. The trial was never published in a medical journal, so the only information that we have is what is on the web site. This alone should raise suspicion, because if it was a well-done clinical trial it should be publishable in a reputable academic journal. The trial lasted 4 weeks and included 26 participants. 19 took the weight loss pill and 7 took placebo. They were instructed to not change the types of foods they were eating and not to add any exercise. At the end of 4 weeks, “100% of clinical trial participants either lost weight or inches off their waistline.” The 19 people taking the weight loss pill lost an average of five pounds in 4 weeks, while the 7 people taking placebo lost an average of less than one pound. One participant lost 8 pounds and the same or another participant lost 3 inches of waist circumference.
What information does this trial give us? For starters, we know the participants were not randomly assigned to take either the weight loss pill or placebo; otherwise there would be equal numbers of participants in both groups. And since it is not mentioned, we can assume that the people being studied and the people interpreting the results knew what group they were in (so the trial was not blinded). Were the people in both groups the same? Were they normal weight or obese? Why were there almost three times as many people in the the weight loss pill group as the placebo group? Were some of the people who started the trial not included in the final results? Were the people in both groups already participating in diet and exercise to the same extent? We are not told the answers to these questions. And since we don’t have this information, it is very possible that in the trial some overweight dieters were assigned to take the product, while some normal weight non-dieters were given a placebo. Because they were told not to change what they were doing, the non-dieters taking the placebo were not likely to lose much weight. If any in the placebo group did happen to lose weight, they could be excluded from the reported results. The trial researchers could say that the non-dieters changed what they were doing, by eating less or exercising more, so they are disqualified. A trial like this would never be accepted for publication in a professional journal, but could easily be used in marketing to claim that a “clinical trial” has “proven” that the method is effective. Is this what happened in the clinical trial that the web site mentioned? We don’t know. But no independent physician would consider an unpublished trial like that to be useful evidence.
Weight loss products are very prone to promotion with outrageous claims. It is helpful to maintain a skeptical attitude.
The weakest form of evidence is the testimonial. A testimonial is similar to endorsements you see on the cover of a book, where someone says something like, “This is a really great book.” These kinds of statements are common in advertisements for weight loss products. You may see comments such as, “I tried this product and lost 30 pounds without even dieting,” or, “I haven’t felt better in years since I started taking this stuff.” The problem with testimonials is that you can always find someone who seems to do better with any product. That is why placebos are used in clinical trials. Depending upon the outcomes measured by a trial, about 5% to 30% of people feel better when taking a placebo (while a similar number may feel worse due to side-effects such as upset stomach or tiredness). If I were to begin selling a new product that I just invented (let’s call it Slimy Joes Snake Oil Remedy), all I would need to do is find the 5% to 30% of people who try my useless product and have a positive effect and ask them to write a testimonial for me about how my snake oil remedy has made their lives so much better. I will have to ignore the much larger group of people who felt no benefit, or actually did worse with my totally useless product. But that is not a problem, because I get to choose which comments to put on my advertisements. So if you see a product endorsed with testimonials, run away. Though the people writing the testimonials may be completely sincere, it means that there is no real evidence that the product is effective.
Another type of study is the observational study. This type of study looks at large groups of people who have a condition, such as obesity, and compares them with people who don’t have the condition. These studies are very useful to generate ideas for future clinical trials, but are not useful for finding the cause of something. Observational studies can tell us that those who spend more time watching television or who drink more soft drinks are more likely to be overweight. But to say for certainty that they cause obesity, we would need to do a randomized controlled trial where we randomly divide a large number of normal weight people into two groups, where one group must watch a lot of television and the other group may not watch any. Then follow these groups for several years and see if one group gains more weight than the other. It is easy to do this with laboratory animals, but not with real people.
The trial that gives us the best evidence is the prospective, double blind, placebo controlled, randomized controlled trial (RCT). Prospective means that we decide in advance what we will be testing, before we do the experiment. Looking backward at what has already happened can give us ideas of what to test, but cannot assure us that it did not happen just by chance. For example, if I happen to notice that two of the five cars passing me on my way to work have New York license plates, I could wrongly conclude that my town in Colorado has been overrun by New Yorkers. But if I decide in advance (prospectively) that I am going to look for New York license plates on my way to work, I’m not likely to see any. Double blind means that the people in the trial and the scientists observing the outcome do not know who is receiving the trial intervention and who is receiving the placebo. This helps to reduce bias that can occur when we anticipate who will or won’t have a good response. Placebo controlled means that some people receive the intervention and some do not. Randomization means people are randomly assigned to be in either the test group or the control group. This is necessary to try to eliminate differences between the two groups, so that the only difference between them is the intervention being studied. In general, larger RCTs with more people enrolled are better, because the outcomes of the trial are less likely to have occurred only by chance.
Systematic reviews and meta-analyses are when researchers look at all of the studies done for a specific intervention and try to draw conclusions based upon all of the data. In this way, the results of several small studies can be combined to simulate a larger trial.
There are many ways that clinical trials can be done badly. Just because there has been a clinical trial does not mean that anything has been clinically “proven”. For practice in reviewing levels of evidence, let’s analyze the clinical trial that I found on the web site of a common weight loss herb. Most of the evidence that the site gives is in the form of testimonials, such as “I notice I was eating less at meal time and I wasn’t snacking in between meals.” Or, “I didn’t think about food and I did not feel deprived.” But there is one clinical trial mentioned on the web site. The trial was never published in a medical journal, so the only information that we have is what is on the web site. This alone should raise suspicion, because if it was a well-done clinical trial it should be publishable in a reputable academic journal. The trial lasted 4 weeks and included 26 participants. 19 took the weight loss pill and 7 took placebo. They were instructed to not change the types of foods they were eating and not to add any exercise. At the end of 4 weeks, “100% of clinical trial participants either lost weight or inches off their waistline.” The 19 people taking the weight loss pill lost an average of five pounds in 4 weeks, while the 7 people taking placebo lost an average of less than one pound. One participant lost 8 pounds and the same or another participant lost 3 inches of waist circumference.
What information does this trial give us? For starters, we know the participants were not randomly assigned to take either the weight loss pill or placebo; otherwise there would be equal numbers of participants in both groups. And since it is not mentioned, we can assume that the people being studied and the people interpreting the results knew what group they were in (so the trial was not blinded). Were the people in both groups the same? Were they normal weight or obese? Why were there almost three times as many people in the the weight loss pill group as the placebo group? Were some of the people who started the trial not included in the final results? Were the people in both groups already participating in diet and exercise to the same extent? We are not told the answers to these questions. And since we don’t have this information, it is very possible that in the trial some overweight dieters were assigned to take the product, while some normal weight non-dieters were given a placebo. Because they were told not to change what they were doing, the non-dieters taking the placebo were not likely to lose much weight. If any in the placebo group did happen to lose weight, they could be excluded from the reported results. The trial researchers could say that the non-dieters changed what they were doing, by eating less or exercising more, so they are disqualified. A trial like this would never be accepted for publication in a professional journal, but could easily be used in marketing to claim that a “clinical trial” has “proven” that the method is effective. Is this what happened in the clinical trial that the web site mentioned? We don’t know. But no independent physician would consider an unpublished trial like that to be useful evidence.
Weight loss products are very prone to promotion with outrageous claims. It is helpful to maintain a skeptical attitude.