Study Validity

Study validity refers to the accuracy of the study’s estimate of the relationship between the exposure and disease. Accuracy is distinguished from precision, which is a function of random error in the measurements in the study (which is inevitable). Accuracy is affected by systematic error, rather than random error. Systematic errors are called biases. Biases and Confounding are threats to internal validity because it leads to an incorrect estimate of the relationship between exposure and disease.

Selection Bias in Case-Control Studies

Selection bias occurs in a case-control study if the cases and/or controls are not selected from the same population as the population from which the cases came. For example, using a control group from a hospital often leads to an underestimation effect because risk factors are often higher in hospitalized populations than in the general population, which is where the cases originated.

Healthy Worker Bias

This is a type of selection bias in which risk of occupational exposures (e.g. toxins) is assessed by comparing exposed workers to a general population composed of workers and non-workers. Because non-workers are less healthy than workers, this makes the exposed workers look more healthy than the general population. Exposed workers must be compared to only workers in the general population to get an adequate measure of the risk of exposure.

Misclassification Bias

This is a bias which results from misclassification of the exposure or disease. It is also called information bias or measurement error. It can be random or non-random. Non-random, or differential misclassification, results in effects that are biased in one group compared to another. Types of differential misclassification biases include:

Recall bias
Detection bias (surveillance bias)

Random, or nondifferential misclassification results when both groups may be incorrectly measured. This results in effects that are closer to the null. That is, random misclassification dilutes any effect.

Confounding

A confounding variable is a variable that is associated with both the exposure and disease and could provide an alternative explanation for why an exposure is associated with a disease in a study. A variable is not considered a confounding variable if it is not associated with both exposure and disease. Also, if a variable is considered to be in the causal pathway between the exposure and disease, it is not considered a confounding variable.

A mediator variable is one that sits on the causal pathway. In Beth’s video, high blood pressure would be a mediator variable.

Dealing with Confounding

Confounding can be addressed in a number of ways. In a cohort study or an RCT you can restrict your inclusion criteria so that only one level of a confounding variable is included in the study (e.g. restrict the study to only men or women). You can also stratify your analysis by levels of the confounding variable. The best way to control confounding variables is to randomize people into groups. That controls for unknown confounding variables also. That is why the RCT is considered a strong study design.

Beth mentions intention to treat analysis at the end of the video. So let’s look at our exercise study where we asked one group to exercise (the intervention group) and the other to continue their normal behavior (the control group). Perhaps all the men dropped out of the study, so now we no longer have balance of men & women between the control and intervention groups. So we count the men who dropped out ANYWAY even though they didn’t finish. We “intended to treat” them (with an exercise regimen). Perhaps they dropped out because they were getting chest pain from heart disease, so not counting them would be missing the outcome we were trying to measure in a lot of people.

Adjusting for Confounding Variables

To adjust for confounding variables you calculate your effect in each level of the confounding variable separately, and then later combine for an overall adjusted effect. In the example in this video, there are more young people in the treatment group than in the control group. The RR for treatment is 0.67, suggesting a protective effect of treatment. However, the lower event rate in the treatment group is actually due to the fact that there are more young people in the treatment group. The event rate is lower for young than for old. In fact, for both young and old patients the event rates are equivalent in treatment and control groups. But because the event rate is lower for young people and there are more young people in the treatment than control groups, it appears that the event rate is lower in the treatment group. Once you adjust for age by calculating the RR separately for young and old, you see that there is no effect of treatment – the RR is 1.0 in both old and young.

TED Talk on Publication Bias by Ben Goldarce

When a new drug gets tested, the results of the trials should be published for the rest of the medical world — except much of the time, negative or inconclusive findings go unreported, leaving doctors and researchers in the dark. In this impassioned talk, Ben Goldacre explains why these unreported instances of negative data are especially misleading and dangerous.