Public Health: Issues, Situation , and Challenges: Study Designs

experimental studies: RCTs and quasi-experiments

Study Designs

To Epidemiology theme page
To EBM theme page

The evidence for evidence-based medicine is all collected via research, which uses a variety of study designs. You will be learning about "critical appraisal of the literature," and judging the quality of a study design is a central part of this.

Different study designs provide information of different quality. Of course, we always try to use the best possible design, but sometimes this is not practical or ethically acceptable (you cannot do an experiment to expose some people to a harmful substance to see what effect it has). Therefore, you need to understand the strengths and limitations of each type of study design, as applied to a particular research purpose. The purposes we will consider include (1) describing the prevalence of health problems; (2) identifying causes of health problems (etiological research), and (3) evaluating therapy, including treatment and prevention.

Types of Study Design

First, distinguish between observational and experimental studies.

In observational studies, the researcher observes and systematically collects information, but does not try to change the people (or animals, or reagents) being observed. In an experiment, by contrast, the researcher intervenes to change something (e.g., gives some patients a drug) and then observes what happens. In an observational study there is nointervention.

Examples of observational studies:

a survey of drinking habits among students;

a researcher who joins a biker gang to study their lifestyle (note, as long as the researcher does not try to change their behavior, it's an observational study);

taking blood samples to measure blood alcohol levels during Monday morning lectures (yes, you are intervening to take the blood, but you are not trying to change the blood alcohol level: it's just a measurement).

Examples of experiments:

plying a law student with beer to see whether lawyers argue better when drunk;

encouraging bikers in one group to stop smoking those funny-looking cigarettes to see whether they get less belligerent;

warning one group of students that you are going to take blood alcohol levels next Monday to test for alcohol, and comparing their levels to another group that you did not warn.

When do you do an observational study?

When you merely want to collect descriptive information: "Is the incidence of diabetes rising?"

When you want to report on the causes of a problem without disturbing the natural setting (I want to find out why students do not attend lectures)

When you can't do an experiment: "How fast does the earth move around the sun?"

When it's not acceptable to do an experiment: "How much does not wearing a condom increase the likelihood of HIV infection?"

Observational Designs:

What types of observational study are there? Lots, but you need to know about three main ones:

Cross-sectional surveys. Example: what is the prevalence of diabetes in this community? Here, you draw a random sample of people and record information about their health in a systematic manner. You can also compare people with, and without, diabetes in terms of characteristics (such as being overweight) that may be associated with the disease. The problem is that you cannot be sure which came first: the diabetes or the weight problem, so this is a very weak design for drawing conclusions about causes.

Cohort, or "longitudinal", or "prospective" studies. These are like surveys, but extend over time. This allows you to study changes and to establish the time-sequence in which things occur. Therefore, you can use this to study causes. For example, you could draw a sample of people (medical students, for example) who do not have the disease you are interested in, and collect information on the factor you have hypothesized to be a cause of the disease. Maybe you want to see whether using a cell phone leads to brain cancer. So, collect information on how many minutes each student uses their phone each week (you might get permission to obtain this from their phone company bills), and collect this information over a long time, and then eventually collect information on who gets brain cancer. You could then see whether the cases of brain cancer arose among the people who used their cell phones most often. In technical terms, you record the incidence of cancer among those who use their phones more than a pre-determined amount and compare this to the incidence in the non-users. You could calculate the relative risk.
The advantages of this study design are that it can establish that the phone usage predates the cancer, and it allows for accurate collection of exposure information ('exposure' = their use of the phones). However, there are some problems with this design. Brain cancer is rare, so you will need a very large cohort of students; you will also need to keep in contact with them for a very long time and you will probably get very bored waiting for the results. We need a quicker solution.
Link to ppt diagram of a cohort study

Self-test question: cohort study

Can you can estimate prevalence from a cohort study? Yes

No

The "case-control" study. This is more practical, but suffers from other disadvantages. It is a "retrospective" study. This means that (like a detective) you begin at the end, with the disease, and then work backwards, to hunt for possible causes. In our example, you could identify a group of patients with brain cancer (these would be the cases in "case-control"). Then identify a control group who do not have brain cancer. [Technical detail: In fact, great care is needed in choosing the control group: should you select patients with other types of cancer, or healthy people, or both?] Then, collect information on their previous use of cell phones, dating back as far as you can manage. Again, you might be able to collect this "exposure" information from their phone bills. The hypothesis would be that phone usage would be significantly higher in the cancer group than the control group; after collecting the data you can test how well the data fit this hypothesis using a statistical test.
The advantages are that a case-control study can be done faster and more cheaply than a cohort study. However, it may be difficult to collect the information you require on past exposures, and there may be other ways in which the cases and controls differ, not just the cell phone use, which could also be causing the cancer. Sometimes you also have difficulty in being sure which came first: the disease or the exposure (the Law of Retrospection: "You cannot tell which way the train went by looking at the track").
Note that with a case-control design you can not calculate incidence of cancer (because the cases already had cancer when you began) and this weakens the analysis. Nor can you calculate prevalence, because it was you who decided how many cases and how many controls to choose, and this determined the apparent prevalence in the study.
Instead of the relative risk, you have to use a calculation called an "odds ratio" to estimate the association between phone use and cancer. But, because case-control studies are much more practical for studying the causes of many chronic diseases, they are used very commonly.
You may come across references to "matched" and "unmatched" case-control studies. A problem with case-control studies is that the cases and controls may differ on a number of factors, including characteristics (such as age, or sex, or wealth) that you are not considering as potential causes. To ensure greater comparability between the two groups, and thereby avoid confounding, the controls could be matched for sex and age to the cases.
Link to ppt diagram of a case-control study
Link to more on confounding.

Self-test question: case-control study

It says that you cannot estimate prevalence from a case-control study.
Why not? Click for an answer

Experimental Designs:

Randomized Controlled Trial ("RCT"), or "Randomized Clinical Trial". The mainstay of experimental medical studies, normally used in testing new drugs and treatments.

A sample of patients with the condition, and who meet other selection criteria, are randomly allocated to receive either the experimental treatment, or the control treatment (commonly the standard treatment for the condition).
Occasionally, a placebo or sham treatment will be used in the control group, but where there is already an accepted treatment, it is unlikely to be ethical to use a placebo.
The experimental and control groups are then followed for a set time, and relevant measurements are taken to indicate the results (or 'outcomes') in each group.
See the diagram:

Some more subtle points:

Be careful! The "random" refers to random allocation to either experimental or control group; it does not refer to randomselection or sampling of the patients to include in the trial.
Random allocation means allocating them by chance (e.g., the toss of a coin). As long as you have relatively large groups (50 or more people in each), This means that the two study groups will end up equivalent (comparable) in terms of factors such as age, sex, and even other things that you do not even know about (such as their reaction to the medication).
Important examination point: Do not confuse random allocation to experimental and control groups with random selectionof a sample. Random selection of a sample ensures that the sample is representative of the broad population; it is typically used in a survey (i.e., an observational study). Random allocation ensures the experimental and control groups are equivalent, but does not ensure they are representative of the broad population. Indeed, they are most likely not, as they all have the disease being studied.
Why do we use random allocation? This is mainly to avoid confounding. Confounding refers to confusing the effects of two or more variables – here the treatment you want to study and some other factor, such as age or sex, on which the 2 groups might differ. To make sure that any differences in the final outcome measurements were due to your experimental treatment and not to something else, you want the two groups to be comparable on all other factors (in other jargon, you want to control all other factors). In theory, if randomly allocated groups are sufficiently large, they will be equivalent (so, directly comparable) on any variable you care to measure.
- Of course, if you know about a confounder before beginning the experiment, you could match the two groups on it (e.g., ensure equal numbers of males and females in each group). However, matching would not remove the effects of a confounder that you do not know about, such as a biochemical parameter that modifies the action of the drug. Herein lies the genius of random allocation: randomization protects against all potential confounding factors, both known and unknown. This is very convenient: you don't have to measure and control for each factor individually!

Because of natural biological variations, the two groups (even though randomly allocated) will not be absolutely, perfectly identical. Therefore, a statistical test is used to indicate whether any difference you observe in the outcomes for the two groups may just have reflected natural variability ("been due to chance alone"), or whether it seems to represent a "statistically significant" difference. Which means that it was very, very unlikely to have been due to chance differences between the 2 groups you compared. Examples of statistical tests include a t-test, or analysis of variance (ANOVA).
Link: More on statistical tests.
RCTs can be used to test preventive interventions. Here, analyses can record several statistics:
Absolute Risk Reduction = ARR = Cumulative incidence in control group minus cumulative incidence in the experimental group.
You can also calculate the Relative Risk Reduction, which is ARR divided by Cumulative Incidence in the control group.
You may also come across the Number Needed to Treat (NNT), which is 1/ARR, or 1/(CI control - CI treatment). This indicates the number of patients you need to treat to get one 'cure'.

Self-test question: RCT
	True or False?A randomized controlled trial begins with a random (i.e. representative) sample from the population of interest.	True False

Quasi-Experimental Designs:

Finally, there is a category of studies that falls between observational and true experimental studies; they are called "quasi-experimental studies". In these, there is an intervention, but it is often not completely planned by the person doing the research. An example would be a study of the effects of removing ophthalmic services from the OHIP billing schedule: is there a decrease in eye tests after the change? A quasi-experimental study might record the number of eye exams per thousand population over the years up to the policy change, and compare this pattern with the pattern afterwards. This is an observational study, but there was also an intervention, although it was not the experimenter who decided when and how the change would occur and to whom it would be applied, so this is a "quasi-experiment." Typically, random allocation is not involved. A "natural experiment" is similar, but refers to naturally occurring events (e.g., a study of mental health following an earthquake).

Summary of designs, showing advantages and disadvantages of each: