Copyright © 2007-2018 Russ Dewey
Pitfalls in Observational Research
So far observational research sounds good. It is relatively easy (all one has to do is collect data) and if one is fortunate enough to find strong correlations in the data, one can make powerful predictions.
Now we return to the theme of critical thinking. Observational research, like any human activity, has pitfalls or things that can go wrong.
Population validity problems
One big problem is population validity. Does data gathered from one population apply accurately to other populations? If so, the results have high population validity.
However, population validity cannot be assumed. We raised this issue on the page about correlation and prediction when discussing a negative correlation between alcohol consumption and grade-point average.
A strong correlation existed between drinking and low grades at the University of Illinois. A weaker correlation was found at Virginia Tech.
I suggested this correlation might not be found at all in France. If that turned out to be true, it would be an example of a population validity issue. Findings from one population would not apply to a different population.
As it happens, there is such a correlation with nationality, related to drinking. In France, drinking is almost universal, if the definition includes wine served with meals. Drunkenness is less common.
A study of alcohol affects among adolescents in Europe and the Americas (Kuntsche et al., 2013) suggests early drunkenness, not early drinking, predicts problems with alcohol. So a negative correlation between drinking and grades might not exist in France among moderate drinkers who have a glass of wine with dinner. The negative correlation between alcohol consumption and grades might re-appear if the data was restricted to those who drink enough to get drunk on a regular basis.
A potential population validity problem occurs when researchers generalize from animals to humans. Some species of mice are 10,000 times more likely to grow cancers than humans. Tests of cancer-causing substances using mice as subjects may have low population validity for humans. Results obtained with one group might not apply to the other.
Results collected in Germany may not apply to people in France. Data collected using a subject pool that draws students from introductory psychology classes might not be comparable to data that draws students from business classes.
What are examples of population validity problems?
For this reason, researchers should specify where and how data was collected. If a target population is well defined (for example, one is interested only in business majors at a particular school) the ideal is to obtain a random sample from that particular population.
Sometimes the target population is very general. A researcher may be trying to prove something about all humans. In that case, conclusions based on a one group of subjects must be regarded as tentative until the research can be replicated with different groups. Cross-cultural studies are highly desirable for this reason.
In cross-cultural studies, populations from different cultures are compared. Researchers who study abnormal psychology find it significant, for example, when a mental disorder occurs in only one culture, not others. This suggests that the disorder may involve a belief system or genetic or environmental factor peculiar to that culture.
How are cross-cultural studies informative to abnormal psychology researchers?
On the other hand, some mental disorders (like schizophrenia and depression) occur in every culture. That suggests they are due to biological factors or environmental hazards common to all humans.
Measurement and Observer Effects
In observational research, results can be changed or biased by the act of measurement itself. This is called a measurement effect.
One type of measurement effect is called an observer effect. It occurs when subjects alter their behavior because an observer is present.
For example, a developmental psychology student might decide to study parent/child interactions by bringing video equipment into homes and recording the behavior of parents and children. The presence of an observer is likely to change the behavior of both parents and children.
What are measurement effects? Observer effects?
Similar problems affect television documentaries about particular societies or cultures. The presence of a camera changes behavior.
One way to avoid observer effects is to use unobtrusive measures (Webb, Campbell, Stanley & Sechrest, 1966). With unobtrusive measures, subjects is not aware of being observed or tested. An unobtrusive study of parent/child interactions in a department store might use video records from security cameras.
What are unobtrusive measures, and why are they used?
Webb, Campbell, Stanley, and Sechrest told about a museum that wanted to measure traffic to its exhibitions. They did not want to bother visitors with a questionnaire.
Instead, they came up with a clever unobtrusive measure: measuring the amount of wear on floor tiles by each exhibit. When the tiles showed greater wear, the museum could assume more people were stopping there to look at an exhibit.
The German ethologist Eibl-Eibesfeldt used a form of unobtrusive measurement in his classic studies of native cultures. He took movies of cultures almost untouched by modern civilization.
To make people less self-conscious, Eibl-Eibesfeldt put a mirror in front of the lens of his camera. In that case, the camera photographed everything off to the side rather than things in front of it.
Eibl-Eibesfeldt would obtain permission from a family to film their everyday behaviors. Then he would point his camera away from them and start the camera.
How did Eibl-Eibesfeldt minimize observer effects?
The people knew enough about cameras to assume he was not photographing them if the camera was pointed a different direction. They "acted natural" and Eibl-Eibesfeldt was able to photograph them while they thought they were off-camera.
This was somewhat sneaky but probably essential for getting an accurate record of normal behavior without observer effects. Prof. Eibl-Eibesfeldt put the film archive of his right-angle camera documentary work on the web. It is full of fascinating material.
.---------------------
References:
Kuntsche, E., Rossow, I., Simons-Morton, B., Bogt, T. T., Kokkevi, A., & Godeau, E. (2013) Not early drinking but early drunkenness is a risk factor for problem behaviors among adolescents from 38 European and North American countries. Alcoholism: Clinical and Experimental Research, 37, 308-314. doi:10.1111/
Webb, E. J.; Campbell, D. T.; Schwartz, R. D. & Sechrest, L. (1966). Unobtrusive Measures. New York: Sage Publications.
Write to Dr. Dewey at psywww@gmail.com.
Don't see what you need? Psych Web has over 1,000 pages, so it may be elsewhere on the site. Do a site-specific Google search using the box below.