The survey is a method of data collection which can be used for any of the three types of research: field experiments, empirical studies and descriptive studies.
Analytical studies are usually empirical. They ask why certain situations exist.
Advantages:
Disadvantages:
Constructing survey instruments is difficult. For validated instruments check the Mental Measurements Yearbook now available on-line through the KU Library.
If you must construct your own instrument, consult several sources.
Types of questions:
1. Open-ended, good for problem definition purposes.The instructions are critical. ALWAYS test them on a sample similar to your test sample.2. Close-ended, force responses from a list. Easy for the researcher to analyze and the subject to fill out, but may overlook some interesting responses. Always precede by a pilot study using open-ended questions.
Mail surveys are NOT cheap! Tabulation is also tedious and subject to error. It is, however, possible to construct surveys in CGI script or using Front Page Extensions which will deliver responses either to your e-mail account, or even directly into an Excel file or database.
There are now companies which will translate your questions into a Web format with "radio buttons," host this survey on their server, and e-mail you the results in ASCII or Excel format, removing the problem of data entry errors. One such company with reasonable rates is Web-Online-Surveys.com.
Guidelines for questionnaire construction:
1. Make questions clear (consider target audience).Guidelines for increasing response rate:2. Keep questions short.
3. Keep to purpose of your project.
4. Avoid double-barreled questions (two in one).
5. Avoid biased words or terms.
6. Avoid leading questions.
7. Avoid questions that ask for too much detail.
8. Avoid embarrassing or personal questions when possible and place them near the end of the survey. Basic demographic information, especially age, are often more sensitive than the researcher realizes. By convention all demographic information is placed at the end of the survey for this reason.
9. When asking for ranking, specify the direction of the ranks.
10. Keep direction of Likert scales consistent (usually more favorable to right).
11. Keep types of responses consistent (i.e. always use boxes or items to be circled).
From a 1989 study:
Telephone Surveys--if in a local area--are probably cheaper than mail and are much quicker. The advent of answering machines and caller ID, however, makes it very hard to get a viable sample.
You should never use a public phone book as a frame because of the high number of unlisted numbers and the fact that you will miss many members of upper demographic groups with this technique.
From the phone book, make a rough determination of the percentage of numbers in the target area that begin with each 3 digit prefix. Then using a 4 digit random numbers table, call the number of subjects you have pre-specified, making sure the percentages for each 3 digit prefix are accurate. A random numbers table of 1,000 unique 4 digit numbers will be provided by the instructor.
It is best to use outsiders to actually make the calls so that there is no experimenter bias. They should be neutral. With multiple callbacks you can get a very high response rate. People are suspicious, however.
The personal interview is probably the technique most capable of getting inside people's heads and finding out the why of their behavior. Selection and training of the interviewers is critical for this technique to work. The interviewer may bias the study unavoidably in certain cases.
All survey research opens up questions of how honest people are. Social Psychology studies suggest that they are more honest than one might expect, when they feel their responses will be confidential. For this reason avoid code numbers on the survey forms if at all possible!
Experimental research is popular because it can suggest cause-effect relationships between variables which can not be inferred from empirical research.
It is more likely that lab research will FOLLOW field research rather than leading to it. We can make suppositions concerning causation from observation and then test these in the lab.
In the "lab" we control subjects and variables to an extent never possible in the field.
The field of experimental design is complex and is summarized in the text. Much research can be done with just the pretest-posttest control group design. Note that the t-test is used to analyze data from this design when the dependent variable measure is at the interval or ratio level.
Let's take a look at some of the most popular experimental designs. Most texts assume random assignment to groups, so omit the "R" used in our text. Whenever we observe or measure behavior, this is indicated by an "O" often accompanied by a subscript number to indicate a sequence of observations or measurements. An "X" is used to indicate some experimental manipulation of a group such as showing them a video, etc.
The pretest-posttest control group design requires random assignment of subjects to 2 groups, experimental and control. We pretest each group for the desired behavior, attitude, etc. and then expose only the experimental group to the experimental variable. Both groups are then tested again with a similar (or even identical) posttest. A t-test would be conducted between the 2 posttest scores, assuming that we are measuring the dependent variable at the interval or ratio level. The process is diagrammed as follows:
Pretest-Posttest Control Group Design
Experimental Group O1 X O2 --|
> t-test
Control Group O1 O2 --|
If you suspect that the random assignment of the 2 groups is imperfect, a t-test can be conducted between the O1 scores for the 2 groups, but this is usually not necessary if you have been careful in your system of randomization.
Sometimes pretesting subjects can introduce unwanted bias, so with very careful control over the random assignment of subjects to groups, the posttest-only control group design may be preferable.
Posttest-Only Control Group Design
Experimental Group X O1 --|
> t-test
Control Group O1 --|
The Solomon four-group design combines the two above and can help control for the effects of pretesting, but it requires a lot more subjects and may not be worth the effort for many studies.
Designs with more than one independent variable are called factorial. The analysis of variance (of which the t-test is a special case employing only 1 ind. variable with exactly 2 values) would be used if the measure is at the interval or ratio level. Factorial designs are beyond the scope of this course.
Time series designs are not truly lab designs because you cannot adequately control all variables over a long period of time. I would call this field research. It is still experimental, however, as long as the researcher is manipulating variables.
Most field research is empirical, but it is possible to do experimental research in the field.
The main advantage over lab research is in external validity. The subjects are being studied in a natural setting and in many cases are not aware of being observed at all. This is useful for studying complex processes.
Ethical considerations come into play in field experiments, however, and it is also difficult to control intervening variables.
Empirical field studies may be thought of as passive manipulation of variables.
Empirical (observational) studies are more concerned with description and explanation that quantification, and can be very valuable. The role of the researcher can vary from total observation to total participation. Ethical concerns must be considered BEFORE the study is conducted.
Field studies make excellent pilot studies.
Observations should be cross-validated by other observers.
Note that video can be used to collect data which can then be analyzed by many raters. Raters should use check sheets for some rudimentary quantification. There is a degree of flexibility in the data analysis for this type of study.
Focus groups are quick ways to get some insight into why people behave in certain ways. While looked down upon by many academic researchers, they are good adjuncts to other studies as they may help to explain the results. The group leader is critical and should not be ego-involved in the study.
Panel studies are expensive and time-consuming, but can yield
valuable results. They can establish a consensus on a
topical issue (ex. the formulation of a non-broadcast
curriculum for college students using the Delphi model).
Return to TVR 400 Home Page