o About Survey Instruments: A Brief Introduction

For students, and those less experienced, what follows is intended to explain what surveys do and to help one understand when and how to use them. It is not a comprehensive tutorial, but rather an introduction with pointers to more detailed sources. It also includes a bibliography on materials about surveys, their construction, and their use.

Epistemologically, surveys provide one way of obtaining and validating knowledge. Consequently, it is important that the context for this type of IS research be considered. Hence, this section also provides a summary that explains when survey research is appropriate and the steps required to use the survey approach correctly.


[What does Survey Research try to Achieve?] [When is Survey Research Appropriate?]
[How are Surveys Developed and Used?] [Can Surveying be Automated?]
[ISWorld Net Navigation Map]




What does Survey Research try to Achieve?

First, one should understand what a survey is actually used for. A survey is a way of going from observations to theory validation. As part of a panel discussion (Newsted, Chin, Ngwenyama, and Lee, 1996), Chin presented an exhibit that nicely outlines how this is done:

Observed responses become data on single questions.
These questions are aggregated into scales.
Appropriate numerical formulas are applied to these numbers.
The results of these formulas lead to conceptual representations of what has been measured.

These concepts are the constructs of interest to researchers. The usual objective for Information Systems researchers using this approach is to determine the relationship of these constructs as a way of making sense of behavior surrounding and involving IS.

The methodology used to study these relationships is also an important aspect of survey research. Techniques range from the reporting of simple scale means, the use of analysis of variance of results in different conditions, through regression analysis and the analysis of paths between constructs, to the use of "second generation" techniques such as LISREL and PLS, which analyze measurement models and structural models simultaneously.

Trochim's (1998) web pages show how some of the simpler of these approaches work. It has links dealing with:

How to select basic statistics.
A useful glossary of statistical terms.
Links to a variety of other statistical sites.
Samples of applying various techniques to the area of web page evaluation.

Chin's (1995) website provides pointers toward more advanced approaches using structural equation modeling (SEM). Among other things it discusses:

The need for a covariance matrix to show the relations among constructs and variables.
Model specification and graphical ways of representing a model.
Steps involved in using structural equation models.
A link describing the Partial Least Squares Approach (PLS) to SEM.
An extensive set of references using the PLS approach

Return to the top of the Survey Development Introduction





When is Survey Research Appropriate?

In a panel discussion at the 1996 International Conference on Information Systems, Newsted, Chin, Ngwenyama, and Lee (1996) raised the question whether surveys had outlived their usefulness. The panelists concluded that surveys are appropriate in certain conditions and less useful in others. But this is more complicated than just saying surveys should be used only in objective or positivist research and not in more subjective or interpretivist research. Burrel and Morgan (1979) provide a detailed distinction between these approaches.

In the '96 panel, Allen Lee indicated that in positivist research, surveys are particularly useful in determining the actual values of variables under study, and the strengths of relationships among them. In an interpretivist context surveys are appropriate as a complement to other forms of data or observations. They can serve as a way to add to one's knowledge through "triangulation" as one of several methods. Thus it is important to realize that while surveys are typically used in quantitative research, they can also help qualitative researchers as well. The panel gives further details. In particular one should review the notes below the slides in Lee's presentation as they discuss the inter-relations between these types of research and describe situations where these approaches and orientations fit together. For those interested in the qualitative approach, Myers provides an excellent site offering considerable detail on this type of research.

Overall the survey approach can be seen to have the following strengths and weaknesses:

Strengths

Surveys are easy to administer.
Surveys are simple to score and code.
Surveys determine the values and relations of variables and constructs.
Responses can be generalized to other members of the population studied and often to other similar populations.
Surveys can be reused easily, and provide an objective way of comparing responses over different groups, times, and places.
Surveys can be used to predict behavior.
Specific theoretical propositions can be tested in an objective fashion.
Surveys can help confirm and quantify the findings of qualitative research.

Weaknesses

Surveys are just a snapshot of behavior at one place and time.
One must be careful about assuming they are valid in different contexts. In particular, different cultures may produce different results. Kettinger, Lee, and Lee (1995) provide a good example of this by showing the effect of cultural differences in the measurement of IS service quality.
They do not provide as rich or "thick" description of a situation as a case study.
They do not provide as strong evidence for causality between surveyed constructs as a well designed experiment.

Return to the top of the Survey Development Introduction





How are Surveys Developed and Used?

After having decided to employ survey research it is advisable to search the literature for measures of your constructs of interest. This process can be expedited by reviewing our list of constructs or using the search engine. If you are unable to find an appropriate instrument with an associated methodology, it may be necessary to develop one. To do so requires a theoretical foundation in the area of study and an understanding of the survey process. But, even when an existing instrument is being used, an appropriate theory and a correct process are necessary.


Theory

A carefully constructed theory is a precursor to the actual use of an instrument. Theories propose constructs and their expected relations. They guide the investigation of these relations as they attempt to help one understand behavior and identify regularities in it. However, this is also a two-way street: instrument development can often refine theories as well. As Straub (1989) indicates, "Attention to instrument issues ... brings greater clarity to the formulation and interpretation of research questions. In the process of validating an instrument, the researcher is engaged, in a very real sense, in a reality check. He or she finds out in relatively short order how well conceptualization of problems and solutions matches with actual experience of practitioners." (p. 148).

A number of exemplary survey-based studies describe instrument development within a theoretical context. In the following two studies researchers have taken particular care to consider theories and to identify the steps in instrument creation and validation:

Straub (1989) uses a detailed example in the measurement of computer abuse to discuss instrument validation and its importance in confirming theories.

Moore and Benbasat (1991) describe the development of multiple constructs to measure IT adoption. In particular they provide detail on item creation and the derivation of items based on the theory of innovation diffusion.


Process

With a theoretical foundation in place, the activities of the survey process can be considered. Grover (1997) provides a detailed check list to be followed in the development and use of an instrument. Significant among these steps are:

determination of the unit of analysis (e.g., the individual, group, or organization)
creation and use of multi-item scales
pre-testing and use of pilot data
assessment of both construct and content validity
assessment of reliability
random sampling from a defined sample frame
determination of an appropriate response rate and evaluation of nonresponse bias
assessment of whether significant correlations imply real causal relations
determination of statistical power of the final analysis.

Dillman (1978) offers a number of very practical tips to facilitate the surveying process and increase response rates. Salant and Dillman (1994) provide a very good update to Dillman's initial classic.

The following references provide more detail on the theories and processes involved in surveying. These are divided into sources specific to IS as well general references on surveying. A broad understanding of survey methodology as well as knowledge about specific IS issues are useful in using the survey approach.

General References on Survey Research
IS-Specific References on Survey Research

Return to the top of the Survey Development Introduction





Can IS Surveying be Automated?

Since surveying involves a number of detailed and sometimes tedious steps, it is reasonable to ask if parts of it can be automated (especially since much of the analysis of survey data has been automated). The answer is a tentative "yes." However, care must be taken to ensure that computerized surveying leads to the same results as traditional, noncomputerized approaches using paper and pencil.

Bratton and Newsted (1995) summarize much of the work in this area and show that differences in results can be due as much to different interfaces as to differences in techniques. Interestingly, fewer differences seem to be found in newer work (Liefeld, 1992; Potosky and Bobko, 1997), as compared to studies before 1990 (e.g. Liefeld, 1988; Kiesler and Sproull, 1986). Recently, however, Webster and Compeau (1996) have found that, while there may not be differences in means between computerized and noncomputerized surveying, there may be differences in correlations with other variables such as those involving training performance. While there appears to be a growing acceptance of automated survey methodologies, there is a need for caution in the use of computerized techniques.

Automated surveying can be done using the Internet. Mueller (1997) provides a good example of how surveys can be done on the Web. He uses an instrument that is part of his on-going research on computer experience.

There are a number of commercial companies that provide computerized surveying for a fee. While we have not tested these products, they appear to have useful capabilities -- perhaps, especially for those wishing to do commercial surveys in IS.

QuestionMark
Quiz Factory
Survey Said
ScreenSurvey from the National Research Council of Canada
Surveyor Manager (in French) créez, diffusez et évaluez des questionnaires.
Jeffrey Dodgson's List of Survey Software

Return to the top of the Survey Development Introduction





o ISWorld Net Navigation Map

The ISWorld Research Home Page

Survey Instruments in IS

About Survey Instruments: A Brief Introduction [where you are now].

Return to the Table of Contents