Industry News AAPOR Proposes Standard Definitions for Calculating Survey Outcome Rates At its annual meeting in St. Louis in May, the American Association of Public Opinion Research (AAPOR) proposed standard definitions for assigning final disposition codes and calculating outcome rates that would be used by all researchers in reporting survey results. The purpose of the standard is to promote the consistent reporting of study and poll results so that meaningful comparisons can be made across studies and studies can be replicated. AAPOR's proposed standard includes: a comprehensive set of final disposition codes and four sets of CASRO-like outcome-rate formulas based on the number of cases assigned each final disposition. To arrive at the standard set of final disposition codes, two-dozen classification schemes were reviewed. According to the committee that developed the standard definitions, none of these schemes was exactly alike. Neither the categories of disposition codes nor the calculations of outcome rates were consistently defined. According to the committee, this makes both comparisons of performance across studies and the interpretation of measures (such as non-response rate) difficult and of limited value. The committee concluded that what's needed is a set of standard definitions analogous to those used in accounting by banks and businesses. The standard definitions proposed at the May meeting do not apply to all survey modalities, only to random-digit-dialed (RDD) telephone and in-person household surveys. In addition, the definitions do not cover surveys that call for hierarchical sampling units, such as those where multiple members of a household or multiple business units within a company are canvassed. Proposed Definitions for Disposition Codes The proposed definitions for final disposition codes divide sample cases into four groups: Interviews Eligible cases that are not interviewed Non-eligible cases Cases of unknown eligibility Interviews are further divided into two groups: Completes and Partials. Although the Interview definition does not state the difference between a Complete and a Partial, it does recommend that each study have an explicit definition of what distinguishes a Complete from a Partial, and, further, what distinguishes a Partial from a Breakoff (see definition below). Eligible cases are divided into Refusals, Breakoffs, Non-contacts, and "Others." Refusals are defined as cases in which some contact has been made with the household unit and a responsible household member has declined to participate in the interview. Breakoffs are defined as cases where an interview was initiated but the respondent refused to complete the interview. The definition of Non-contact varies depending on whether interviews are conducted in-person or over the telephone. For in-person interviews, a Non-contact results when an interviewer is unable to gain access to a building, when no one is reached at a housing unit, or when the respondent is away or unavailable. For telephone interviews, a Non-contact results when the selected respondent is never available or when only an answering machine can be reached, but the message confirms that the telephone number is that of a household unit. "Others" include cases in which the respondent is located and does not refuse the interview, but is unavailable because of death, physical or mental limitations, language problems, or other "miscellaneous" reasons. For in-person household surveys, non-eligible cases consist of household units included in the sample by error, non-residential units, vacant households, household units with no eligible respondent, and situations where quotas have been filled. In addition, for telephone household surveys, non-eligible cases include: fax or data lines, non-working numbers, or situations where the phone-line quality was too poor to conduct an interview. Cases of unknown eligibility include situations in which it's not clear whether an eligible household exists and also those in which a household unit exists, but it's not clear whether an eligible respondent is present. For in-person interviews, cases of unknown eligibility include cases that were not attempted or worked, those that could not be reached or are in an unsafe area, and those for which an address could not be located. Also included in this category for telephone surveys are: numbers that are always busy, no answers, answering-machine messages that do not indicate whether the number is that of a household unit, and technical phone problems. AAPOR's standard definitions distinguish between temporary and final disposition codes. Temporary disposition codes are used to record the outcomes of contact attempts when the contact has not resulted in a final disposition. Examples of temporary disposition codes are: maximum call limit met, callback, no callback by date of collection cut-off, and broken appointment. Temporary disposition codes must be replaced with final disposition codes before these cases can be included in the calculation of outcome rates. This is done by reviewing the pattern of disposition codes recorded for a case and determining which final disposition code best describes the case. The AAPOR committee did not propose a standard set of temporary disposition codes. Instead, they left this to individual survey centers to define. Similarly, the committee made no proposal as to how final disposition codes are to be assigned to unresolved cases. Calculating Outcome Rates The AAPOR committee divided the standard definition for outcome rate calculations into four categories: Response rates Cooperation rates Refusal rates Contact rates Response rate is defined as the number of complete interviews divided by the number of eligible reporting units in the sample. The committee proposed six response rate calculations, differing in how they count partial interviews and cases of unknown eligibility. Cooperation rate is defined as the number of all cases interviewed divided by the number of all eligible respondents ever contacted. The committee propsed four cooperation rates, differing in how partially completed interviews and "other" eligible, non-interviews are counted. Refusal rate is defined as the number of all cases in which a household unit or respondent refuses to do an interview, or breaks-off an interview, divided by the number of all potentially eligible cases. The committee proposed three refusal rates. Non-contact rates are calculated in a similar manner to refusal rates, and together with refusal rates are needed for computing non-response rates. Finally, contact rate is the proportion of all cases in which some responsible household member was reached by the survey. The committee proposed three contact rates. As with response rates, these differ in how they count partial interviews and cases of unknown eligibility. In using outcome rates, the AAPOR committee does not suggest that researchers report all measures. Rather, they propose that researchers use the standard calculations they deem most appropriate for a study and then be specific about which rates are being reported. Ci3 CATI for Windows Supports AAPOR Standards Ci3 CATI for Windows supports the proposed AAPOR standard definitions. Ci3 CATI for Windows: Allows for the specification of the AAPOR disposition codes (See figure below) Automatically assigns final disposition codes to "obvious" cases and lets you assign final codes to the rest Reports the distribution of final disposition codes and automatically computes all outcome rates (See figure below) For a copy of the report, Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for RDD Telephone Surveys and In-Person Household Surveys, contact: AAPOR, P.O. Box 1248, Ann Arbor, MI 48106-1248, Tel: 734-764-1555, Fax: 734-764-3341, E-mail: AAPOR@umich.edu, Web: http://www.aapor.org Back to Sawtooth News Index