{"id":280,"date":"2011-08-17T15:20:57","date_gmt":"2011-08-17T15:20:57","guid":{"rendered":"http:\/\/commons.trincoll.edu\/aris\/"},"modified":"2011-08-17T20:26:18","modified_gmt":"2011-08-17T20:26:18","slug":"aris-2001-methodology","status":"publish","type":"page","link":"http:\/\/commons.trincoll.edu\/aris\/surveys\/aris-2001\/aris-2001-methodology\/","title":{"rendered":"ARIS 2001 Methodology"},"content":{"rendered":"

SAMPLE DESIGN <\/strong><\/p>\n

As mentioned previously, these studies were conducted as part of established, ongoing national telephone omnibus programs.\u00a0 The inherent nature of commercial omnibus surveys provides both advantages and disadvantages from the perspective of preferred survey research practices.\u00a0\u00a0 Omnibus surveys provide a means of reaching and interviewing extremely large household samples in relatively short periods of time while taking advantage of the shared nature of the high costs of survey research.\u00a0 The economic advantage is offset somewhat by the exigencies of the periodic nature of these surveys; the relatively short field periods and the need for minimal geographic sample stratification and controls necessary to insure minimum sample sizes among population subgroups within those strata, tend to depress response rates somewhat.<\/p>\n

A brief description of each of these omnibus services will aid in understanding these issues:<\/p>\n

\u00b7\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 EXCEL is the research industry\u2019s largest telephone omnibus service and has been in continuous operation for over fifteen (15) years.\u00a0 EXCEL surveys are fielded at least twice each week, with each survey having a minimum of 1,000 interviews.\u00a0 Approximately one-half of these are Male and one-half Female.\u00a0 The sample employs basic geographic stratification at the Census Division level, with target sample sizes allocated proportionately.\u00a0 Although there is some flexibility in terms of final sample size, it is necessary to adhere fairly closely to the established targets of 50% Male\/Female within geographic stratum.\u00a0 Respondents are randomly designated using the Last Birthday Selection Method.\u00a0 The RDD sample utilized is provided by GENESYS Sampling Systems.\u00a0 The field period for each survey is five (5) days \u2013 one wave of EXCEL runs Tuesday through Sunday each week, the other Friday through Tuesday, so both include weekends and the call rule is an original plus four attempts.<\/p>\n

\u00b7\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ACCESS has a more restricted set of question topics than does the more general and varied nature of EXCEL.\u00a0 ACCESS was designed as primarily an omnibus vehicle focusing on residential telecommunications, entertainment and technology issues.\u00a0 Both are national in representation, although ACCESS targets only about 1,000 completed interviews per week.\u00a0 The other major difference between the two omnibus surveys is in the execution of data collection.\u00a0 ACCESS is an ongoing survey as opposed to periodic, with flexible daily and more rigid weekly sample size targets identical to those of EXCEL.\u00a0 The everyday, ongoing nature of the data collection provides the ability to utilize a single large replicated sample, with additional replicates added as required.\u00a0 Sample stratification and respondent selection procedures are handled identically.\u00a0 Similarly, the RDD sample was supplied by GENESYS Sampling Systems.<\/p>\n

In summary, both of the telephone omnibus programs utilize National RDD samples.\u00a0 They were both designed by the same research group and are operated and overseen by ICR personnel.\u00a0 Moreover, EXCEL was the vehicle used in the 1990 NSRI and GENESYS also provided the sample in that survey effort.\u00a0 In addition, the demographic battery embedded within the two omnibus surveys are virtually identical and both incorporate questions to determine the number of voice lines each residence maintains in order to develop probability of selection adjustments to the individual sample household records.<\/p>\n

The underlying RDD samples used in both omnibus programs are provided by GENESYS Sampling Systems.\u00a0 These epsem <\/em>(equal probability selection mechanism) RDD samples are designed using the latest list-assisted methods and are identical to those used almost exclusively by governmental (e.g., the Census Bureau and CDC), social science and academic researchers.\u00a0\u00a0 The GENESYS RDD sample frame is completely redefined and rebuilt every quarter and incorporates a precisely defined, extremely fine implicit stratification that underlies every individual sample selection, thus minimizing sample variance.\u00a0 The sample frame was consistently defined as two-digit working blocks in residential exchanges (NPA-NXXs) containing two or more directory listed telephone households.<\/p>\n

SURVEY COMPONENTS AND DATA COLLECTION <\/strong><\/p>\n

The survey and data collection incorporated three phases corresponding to the gathering of information for distinct sub-samples and questionnaire segments:<\/p>\n

1)\u00a0\u00a0\u00a0\u00a0\u00a0 The overall effort was fielded from 2 February 2001 through 7 June 2001.\u00a0 During this five (5) month period a total of 34,295 interviews were conducted in the EXCEL omnibus and 15,987 were conducted through ACCESS.\u00a0 All respondents were screened to determine their religious identification, the identification of their spouse if any.<\/p>\n

2)\u00a0\u00a0\u00a0\u00a0\u00a0 Between 19 April and 7 June, the Comparative Belief\/Secularity (CB\/S) battery was administered to a total of 14,155 non-Catholics.<\/p>\n

3)\u00a0\u00a0\u00a0\u00a0\u00a0 From 19 April to 16 May the CB\/S component was administered to 2043 self-identified Catholic respondents.<\/p>\n

4)\u00a0\u00a0 The CB\/S battery is not archived at present.<\/p>\n

The individual sub-samples and corresponding questionnaire segments were designed in such a manner as they can be combined in a straightforward manner.\u00a0 The sample of Catholic respondents is a representative subset of all those asked the CB\/S questions and the sub-sample asked the CB\/S questions is representative of the entire sample.\u00a0 The following section describes the process of combining these samples and the manner in which each subset can be used analytically.<\/p>\n

WEIGHTING & ESTIMATION <\/strong><\/p>\n

As in most surveys designed to fulfill multiple objectives, the research team found it necessary to make a series of trade-offs.\u00a0 In this case, there were two critical components of the research design.\u00a0 First, was the overall sample of Religious Identification identified as part of that overall screening process.\u00a0 The second, was the sample of respondents to whom the CB\/S questions were administered, which was comprised of a sub-sample of Catholics, with all other respondents being sampled at 100%.\u00a0 This was actually a very straightforward design with the corresponding weighting and estimation being carried out in a few simple steps.<\/p>\n

The initial phase of the estimation process dealt with the entire sample of 50,282 respondents.\u00a0 One of the primary objectives of this survey was to provide estimates of the population by religious identification.\u00a0 Consequently it was deemed desirable to reduce the role of geographic variation in these estimates, as many adherents to specific religions are highly concentrated geographically.\u00a0 To accomplish thisthe data set was post-stratified into the following geographic components:<\/p>\n

\u00b7\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 The largest seventy-seven (77) MSAs\/PMSAs by central city and non-central city \u2013 this resulted in a total of 153 county defined strata (Note: Nassau-Suffolk PMSA contains no central city).<\/p>\n

\u00b7\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Forty-eight (48) strata, each comprising the residual geography of individual States not defined as part of the MSA\/PMSA strata.<\/p>\n

For each of the 201 geographic strata, estimates of demographic distributions were then derived from CLARITAS for the following categories: (1) Age within Sex, (2) Race\/Ethnicity, (3) and HH income.\u00a0 In addition, estimates of Total HHs and the population 18+ were also secured for each of the geographic strata.<\/p>\n

An initial HH weight for each respondent was computed based on the number of voice lines serving their household.\u00a0 This weight is actually the inverse of the number of phone lines as it adjusts for the greater probability of selecting that household with two, three or more phone lines have, relative to a HH with just one line.\u00a0 (One can easily envision that a sample of random telephone numbers will result in twice the number of households with two lines as one would reasonably expect, as this class has twice the probability of selection.)<\/p>\n

A second weight, corresponding to the selection of the adult member is then computed: a household with one adult has weight of 1.0; two adults, 2.0 and so on.<\/p>\n

With these initial weights computed, the interviews were segregated into the 201 post-strata and a sample balancing (i.e., raking) routine was conducted within each stratum.\u00a0 This is an iterative process that utilizes the marginal distributions of each of the target demographic variables and the corresponding weighted sample variable categories to compute a series of adjustment factors, which successively bring the sample and population demographic distributions into close alignment.\u00a0 The final step in this process is the calculation of simple expansion factors to bring the weighted sample totals within each of the 201 strata to the Total HH and Population 18+ estimates derived previously.\u00a0 Following this process each respondent record contains two weights: one for Household estimates, the other for estimates of the Adult Population.<\/p>\n

The next phase in the weighting process involved adjusting for the sub-sampling of all respondents for the CB\/S comparative study.\u00a0\u00a0 One simple alternative here would have been to simply treat these sub-samples as an independent survey, and replicate the weighting process used for the full survey.\u00a0 Although straightforward given that the process and procedures were already in place, the result would have produced estimates of religious groups and demographic distributions at variance with the total sample.\u00a0 Although the Comparative Study sample was a random subset of the larger, it would have also been subject to sampling variance.\u00a0 It was decided that this was a complication that should be avoided.<\/p>\n

It was decided that a better approach would be to use the larger sample to produce estimates to which the sub-sample could then be adjusted; this would also enable one to treat the Catholic sub-sample directly.\u00a0 The full sample and the CB\/S sub-sample were post-stratified into seventeen (17) groups based on religious identification as determined in the questionnaire.\u00a0 These strata included the largest religious groups individually (Baptist, Catholic, Lutheran, etc) as well as categories corresponding to None, Refused, etc.\u00a0 Based on the total sample, weighted estimates of Household and Population totals for each religion stratum were created as well as distributions of age, gender, income, race\/ethnicity, census region and metropolitan status.<\/p>\n

These estimates as well as the CB\/S sub-samples were used as input to a similar sample balancing routine as used for the full sample.\u00a0 However, in this case, the input weight for each record corresponded to the final weight developed during the full sample weighting process.\u00a0 The process was repeated for each of the seventeen strata created based on religious identification.\u00a0 As noted above, the post stratification process treated the Catholic sub-sample directly and independently, correcting for the intended under-sampling.<\/p>\n

The final data record for each respondent includes Population and Household weights in the full sample file.\u00a0 For those included in the smaller CB\/S comparative study, the data file also includes a set of Household and Population weights developed to produce estimates of Totals and distributions. For reference purposes the approximate relative sampling rates for the CB\/S study are as follows: non-catholic HHs, 40%; and Catholic HHs, 20%.<\/p>\n

USE OF THE WEIGHTING FACTORS<\/p>\n

The correct application of household and population weights can oftentimes be confusing.\u00a0 The choice of one or the other may be determined by the question or variable under consideration, or by the analytic intent.\u00a0 Just a few guidelines and examples may be instructive.<\/p>\n

The Population weights produce estimates of people – specifically, people over 18 years of age.\u00a0 In the CB\/S comparative study a question is asked to determine the length time respondents have been married.\u00a0 Using the Population Weight will produce an estimate of the number of people married for any given length of time.\u00a0 However, this is not the same as the number of couples, which would be produced\u00a0 by using the HH weight.<\/p>\n

Similarly, the number of adults with a specific religious identification can be computed by applying the Population Weight.\u00a0 However, there are theoretical problems with using the Household Weight in combination with religious identification because that is a respondent variable.\u00a0 Using the Household Weight in this case would be the equivalent of classifying a household based solely on the gender of the respondent and ignoring the fact that HHs can be mixed religions as most contain both males and females.<\/p>\n

Demographics present similar difficulties.\u00a0 Income is a household level variable and intuitively one would use the HH Weight to produce a distribution of HH incomes.\u00a0 But one could use the Population Weight to show the distribution of adults with certain HH incomes.\u00a0 These will not be the same because HH income is not perfectly correlated with HH size.<\/p>\n

Classifying the sample into subgroups by using a HH level variable (e.g., number of children) does not mean that one then needs to use the HH weight to examine religious identification.\u00a0 By using the Population Weight one could then produce estimates of the religious identification of adults in HHs with None, One, Two, etc., numbers of children.<\/p>\n

In summary, it is critical to explore the relationship between the context of the variable, or variable being used and the resultant base produced by a given weight.<\/p>\n

SURVEY ERRORS <\/strong><\/p>\n

Surveys are subject to a wide variety of errors.\u00a0 Some of these are related to the sampling process itself and the inherent variation one expects from the process of selecting samples of households at random \u2013 two samples are never identical, but one can predict the distribution of differences one can reasonably expect.<\/p>\n

Other errors are of a non-sampling nature: limitations in the sampling frame, non-response biases, etc.\u00a0 These are generally more difficult to quantify since the difference due to non-response, for example, is only directly quantifiable if one has interviewed the non-respondents.<\/p>\n

During the weighting and estimation phase one attempt to incorporate and compensate for biases in the sample selection and data collection phases.\u00a0 In some cases, this is fairly straightforward as in the case of HHs with multiple telephone lines \u2013 those with fewer lines are under represented relative to those with more lines.\u00a0 Of course, this does not directly address the issue as to whether the proportions of single line, two-line, etc., HHs in the resultant weighted sample are the exact proportion as exist in the general population.\u00a0 In other words, there may be secondary bias contributions because in the data collection process itself it may be more difficult to reach HHs with a single voice line because these HHs tend to have fewer adults.<\/p>\n

One would hope that the weighting and estimation process compensates for both sampling and non-sampling errors and that is the objective, under the assumption that there are no systematic biases introduced by either set of errors.\u00a0 The difference one finds in a sample distribution from what one might expect \u2013 say the number of interviews completed in a given State can be a result of either sampling error or non-sampling errors.\u00a0 One can easily correct for the variation, but to the extent that the shortfall is due to a failure to complete interviews among a distinct subgroup within the State, there remains a risk of potential bias in the overall results.<\/p>\n

The combined final sample disposition for all weeks of the survey effort is shown below.\u00a0\u00a0 It should be understood that the total number of sample records utilized is substantially understated due to the pres-screening of the RDD sample prior to the actual field period.\u00a0 Although the actual number of Non-Residential sample records eliminated is unavailable, based on the expected eliminations, it can be estimated that the original sample total was approximately 910,000.<\/p>\n

This disposition has been constructed to take into account the limitations of limited field period omnibus surveys by placing callbacks beyond the interviewing period into the Not Eligible Category.\u00a0 Using the most conservative approach, with a base of all residences, the estimated Response Rate is 16.1%.\u00a0 Eliminating HHs deemed Not Eligible, raises the response rate somewhat to 18.2%.<\/p>\n

Table 1 <\/strong><\/p>\n

\"\"<\/a><\/p>\n

We have taken care here to insure that the final weighted sample is accurately proportioned across critical geographic and demographic variables, but the risk of response and other biases can not be fully reflected in a simple estimate of sampling variability.<\/p>\n

SAMPLING VARIABILITY <\/strong><\/p>\n

All sample surveys are subject to sampling errors.\u00a0 Samples always differ from what one would expect if one had measured the entire population.\u00a0 The expected size of that error is a function of both the sample design and the ultimate sample size.\u00a0 The size of this error is also influenced by the specific weighting process and variation in resultant weights designed to compensate for non-sampling errors such as non-response.<\/p>\n

In addition, we have two samples here: one of approximately 50,000 and one about 17,000.\u00a0 The accompanying standard error tables provide estimates of the sample variability for each data set along with instructions on constructing confidence intervals based on the estimate and the size of the subgroup being examined.\u00a0 These estimates were computed from the weighted sample itself using a balanced replication routine (BRR) across a number of survey variables.\u00a0 Any such table of standard errors is a compromise and an estimate since each survey variable theoretically has its own specific error of measurement and variability.<\/p>\n

By examining a range of variables however, one is able to produce an average error, which is then utilized to produce the accompanying Table 2.<\/p>\n

Table 2 Estimates of\u00a0 Survey Standard Errors<\/strong><\/p>\n

\"\"<\/a><\/p>\n

For further details and information you should consult Barry A. Kosmin & Ariela Keysar, Religion in A Free Market, Ithaca, NY: Paramount Market Publishing, 2006<\/p>\n

ARIS 2001\u00a9<\/p>\n","protected":false},"excerpt":{"rendered":"

SAMPLE DESIGN As mentioned previously, these studies were conducted as part of established, ongoing national telephone omnibus programs.\u00a0 The inherent nature of commercial omnibus surveys provides both advantages and disadvantages from the perspective of preferred survey research practices.\u00a0\u00a0 Omnibus surveys provide a means of reaching and interviewing extremely large household samples in relatively short periods … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":21,"featured_media":0,"parent":139,"menu_order":0,"comment_status":"closed","ping_status":"open","template":"page-right.php","meta":{"ngg_post_thumbnail":0},"_links":{"self":[{"href":"http:\/\/commons.trincoll.edu\/aris\/wp-json\/wp\/v2\/pages\/280"}],"collection":[{"href":"http:\/\/commons.trincoll.edu\/aris\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"http:\/\/commons.trincoll.edu\/aris\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"http:\/\/commons.trincoll.edu\/aris\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"http:\/\/commons.trincoll.edu\/aris\/wp-json\/wp\/v2\/comments?post=280"}],"version-history":[{"count":4,"href":"http:\/\/commons.trincoll.edu\/aris\/wp-json\/wp\/v2\/pages\/280\/revisions"}],"predecessor-version":[{"id":381,"href":"http:\/\/commons.trincoll.edu\/aris\/wp-json\/wp\/v2\/pages\/280\/revisions\/381"}],"up":[{"embeddable":true,"href":"http:\/\/commons.trincoll.edu\/aris\/wp-json\/wp\/v2\/pages\/139"}],"wp:attachment":[{"href":"http:\/\/commons.trincoll.edu\/aris\/wp-json\/wp\/v2\/media?parent=280"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}