Joint Program in Survey Methodology Theses and Dissertations

Permanent URI for this collectionhttp://hdl.handle.net/1903/2782

Browse

Now showing 1 - 20 of 38

PANEL SURVEY ESTIMATION IN THE PRESENCE OF LATE REPORTING AND NONRESPONSE
(2004-08-06) Copeland, Kennon R; Lahiri, Partha; Survey Methodology; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
Estimates from economic panel surveys are generally required to be published soon after the survey reference period, resulting in missing data due to late reporting as well as nonresponse. Estimators currently in use make some attempt to correct for the impact of missing data. However, these approaches tend to simplify the assumed nature of the missing data and often ignore a portion of the reported data for the reference period. Discrepancies between preliminary and revised estimates highlight the inability of the estimation methodology to correct for all error due to late reporting. The current model for one economic panel survey, the Current Employment Statistics survey, is examined to identify factors related to potential model misspecification error, leading to identification of an extended model. An approach is developed to utilize all reported data from the current and prior reference periods, through missing data imputation. Two alternatives to the current models that assume growth rates are related to recent reported data and reporting patterns are developed, one a simple proportional model, the other a hierarchical fixed effects model. Estimation under the models is carried out and performance compared to that of the current estimator through use of historical data from the survey. Results, although not statistically significant, suggest the potential associated with use of reported data from recent time periods in the working model, especially for smaller establishments. A logistic model for predicting likelihood of late reporting for sample units that did not report for preliminary estimates is also developed. The model uses a combination of operational, respondent, and environmental factors identified from a reporting pattern profile. Predicted conditional late reporting rates obtained under the model are compared to actual rates through use of historical information for the survey. Results indicate the appropriateness of the parameters chosen and general ability of the model to predict final reporting status. Such a model has the potential to provide information to survey managers for addressing late reporting and nonresponse.
STATISTICAL ESTIMATION METHODS IN VOLUNTEER PANEL WEB SURVEYS
(2004-11-17) Lee, Sunghee; Valliant, Richard; Survey Methodology; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
Data collected through Web surveys, in general, do not adopt traditional probability-based sample designs. Therefore, the inferential techniques used for probability samples may not be guaranteed to be correct for Web surveys without adjustment, and estimates from these surveys are likely to be biased. However, research on the statistical aspect of Web surveys is lacking relative to other aspects of Web surveys. Propensity score adjustment (PSA) has been suggested as an alternative for statistically surmounting inherent problems, namely nonrandomized sample selection, in volunteer Web surveys. However, there has been a minimal amount of evidence for its applicability and performance, and the implications are not conclusive. Moreover, PSA does not take into account problems occurring from uncertain coverage of sampling frames in volunteer panel Web surveys. This study attempted to develop alternative statistical estimation methods for volunteer Web surveys and evaluate their effectiveness in adjusting biases arising from nonrandomized selection and unequal coverage in volunteer Web surveys. Specifically, the proposed adjustment used a two-step approach. First, PSA was utilized as a method to correct for nonrandomized sample selection, and secondly calibration adjustment was used for uncertain coverage of the sampling frames. The investigation found that the proposed estimation methods showed a potential for reducing the selection and coverage bias in estimates from volunteer panel Web surveys. The combined two-step adjustment not only reduced bias but also mean square errors to a greater degree than each individual adjustment. While the findings from this study may shed some light on Web survey data utilization, there are additional areas to be considered and explored. First, the proposed adjustment decreased bias but did not completely remove it. The adjusted estimates showed a larger variability than the unadjusted ones. The adjusted estimator was no longer in the linear form, but an appropriate variance estimator has not been developed yet. Finally, naively applying the variance estimator for linear statistics highly overestimated the variance, resulting in understating the efficiency of the survey estimates.
GRICEAN EFFECTS IN SELF-ADMINSTERED SURVEYS
(2005-10-31) Yan, Ting; Tourangeau, Roger; Survey Methodology; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
Despite the best efforts of questionnaire designers, survey respondents don't always interpret questions as the question writers intended. Researchers have used Grice's conversational maxims to explain some of these discrepancies. This dissertation extends this work by reviewing studies on the use of Grice's maxims by survey respondents and describing six new experiments that looked for direct evidence that respondents apply Grice's maxims. The strongest evidence for respondents' use of the maxims came from an experiment that varied the numerical labels on a rating scale; the mean shift in responses to the right side of the rating scale induced by negative numerical labels was robust across items and fonts. Process measures indicated that respondents applied the maxim of relation in interpreting the questions. Other evidence supported use of the maxim of quantity -- as predicted, correlations between two highly similar items were lower when they were asked together. Reversing the wording of one of the items didn't prevent respondents from applying the maxim of quantity. Evidence was weaker for the application of Grice's maxim of manner; respondents still seemed to use definitions (as was apparent from the reduced variation in their answers), even though the definitions were designed to be uninformative. That direct questions without filters induced significantly more responses on the upper end of the scale -- presumably because of the presuppositions direct questions carried -- supported respondents' application of the maxim of quality. There was little support for respondents' use of the maxim of relation from an experiment on the physical layout of survey questions; the three different layouts didn't influence how respondents perceived the relation among items. These results provided some evidence that both survey "satisficers" and survey "optimizers" may draw automatic inferences based on Gricean maxims, but that only "optimizers" will carry out the more controlled processes requiring extra effort. Practical implications for survey practice include the need for continued attention to secondary features of survey questions in addition to traditional questionnaire development issues. Additional experiments that incorporate other techniques such as eye tracking or cognitive interviews may help to uncover other subtle mechanisms affecting survey responses.
ANALYSIS OF COMPLEX SURVEY DATA USING ROBUST MODEL-BASED AND MODEL-ASSISTED METHODS
(2006-09-19) Li, Yan; Lahiri, Partha; Survey Methodology; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
Over the past few decades, major advances have taken place in both model-based and model-assisted approaches to inferences in finite population sampling. In the standard model-based approach, the finite population is assumed to be a realization from a superpopulation characterized by a probability distribution, and that the distribution of the sample is identical to that of the finite population. The model-based method could lead to a misleading inference if either assumption is violated. The model-assisted estimators typically are consistent or at least approximately unbiased with respect to the sampling design, and yet more efficient than the customary randomization-based estimators in the sense of achieving smaller variance with respect to the design if the assumed model is appropriate. Since both approaches rely on the assumed model, there is a need to achieve robustness with respect to the model selection. This is precisely the main theme of this dissertation. This study uses the well-known Box-Cox transformation on the dependent variable to generate certain robust model-based and model-assisted estimators of finite population totals. The robustness is achieved since the appropriate transformation on the dependent variable is determined by the data. Both Monte Carlo simulation study and real data analyses are conducted to illustrate the robustness properties of the proposed estimation method using two different ways: (i) design-based, and (ii) model-based, wherever appropriate. A few potential areas of future research within the context of transformations in linear regression models, as well as linear mixed models, for analysis of complex survey data are identified.
The Relationship Between Response Propensity and Data Quality in the Current Population Survey and the American Time Use Survey
(2007-04-26) Fricker, Scott; Tourangeau, Roger; Survey Methodology; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
An important theoretical question in survey research over the past fifty years has been: How does bringing in late or reluctant respondents affect total survey error? Does the effort and expense of obtaining interviews from difficult to contact or reluctant respondents significantly decrease the nonresponse error of survey estimates? Or do these late respondents introduce enough measurement error to offset any reductions in nonresponse bias? This dissertation attempted to address these questions by examining nonresponse and data quality in two national household surveys--the Current Population Survey (CPS) and the American Time Use Survey (ATUS). Response propensity models were first developed for each survey, and busyness and social capital explanations of nonresponse were evaluated in light of the results. Using respondents' predicted probability of response, simulations were carried out to examine whether nonresponse bias was linked to response rates. Next, data quality in each survey was assessed by a variety of indirect indicators of response error--e.g., item missing data rates, round value reports, interview-reinterview response inconsistencies, etc.--and the causal roles of various household, respondent, and survey design attributes on the level of reporting error were explored. The principal analyses investigated the relationship between response propensity and the data quality indicators in each survey, and examined the effects of potential common causal factors when there was evidence of covariation. The implications of the findings from this study for survey practitioners and for nonresponse and measurement error studies are discussed.
Regression Diagnostics for Complex Survey Data: Identification of Influential Observations
(2007-09-13) Li, Jianzhu; Valliant, Richard; Survey Methodology; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
Discussion of diagnostics for linear regression models have become indispensable chapters or sections in most of the statistical textbooks. However, survey literature has not given much attention to this problem. Examples from real surveys show that sometimes the inclusion and exclusion of a small number of the sampled units can greatly change the regression parameter estimates, which indicates that techniques of identifying the influential units are necessary. The goal of this research is to extend and adapt the conventional ordinary least squares influence diagnostics to complex survey data, and determine how they should be justified. We assume that an analyst is looking for a linear regression model that fits reasonably well for the bulk of the finite population and chooses to use the survey weighted regression estimator. Diagnostic statistics such as DFBETAS, DFFITS, and modified Cook's Distance are constructed to evaluate the effect on the regression coefficients of deleting a single observation. As components of the diagnostic statistics, the estimated variances of the coefficients are obtained from design-consistent estimators which account for complex design features, e.g. clustering and stratification. For survey data, sample weights, which are computed with the primary goal of estimating finite population statistics, are sources of influence besides the response variable and the predictor variables, and therefore need to be incorporated into influence measurement. The forward search method is also adapted to identify influential observations as a group when there is possible masked effect among the outlying observations. Two case studies and simulations are done in this dissertation to test the performance of the adapted diagnostic statistics. We reach the conclusion that removing the identified influential observations from the model fitting can obtain less biased estimated coefficients. The standard errors of the coefficients may be underestimated since the variation in the number of observations used in the regressions was not accounted for.
Sampling Weight Calibration with Estimated Control Totals
(2008-11-12) Dever, Jill A; Valliant, Richard; Survey Methodology; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
Sample weight calibration, also referred to as calibration estimation, is a widely applied technique in the analysis of survey data. This method borrows strength from a set of auxiliary variables and can produce weighted estimates with smaller mean square errors than those estimators that do not use the calibration adjustments. Poststratification is a well-known calibration method that forces weighted counts within cells generated by cross-classifying the categorical (or categorized) auxiliary variables to equal the corresponding population control totals. Several assumptions are critical to the theory developed to date for weight calibration. Two assumptions relevant to this research include: (i) the control totals calculated from the population of interest and known without (sampling) error; and (ii) the sample units selected for the survey are taken from a sampling frame that completely covers the population of interest (e.g., no problems with frame undercoverage). With a few exceptions, research to date generally is conducted as if these assumptions hold, or that any violation does not affect estimation. Our research directly examines the violation of the two assumptions by evaluating the theoretical and empirical properties of the mean square error for a set of calibration estimators, newly labeled as estimated-control (EC) calibration estimators. Specifically, this dissertation addresses the use of control totals estimated from a relatively small survey to calibrate sample weights for an independent survey suffering from undercoverage and sampling errors. The EC calibration estimators under review in the current work include estimated totals and ratios of two totals, both across all and within certain domains. The ultimate goal of this research is to provide survey statisticians with a sample variance estimator that accounts for the violated assumptions, and has good theoretical and empirical properties.
The Bayesian and Approximate Bayesian Methods in Small Area Estimation
(2008-11-20) Pramanik, Santanu; Lahiri, Partha; Survey Methodology; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
For small area estimation, model based methods are preferred to the traditional design based methods because of their ability to borrow strength from related sources. The indirect estimates, obtained using mixed models, are usually more reliable than the direct survey estimates. To draw inferences from mixed models, one can use Bayesian or frequentist approach. We consider the Bayesian approach in this dissertation. The Bayesian approach is straightforward. The prior and likelihood produce the posterior, which is used for all inferential purposes. It overcomes some of the shortcomings of the empirical Bayes approach. For example, the posterior variance automatically captures all sources of uncertainties in estimating small area parameters. But this approach requires the specification of a subjective prior on the model parameters. Moreover, in almost all situation, the posterior moments involve multi-dimensional integration and consequently closed form expressions cannot be obtained. To overcome the computational difficulties one needs to apply computer intensive MCMC methods. We apply linear mixed normal models (area level and unit level) to draw inferences for small areas when the variable of interest is continuous. We propose and evaluate a new prior distribution for the variance component. We use Laplace approximation to obtain accurate approximations to the posterior moments. The approximations present the Bayesian methodology in a transparent way, which facilitates the interpretation of the methodology to the data users. Our simulation study shows that the proposed prior yields good frequentist properties for the Bayes estimators relative to some other popular choices. This frequentist validation brings in an objective flavor to the so-called subjective Bayesian approach. The linear mixed models are, usually, not suitable for handling binary or count data, which are often encountered in surveys. To estimate the small area proportions, we propose a binomial-beta hierarchical model. Our formulation allows a regression specification and hence extends the usual exchangeable assumption at the second level. We carefully choose a prior for the shape parameter of the beta density. This new prior helps to avoid the extreme skewness present in the posterior distribution of the model parameters so that the Laplace approximation performs well.
HIERARCHICAL BAYES ESTIMATION AND EMPIRICAL BEST PREDICTION OF SMALL-AREA PROPORTIONS
(2009) Liu, Benmei; Lahiri, Partha; Survey Methodology; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
Estimating proportions of units with a given characteristic for small areas using small area estimation (SAE) techniques is a common problem in survey research. The direct survey estimates, usually based on area-specific sample data, are very imprecise or even unavailable due to the small or zero sample sizes in the areas. In order to provide precise estimates, a variety of model-dependent techniques, using Bayesian and frequentist approaches, have been developed. Among those, empirical best prediction (EBP) and hierarchical Bayes (HB) methods relying on mixed models have been considered for estimating small area proportions. Mixed models can be broadly classified as area or unit level models in SAE. When an area level model is used to produce estimates of proportions for small areas, it is commonly assumed that the survey weighted proportion for each sampled small area has a normal distribution and that the sampling variance of this proportion is known. However, these assumptions are problematic when the small area sample size is small or when the true proportion is near 0 or 1. In addition, normality is commonly assumed for the random effects in area level and unit level mixed models. However, this assumption maybe violated for some cases. To address those issues, in this dissertation, we first explore some alternatives to the well-known Fay-Herriot area level model. The aim is to consider models that are appropriate for survey-weighted proportions and can capture different sources of uncertainty, including the uncertainty that arises from the estimation of the sampling variances of the design-based estimators. Then we develop an adaptive HB method for SAE using data from a simple stratified design. The main goal is to relax the usual normality assumption for the random effects and instead determine the distribution of the random effects adaptively from the survey data. The Jiang-Lahiri type frequentist's alternative to the hierarchical Bayesian methods is also developed. Finally we propose a generalized linear mixed model that is suitable for binary data collected from a two-stage sampling design.
Understanding the Mechanism of Panel Attrition
(2009) Lemay, Michael; Kreuter, Frauke; Survey Methodology; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
Nonresponse is of particular concern in longitudinal surveys (panels) for several reasons. Cumulative nonresponse over several waves can substantially reduce the proportion of the original sample that remains in the panel. Reduced sample size increases the variance of the estimates and reduces the possibility for subgroup analysis. Also, the higher the attrition, the greater the concern that error (bias) will arise in the survey estimates. The fundamental purpose of most panel surveys is to allow analysts to estimate dynamic behavior. However, current research on attrition in panel surveys focuses on the characteristics of respondents at wave 1 to explain attrition in later waves, essentially ignoring the role of life events as determinants of panel attrition. If the dynamic behaviors that panel surveys are designed to examine are also prompting attrition, estimates of those behaviors and correlates of those behaviors may be biased. Also, current research on panel attrition generally does not differentiate between attrition through non-contacts and attrition through refusals. As these two source of nonresponse have been shown to have different determinants, they can also be expected to have different impacts on data quality. The goal of this research is to examine these issues. Data for this research comes from the Panel Survey of Income Dynamics (PSID) conducted by the University of Michigan. The PSID is an ongoing longitudinal survey that began in 1968 and with a focus on the core topics of income, employment, and health.
Reliability of The CVI Range: A Functional Vision Assessment for Children with Cortical Visual Impairment
(2009) Newcomb, Sandra; Beckman, Paula; Survey Methodology; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
ABSTRACT Reliability of The CVI Range - A Functional Vision Assessment for Children with Cortical Visual Impairment Children identified as visually impaired under the Individuals with Disabilities Education Act (IDEA) need to have a functional vision assessment to determine how the visual impairment affects educational performance. Most current functional vision assessments have been based on the needs of children with ocular visual impairments (children with damage to the eye structures). Children with visual impairment due to brain damage, or cortical visual impairment (CVI), have unique vision characteristics that are often different from children with ocular visual impairments. Given this situation, Roman-Lantzy (2007) developed The CVI Range for conducting a functional vision assessment of children with CVI. The purpose of this study was to examine the reliability of The CVI Range. In this study, 104 children were assessed with The CVI Range. Twenty-seven children were tested by two examiners to determine inter-rater reliability; 20 children were tested on two occasions to determine the test-retest reliability; and 57 children were tested one time by a single examiner. The CVI Range had an internal consistency measure or alpha of .96. The inter-rater reliability coefficient was .98 and the test-retest reliability coefficient was .99. In addition, the CVI Range has two sections that are scored differently and the scores from the two sections were compared to determine if they provided similar scores and therefore similar implications for intervention. Kappa, or the index of agreement, for the two parts of the assessment was .88. Results of this study indicated that The CVI Range is a reliable instrument. Future research needs to focus on training needs related to administration of The CVI Range as well as training of the many professionals that serve children with CVI. Research is also needed to determine appropriate and effective interventions for children with CVI. The CVI Range can be used to document progress and therefore determine the effectiveness of interventions and further knowledge in the field of evidence-based practices that are appropriate for children with CVI.
Neighborhood Characteristics and Participation in Household Surveys
(2010) Casas-Cordero Valencia, Carolina; Kreuter, Frauke; Survey Methodology; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
Declining response rates in household surveys continue to demand not only a better understanding of the mechanisms underlying nonresponse, but also the identification of auxiliary variables that can help assess, reduce, and hopefully correct for this source of error in survey estimates. Using data from L.A. Family and Neighborhood Study (L.A. FANS), this dissertation shows that observable characteristics of the sampled neighborhoods have the potential to advance both survey research topics. Paper 1 of this dissertation advances our understanding of the role that local neighborhood processes play in survey participation. The measures of social and physical environments are shown to be significant predictors of household cooperation in the L.A.FANS, even after controlling for the socio-economic composition of households and neighborhoods. A nice feature of the indicators of the physical environment is that they can be observed without performing the actual interview. Thus they are available for both respondents and nonrespondents. However, survey interviewers charged with this task might make errors that can limit the usability of these observations. Paper 2 uses a multilevel framework to examine 25 neighborhood items rated by survey interviewers. The results show that errors vary by type of item and that interviewer perceptions are largely driven by characteristics of the sampled areas -- not by characteristics of the interviewers themselves. If predictive of survey participation, neighborhood characteristics can be useful for survey fieldwork decisions aimed at increasing response rates. If neighborhood characteristics are also related to survey outcome variables they furthermore can be used to inform strategies aimed at reducing nonresponse bias. Paper 3 compares the effectiveness of several different neighborhood characteristics in nonresponse adjustments for the L.A.FANS, and shows that interviewer observations perform similar to Census variables when used for weighting key estimates of L.A. FANS. Results of this dissertation can be relevant for those who want to increase response rates by tailoring efforts according to neighborhood characteristics. The most important contribution of this dissertation, however, lies in re-discovering intersections between survey methodology and urban sociology.
Collinearity Diagnostics for Complex Survey Data
(2010) Liao, Dan; Valliant, Richard; Survey Methodology; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
Survey data are often used to fit models. The values of covariates used in modeling are not controlled as they might be in an experiment. Thus, collinearity among the covariates is an inevitable problem in the analysis of survey data. Although many books and articles have described the collinearity problem and proposed strategies to understand, assess and handle its presence, the survey literature has not provided appropriate diagnostic tools to evaluate its impact on the regression estimation when the survey complexities are considered. The goal of this research is to extend and adapt the conventional ordinary least squares collinearity diagnostics to complex survey data when a linear model or generalized linear model is used. In this dissertation we have developed methods that generally have either a model-based or design-based interpretation. We assume that an analyst uses survey-weighted regression estimators to estimate both underlying model parameters (assuming a correctly specified model) and census-fit parameters in the finite population. Diagnostics statistics, variance inflation factors (VIFs), condition indexes and variance decomposition proportions are constructed to evaluate the impact of collinearity and determine which variables are involved. Survey weights are components of the diagnostic statistics and the estimated variances of the coefficients are obtained from design-consistent estimators which account for complex design features, e.g. clustering and stratification. Illustrations of these methods are given using data from a survey of mental health organizations and a household survey of health and nutrition. We demonstrate that specialized collinearity diagnostic statistics are needed to account for survey weights and complex finite population features that are reflected in the sample design and considered in the regression analysis.
Errors in Housing Unit Listing and their Effects on Survey Estimates
(2010) Eckman, Stephanie; Kreuter, Frauke; Survey Methodology; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
In the absence of a national population or housing register, field work organizations in many countries use in-field housing unit listings to create a sampling frame for in-person household surveys. Survey designers select small geographic clusters called segments and specially trained listers are sent to the segments to record the address and/or description of every housing unit. These frames are then returned to the central office where statisticians select a sample of units for the survey. The quality of these frames is critically important for the overall survey quality. A well designed and executed sample, efforts to reduce nonresponse and measurement error, and high quality data editing and analysis cannot make up for errors of undercoverage and overcoverage on the survey frame. Previous work on housing unit frame quality has focused largely on estimating net coverage rates and identifying the types of units and segments that are vulnerable to undercoverage. This dissertation advances our understanding of the listing process, using sociological and psychological theories to derive hypotheses about lister behavior and frame coverage. Two multiple-listing datasets supporttests of these hypotheses. Chapter 1 demonstrates that two well-trained and experienced listers produce different housing unit frames in the same segments. Chapter 2 considers listing as a principal-agent interaction, but finds limited support for the ability of this perspective to explain undercoverage in traditional listing. Chapter 3 has more success explaining the mechanisms of error in dependent listing. Listers tend not to correct the errors of inclusion and exclusion on the frame they update, leading to undercoverage and overcoverage. Chapter 4 tests for bias due to the observed undercoverage, but finds little evidence that lister error would lead to substantial changes in survey estimates. Housing unit listing is a complex task that deserves more research in the survey methods literature. This work fills in some of the gaps in our understanding of the listing process, but also raises many questions. The good news for survey researchers is that the listers' errors appear to be somewhat random with respect to the household and person characteristics, at least for the variables and datasets studied in this work.
Clarifying Survey Questions
(2011) Redline, Cleo D.; Tourangeau, Roger; Survey Methodology; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
Although comprehension is critical to the survey response process, much about it remains unknown. Research has shown that concepts can be clarified through the use of definitions, instructions or examples, but respondents do not necessarily attend to these clarifications. This dissertation presents the results of three experiments designed to investigate where and how to present clarifying information most effectively. In the first experiment, eight study questions, modeled after questions in major federal surveys, were administered as part of a Web survey. The results suggest that clarification improves comprehension of the questions. There is some evidence from that initial experiment that respondents anticipate the end of a question and are more likely to ignore clarification that comes after the question than before it. However, there is considerable evidence to suggest that clarifications are most effective when they are incorporated into a series of questions. A second experiment was conducted in both a Web and Interactive Voice Response (IVR) survey. IVR was chosen because it controlled for the effects of interviewers. The results of this experiment suggest that readers appear no more capable of comprehending complex clarification than listeners. In both channels, instructions were least likely to be followed when they were presented after the question, more likely to be followed when they were placed before the question, and most likely to be followed when they were incorporated into a series of questions. Finally, in a third experiment, five variables were varied to examine the use of examples in survey questions. Broad categories elicited higher reports than narrow categories and frequently consumed examples elicited higher reports than infrequently consumed examples. The implication of this final study is that the choice of categories and examples require careful consideration, as this choice will influence respondents' answers, but it does not seem to matter where and how a short list of examples are presented.
TREATMENT OF INFLUENTIAL OBSERVATIONS IN THE CURRENT EMPLOYMENT STATISTICS SURVEY
(2011) Gershunskaya, Julie; Lahiri, Partha; Survey Methodology; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
It is common for many establishment surveys that a sample contains a fraction of observations that may seriously affect survey estimates. Influential observations may appear in the sample due to imperfections of the survey design that cannot fully account for the dynamic and heterogeneous nature of the population of businesses. An observation may become influential due to a relatively large survey weight, extreme value, or combination of the weight and value. We propose a Winsorized estimator with a choice of cutoff points that guarantees that the resulting mean squared error is lower than the variance of the original survey weighted estimator. This estimator is based on very un-restrictive modeling assumptions and can be safely used when the sample is sufficiently large. We consider a different approach when the sample is small. Estimation from small samples generally relies on strict model assumptions. Robustness here is understood as insensitivity of an estimator to model misspecification or to appearance of outliers. The proposed approach is a slight modification of the classical linear mixed model application to small area estimation. The underlying distribution of the random error term is a scale mixture of two normal distributions. This setup can describe outliers in individual observations. It is also suitable for a more general situation where units from two distinct populations are put together for estimation. The mixture group indicator is not observed. The probabilities of observations coming from a group with a smaller or larger variance are estimated from the data. These conditional probabilities can serve as the basis for a formal test on outlyingness at the area level. Simulations are carried out to compare several alternative estimators under different scenarios. Performance of the bootstrap method for prediction confidence intervals is investigated using simulations. We also compare the proposed method with alternative existing methods in a study using data from the Current Employment Statistics Survey conducted by the U.S. Bureau of Labor Statistics.
Weight Adjustment Methods and Their Impact on Sample-based Inference
(2011) Henry, Kimberly Anne; Valliant, Richard V; Survey Methodology; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
Weighting samples is important to reflect not only sample design decisions made at the planning stage, but also practical issues that arise during data collection and cleaning that necessitate weighting adjustments. Adjustments to base weights are used to account for these planned and unplanned eventualities. Often these adjustments lead to variations in the survey weights from the original selection weights (i.e., the weights based solely on the sample units' probabilities of selection). Large variation in survey weights can cause inferential problems for data users. A few extremely large weights in a sample dataset can produce unreasonably large estimates of national- and domain-level estimates and their variances in particular samples, even when the estimators are unbiased over many samples. Design-based and model-based methods have been developed to adjust such extreme weights; both approaches aim to trim weights such that the overall mean square error (MSE) is lowered by decreasing the variance more than increasing the square of the bias. Design-based methods tend to be ad hoc, while Bayesian model-based methods account for population structure but can be computationally demanding. I present three research papers that expand the current weight trimming approaches under the goal of developing a broader framework that connects gaps and improves the existing alternatives. The first paper proposes more in-depth investigations of and extensions to a newly developed method called generalized design-based inference, where we condition on the realized sample and model the survey weight as a function of the response variables. This method has potential for reducing the MSE of a finite population total estimator in certain circumstances. However, there may be instances where the approach is inappropriate, so this paper includes an in-depth examination of the related theory. The second paper incorporates Bayesian prior assumptions into model-assisted penalized estimators to produce a more efficient yet robust calibration-type estimator. I also evaluate existing variance estimators for the proposed estimator. Comparisons to other estimators that are in the literature are also included. In the third paper, I develop summary- and unit-level diagnostic tools that measure the impact of variation of weights and of extreme individual weights on survey-based inference. I propose design effects to summarize the impact of variable weights produced under calibration weighting adjustments under single-stage and cluster sampling. A new diagnostic for identifying influential, individual points is also introduced in the third paper.
Respondent Consent to Use Administrative Data
(2012) Fulton, Jenna Anne; Presser, Stanley; Survey Methodology; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
Surveys increasingly request respondents' consent to link survey responses with administrative records. Such linked data can enhance the utility of both the survey and administrative data, yet in most cases, this linkage is contingent upon respondents' consent. With evidence of declining consent rates, there is a growing need to understand factors associated with consent to record linkage. This dissertation presents the results of three research studies that investigate factors associated with consenting. In the first study, we draw upon surveys conducted in the U.S. with consent requests to describe characteristics of surveys containing such requests, examine trends in consent rates over time, and evaluate the effects of several characteristics of the survey and consent request on consent rates. The results of this study suggest that consent rates are declining over time, and that some characteristics of the survey and consent request are associated with variations in consent rates, including survey mode, administrative record topic, personal identifier requested, and whether the consent request takes an explicit or opt-out approach. In the second study, we administered a telephone survey to examine the effect of administrative record topic on consent rates using experimental methods, and through non-experimental methods, investigated the influence of respondents' privacy, confidentiality, and trust attitudes and consent request salience on consent rates. The results of this study indicate that respondents' confidentiality attitudes are related to their consent decision; the other factors examined appear to have less of an impact on consent rates in this survey. The final study used data from the 2009 National Immunization Survey (NIS) to assess the effects of interviewers and interviewer characteristics on respondents' willingness to consent to vaccination provider contact. The results of this study suggest that interviewers vary in their ability to obtain respondents' consent, and that some interviewer characteristics are related to consent rates, including gender and amount of previous experience on the NIS.
Adjustments for Nonresponse, Sample Quality Indicators, and Nonresponse Error in a Total Survey Error Context
(2012) Ye, Cong; Tourangeau, Roger; Survey Methodology; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
The decline in response rates in surveys of the general population is regarded by many researchers as one of the greatest threats to contemporary surveys. Much research has focused on the consequences of nonresponse. However, because the true values for the non-respondents are rarely known, it is difficult to estimate the magnitude of nonresponse bias or to develop effective methods for predicting and adjusting for nonresponse bias. This research uses two datasets that have records on each person in the frame to evaluate the effectiveness of adjustment methods aiming to correct nonresponse bias, to study indicators of sample quality, and to examine the relative magnitude of nonresponse bias under different modes. The results suggest that both response propensity weighting and GREG weighting, are not effective in reducing nonresponse bias present in the study data. There are some reductions in error, but the reductions are limited. The comparison between response propensity weighting and GREG weighting shows that with the same set of auxiliary variables, the choice between response propensity weighting and GREG weighting makes little difference. The evaluation of the R-indicators and the penalized R-indicators using the study datasets and from a simulation study suggests that the penalized R-indicators perform better than the R-indicators in terms of assessing sample quality. The penalized R-indicator shows a pattern that has a better match to the pattern for the estimated biases than the R-indicator does. Finally, the comparison of nonresponse bias to other types of errors finds that nonresponse bias in these two data sets may be larger than sampling error and coverage bias, but measurement bias can be bigger in turn than nonresponse bias, at least for sensitive questions. And postsurvey adjustments do not result in substantial reduction in the total survey error. We reach the conclusion that 1) efforts put into dealing with nonresponse bias are warranted; 2) the effectiveness of weighting adjustments for nonresponse depends on the availability and quality of the auxiliary variables, and 3) the penalized R-indicator may be more helpful in monitoring the quality of the survey than the R-indicator.
Effects of Acoustic Perception of Gender on Nonsampling Errors in Telephone Surveys
(2012) Kenney McCulloch, Susan; Kreuter, Frauke; Survey Methodology; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
Many telephone surveys require interviewers to observe and record respondents' gender based solely on respondents' voice. Researchers may rely on these observations to: (1) screen for study eligibility; (2) determine skip patterns; (3) foster interviewer tailoring strategies; (4) contribute to nonresponse assessment and adjustments; (5) inform post-stratification weighting; and (6) design experiments. Gender is also an important covariate to understand attitudes and behavior in many disciplines. Yet, despite this fundamental role in research, survey documentation suggests there is significant variation in how gender is measured and collected across organizations. Variations of collecting respondent gender may include: (1) asking the respondent; (2) interviewer observation only; (3) a combination of observation aided by asking when needed; or (4) another method. But what is the efficacy of these approaches? Are there predictors of observational errors? What are the consequences of interviewer misclassification of respondent gender to survey outcomes? Measurement error in interviewer's observations of respondent gender has never been examined by survey methodologists. This dissertation explores the accuracy and utility of interviewer judgments specifically with regard to gender observations. Using the recent paradata work and linguistics literature as a foundation to explore acoustic gender determination, the goal of my dissertation is to identify implications for survey research of using interviewers' observations collected in a telephone interviewing setting. Organized into three journal-style papers, through a survey of survey organizations, the first paper finds that more than two-thirds of firms collect respondent gender by some form of interviewer observation. Placement of the observation, rationale for chosen collection methods, and uses of these paradata are documented. In paper two, utilizing existing recording of survey interviews, the experimental research finds that the accuracy of interviewer observations improves with increased exposure. The noisy environment of a centralized phone room does not appear to threaten the quality of gender observations. Interviewer and respondent level covariates of misclassification are also discussed. Analyzing secondary data, the third paper finds there are some consequences of incorrect interviewer observations of respondents' gender on survey estimates. Findings from this dissertation will contribute to the paradata literature and provide survey practitioners guidance in the use and collection of interviewer observations, specifically gender, to reduce sources of nonsampling error.