Improving comparability of self-reported health through direct or implied comparisons to anchoring vignettes

Presenter: Joshua Salomon, Harvard University

Abstract

Background and Objectives: Use of anchoring vignettes has been proposed as a strategy for improving comparability of self-reports on health and other constructs. Proposed techniques for analyzing anchoring vignette data include statistical modelsand non-parametric approaches that rely on implied comparisons between an individual’s own level on a health domain and the level described in a vignette. In this study we consider an alternative approach based on direct comparisons to anchoring vignettes, and also assess sequence effects that derive from the location of vignette questions either before or after self-reports.

Methods: We surveyed a nationally representative sample of the US adult population (N=1,278), drawn from an established Internet research panel. The questionnaire included self-ratings and vignette ratings on physical functioning. Respondents were randomized to complete one of four different variants of the survey, which differed by whether the vignette questions preceded or followed the self-ratings, and whether implied or direct comparisons were used. All versions included two different physical functioning vignettes. In the direct comparison version, respondents were asked explicitly whether their functioning was better than, the same as or worse than the level described in each vignette. In the implied comparison version, this ordering wasinferred from the combination of self-ratings and vignette ratings.

Results: We observed significant sequence effects, in that self-reports elicited after the vignette questions reflected more severe limitations in physical activity than self-reports elicited before the vignette questions (p<0.001 in a chi-square test for equality). We also found evidence that the direct comparison approach may have higher validity than the implied comparison approach. Based on direct comparisons,we observed a monotonic decline in vignette-adjusted physical function with advancing age, whereas the implied comparisons showed an inconsistent age gradient.

Conclusions: Direct comparisons to anchoring vignettes do not require the assumption of response consistency that is demanded by both statistical approachesand approaches that rely on implied comparisons. The direct comparison strategy may offer a promising new direction for application of anchoring vignettes to improve comparability of self-reports, with the advantages of higher validity and improved efficiency.