7

I'm working with a dataset wherein participants rate five different attributes of six device variants; the attribute ratings different variants are very tightly correlated, suggesting that this dataset has a problem with halo error--participants form an overall impression of the quality of the device, and then instead of reassessing the device for each attribute, they answer each attribute with their overall evaluation of the device.

What strategies are used to combat this effect, either before execution in the design of the study or after execution in the analysis? Citations for evidence for any strategies particularly desired.

Jeromy Anglim
  • 30,741
  • 11
  • 93
  • 221
Krysta
  • 2,943
  • 20
  • 43
  • What's the process - do they look at one device variant and answer all five attributes, then move onto the next device .. or do they look at each device and rate for one attribute, and then repeat for the next attribute? – Erics Jul 24 '13 at 14:36
  • Process modifications are possible (and one of the things I'm interested in). They have previously looked at one device and answered all five, then moved on; I think grouping the ratings by attribute might help, but since they need to see each device and no other device while rating it, it would take a lot of square-dancing to manage and probably quite a bit more time. – Krysta Jul 25 '13 at 13:31

1 Answers1

5

Murphy & Cleveland (1995) mention, that a good way to reduce rater errors in general is to inform raters of the existence and nature of these errors and then to simply urge to avoid them. While this reduces rater errors, it also decreases the accuracy of ratings, though. These findings come from the literature on performance assessment, where halo is usually thought of as the opposite of accuracy. The unexpected association has been termed the halo-accuracy-paradox. 

Some authors have proposed that the paradoxical effect is due to different operational definitions of halo (Fisicaro, 1988). Apparently there is some evidence that the paradoxcal effect vanishes when this problem is taken care of.

An interesting explanation comes from Latham (Woehr & Huffut, 1994). For him it's all in the way that raters are informed about halo. If raters are told that halo is a global tendency across different ratings and that it is a bad thing, then raters are going to avoid exactly that. But that does not make the ratings more accurate. So when training raters one has to be careful not to create this kind if effect.

In contrast, a meta-analysis by Woehr & Huffcut (1994) that investigates the effectiveness of different kinds of rater trainings does not find the paradoxical effect. Instead, rater trainings moderately decreases halo and increases accuracy. The authors also found support for Latham's hypothesis. The mean effect size when rater training was in accordance with his view was bigger compared to the other cases.

Still, Murphy and Cleveland (1995) are quite radical in their proposal. To them, there are serious problems with all operational definitions of halo (and rater errors in general). Therefore measurements of rater errors should be abandoned altogether. Hence, in their view it doesn't make sense to speak of a paradox.

References:
Fisicaro,  S. A. (1988). A reexamination of the relation between halo error and accuracy. Journal of Applied Psychology. 73(2), 239.
Murphy, K. R. & Cleveland, J.N. (1995). Understanding performance appraisal: Social, organizational, and goal-based perspectives. Thousand Oaks: Sage.
Woehr, D. J., & Huffcutt, A. I. (1994). Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology, 67(3), 189-205.

Jens Kouros
  • 2,526
  • 18
  • 26