8

I am designing a memory experiment with a deceptive critical trial, wherein an item is swapped without the participant's knowledge.

What is the best way to determine if the participant has noticed the swap, without actually informing them of it?

I.e., asking the participant 'did you notice the swap we just made' is rather informative about the possibility of a swap :)

Current thoughts:

  1. a statement at the beginning asking them to mention any inconsistencies noticed
  2. a direct question directly after the swap, and
  3. a direct question at the end of the experiment

I think the first and second are likely to induce noticing (the latter by comparison to earlier memories that otherwise would not have been performed) and the last to potentially under-report.

Jeromy Anglim
  • 30,741
  • 11
  • 93
  • 221
Charlie
  • 390
  • 3
  • 7
  • It might be helpful if you add some information about the nature of the test items (e.g., are they words lists, digits, pictures), how they are presented, and which response measures are recorded. – H.Muster Apr 15 '12 at 11:42
  • essentially: there is an image of a scene with items in it, the scene is then altered somewhat, I want to attain most accurate reports of noticed alteration with minimal provocation of the noticing. – Charlie Apr 15 '12 at 13:15
  • 3
    Did you have a look at change blindness studies (http://en.wikipedia.org/wiki/Change_blindness)? Some of them should have faced the same problem. – H.Muster Apr 15 '12 at 14:28
  • it's related to change blindness yes, and I am going through that literature - but this is a broader problem than that, 'how to ask someone something without them being informed by the question'. Quite possibly a hopeless task, I realise, but I wanted to see if anyone had further ideas. – Charlie Apr 15 '12 at 23:38

2 Answers2

8

Procedures like this are often called "funnel debriefing" procedures, and they basically consist of an extended version of what you already had in mind.

You begin with some very general questions about the nature of the experiment (e.g., "Do you have any initial thoughts or reactions about this experiment?"), then move on to some questions that are slightly more specific, but still not fully revealing (e.g., "Why do you think we showed you X?", "Did you notice anything unusual or inconsistent about task Y in the experiment?"), and finally terminate with direct inquiries of awareness (e.g., "Did you notice that we swapped one of the images during task Y?").

These questions may or may not be accompanied by recall or recognition tasks where awareness is gauged through participants' performance on the memory task.

Unfortunately there is typically little or no standardization (at least that I have seen) of how extensive these procedures should be, or for how to score and otherwise handle the results.

Jake Westfall
  • 763
  • 4
  • 14
  • ok, cheers, though I still can't find much info, as you suggest... is the 'funnel', starting with vague questions, thought to minimise demand characteristics from the final direct question? – Charlie Apr 16 '12 at 04:40
  • Sort of. Part of the idea is that if a participant never spontaneously mentions the thing you're probing for until you ask about it point-blank, it may be more safe to assume that they didn't really notice it or at least didn't think much of it. On the other hand, indicating it on one of the first couple of questions would be a big red flag. – Jake Westfall Apr 16 '12 at 05:08
  • Also, people sometimes use these procedures to probe separately for awareness of the manipulation vs. awareness of the experimental hypothesis, in cases where the distinction is potentially of importance (like in some learning research). – Jake Westfall Apr 16 '12 at 05:15
  • 1
    I think the 'classical' introduction to funneled debriefings is Bargh, J. A., & Chartrand, T. L. (2000). The mind in the middle: A practical guide to priming and automaticity research. Handbook of research methods in social and personality psychology. (S. 253–285). New York, NY, US: Cambridge University Press. WHich can be found here: http://www2.psych.ubc.ca/~schaller/528Readings/BarghChartrand2000.pdf – Henrik Apr 18 '12 at 19:20
3

You asked for the best way, and though it may be impossible to decide upon the best way, there are certainly some good ways.

As you know, experimental design is, in part, an art. If your experiment is novel, you must devise a novel way of testing whether the deception worked. If your experiment is not so novel, you may be able to leverage the technique used in another experiment. Sometimes, converting an overt question into a quantitative question is sufficient to hide the true intent.

Here's how I might run your experiment:

Beforehand: "We're showing each participant x sets of pictures, each set containing between 1 and 5 pictures. After each set, we'll ask you how many we showed you." The control set should contain between 1 and 5 dissimilar pictures. The experimental set should contain just the 2 similar pictures (i.e., the swap pair).

After each control set: "How many pictures did we show you?" The answer here determines whether the subject can be relied upon.

After the experimental set: "How many pictures did we show you?" Subjects who noticed the swap will say 2; the rest will say 1.

Confirmation questions:

  • for subjects who answered 2: "What were the differences between the 2 pictures?"

  • for subjects who answered 1: "Did anything in that picture change during the time it was shown to you?" The interviewer's intonation should be neutral to avoid making this a leading question.

You will, of course, refine the questions for your particular experiment.

John Pick
  • 2,193
  • 1
  • 19
  • 23
  • Yes, the inclusion of a question that's innocuous when the answer would be 'no' but still clear when the answer would be 'yes' is a great idea. I'm not sure whether I can pull it off for mine, but will try. – Charlie Apr 20 '12 at 04:26