4

In the Autism-Spectrum Quotient test, the subject is confronted with fifty statements such as:

I don’t particularly enjoy reading fiction.

I prefer to do things the same way over and over again.

and is asked whether they

  • definitely agree,
  • slightly agree,
  • slightly disagree, or
  • definitely disagree

with them. There is no neutral option.

The test is scored as follows: If the subject agrees with an “autistic” statement, they score a point. The same applies, if they disagree with a “non-autistic” statement. There is no distinction between slightly and definitely agreeing (or disagreeing).

I would expect that this test design (no neutral option, no distinction between slight and strong tendencies) leads to an amplification of small effects, such as interpretation of the question, slight tendencies, priming, chance, and so on. While I acknowledge that this effect may be desired in customer surveys and similar, it seems problematic to me in a test for the purposes of diagnosis and epidemology (such as this test). For example, I would expect that people who are already diagnosed with autism (and know this) or people with clichéically autistic interests like mathematics are more likely to tend for the more autistic option – even if their actual stance on the question in neutral.

Now, I am a layman and I acknowledge that designing such a test is not easy, as one has to account for several effects. Hence I wonder: What are the justifications for such a test design, in particular in light of its applications (diagnosis and epidemology) and my above criticism? The paper introducing the test does not provide any citations or reasons for this, at least not in those sections where I would expect it.

Community
  • 1
Wrzlprmft
  • 235
  • 1
  • 9
  • 1
    This is a totally valid, but super-broad question. There are 4 sub-questions to this one question. First, was is the history of the Autism-Spectrum Quotient test and how has it been validated/contested over the years. Second, how much are the types of tests affected by priming and stereotypes. Third, is this test undermined by a lack of neutral option. Fourth, are surveys without methods of comparison still valid. Please split this question into sub-questions and I will upvote them all. – Seanny123 Dec 21 '16 at 08:51
  • @Seanny123: I understand your point and given that I cannot even remotely estimate how long potential answers would be, you may very well be correct. However, before I follow your suggestion, please consider the following: 1) I am not much interested in the test’s history or validation per se, unless it provides answers to my main question(s). 2) I am not that much concerned about the issues addressed in your last three questions on their own (e.g., I fully acknowledge that the influence of priming and stereotypes is inevitable). Instead I am concerned in their interplay. – Wrzlprmft Dec 21 '16 at 09:13
  • I understand that you're (legitimately) only interested in their interplay, but I guess what I'm claiming is that I personally think we have to first establish their individual validity before considering their interplay. – Seanny123 Dec 21 '16 at 09:15
  • @Seanny123: I narrowed down the question, though not as far as you suggested. I still hope that this is sufficiently narrow now. – Wrzlprmft Dec 21 '16 at 13:30
  • I believe the question has been reduced in such a degree that it became answerable. Simply, why do the developers of the Autism-Spectrum Quotient test give four options, if they scale it down to just two (agree or disagree)? – Robin Kramer-ten Have Feb 08 '17 at 07:56
  • Why do you assume that the high-scoring mathematicians do not have Asperger or autism? Currently, high-tech and IT companies are specifically recruiting high-functioning autists because they are often better at maths and programming than non-autists. Maybe the test results just reflect the fact that many (possibly subclinical) autists choose a career in a field that rewards their traits. –  Feb 28 '17 at 21:54
  • @what: Why do you assume that the high-scoring mathematicians do not have Asperger or autism? – Where do say this? Just that I am concerned whether a test is flawed does not mean that I consider all conclusions drawn from its results wrong. My problem with the test design is (to be blunt) that it may measure whether somebody is a mathematician and not whether somebody is autistic. This would be problematic since the test may e.g. diagnose autism in a non-autistic mathematician. – Wrzlprmft Mar 01 '17 at 09:36
  • @Wrzlprmft "people with clichéically autistic interests like mathematics" Very likely that similarity isn't "clichéically" but very meaningful. –  Mar 01 '17 at 12:23
  • @what: There is an underlying truth to many clichés, but that does not free them from being clichés. Not everybody interested in mathematics is automatically autistic. Still such a person may be primed that they are more “autistic” than the average population and thus be more likely to choose the “autistic” option in the test (because they are forced to make a choice). But such priming is something that you do not want to be reflected by a test (of course you cannot completely avoid it, but this test seems to amplify it). – Wrzlprmft Mar 01 '17 at 16:54

1 Answers1

3

Many construct scales are developed with relatively little attention to the content. Researchers may start with a large set of candidate questions, versions, formats, and phrasings, and then through a validation process, narrow them down to a subset that correlates well with the construct that they are trying to measure. The goal is a balance between keeping the test short while also retaining its predictive value. In Baron-Cohen et al's (2001) introduction, they note:

The AQ as shown in the Appendix is the outcome of piloting multiple versions, over several years.

In many cases, researchers care less about why the questions work, and more just that they work.

Having said that, the decision to go with a forced-choice format vs. say, a Likert-type rating scale with more granularity and a neutral option, can be political:

Ipsative measures may be more useful for evaluating traits within an individual, whereas Likert-type scales are more useful for evaluating traits across individuals.

This can make forced-choice measures desirable for stigmatized labels to discourage comparing individuals to one another:

Encouraging pupils to beat their previous scores can take peer pressure out of situations and eliminates the competitive element associated with norm-based referencing.

Note also that the original study confirms that mathematicians score significantly higher on this scale, as is to be expected:

... scientists score higher than nonscientists; and within the sciences, mathematics, physical scientists, computer scientists, and engineers score higher than the more human or life-centered sciences of medicine (including veterinary science) and biology. This latter finding replicates our earlier studies finding a link between autism spectrum conditions and occupations/skills in maths, physics, and engineering.

This suggests that the authors thought it perfectly valid for mathematicians to score higher on their autism scale (ie, this scale is not designed to differentiate the autistic tendencies of mathematicians from those of individuals diagnosed with autism).

Arnon Weinberg
  • 19,600
  • 8
  • 56
  • 91
  • Unless I completely misunderstood the terminology, the AS test is not an ipsative test. Rather it is a Likert-scale test without a neutral option. I also fail to see what assessing pupils and beating previous scores has to do with the AS test. – Wrzlprmft Feb 20 '17 at 11:25
  • the original study confirms that mathematicians score significantly higher on this scale, as is to be expected – Sure they do, but that does not address my doubts, which (somewhat exaggerated) are: The way the test is designed, small tendencies are amplified. For example, a mathematician would will slightly agree with an “autistic” option, because they are primed that they are somewhat more autistic than average. Therefore a high AS score for mathematicians is expected from the test design and does not tell us anything about reality. – Wrzlprmft Feb 20 '17 at 11:35
  • @Wrzlprmft Yes, it is a forced-choice (ipsative) test, the authors themselves state this in the paper. A Likert-scale would have different scoring for slightly vs strongly (dis)agree. I was merely pointing out that scoring in such tests discourages comparisons between patients (compared to say, IQ tests), which may make them preferable for diagnostic tests that have potentially stigmatizing labels. The authors do not explain their decision one way or the other. – Arnon Weinberg Feb 20 '17 at 17:00
  • Re telling us about reality, this is the purpose of validation (a topic far too large for this post, but see http://cogsci.stackexchange.com/questions/9610/what-is-the-relation-between-measures-constructs-and-concepts for more detail). The validation process confirmed that the amplification is both expected and realized, and does in fact tell us what we want to know from this scale about reality - ie, that mathematicians do have higher than average autistic tendencies. It's possible that you think this scale should measure something different than what the authors think it should measure. – Arnon Weinberg Feb 20 '17 at 17:05
  • Regarding the subject of validation: I think I understand this. My problem is rather that the test has properties that will make it succeed validation irrespective of the desired qualities. As a simple and blatant example, consider a world where 80 % of all mathematicians are autistic and vice versa and consider a test that simply asks the subject whether they are mathematicians. This test is obviously flawed, yet it will have pass a basic validation. – Wrzlprmft Feb 20 '17 at 19:29
  • Yes, absolutely. As I said, that is a discussion beyond the scope of this question (though the post I linked to does discuss this issue - for example, your assertion that autism and mathematics are separate constructs is just as contentious as a scale that measures mathematics when it purports to measure autism!). Anyways, it sounds like you have a good grasp of the issues already. – Arnon Weinberg Feb 20 '17 at 20:01
  • beyond the scope of this question – well, it is more or less my central question. However, I am open to the possibility that the answer to this question is that the test is actually flawed in this respect. – Wrzlprmft Feb 22 '17 at 09:26
  • Your question was "What are the justifications for such a test design", not "Is this test design flawed". Had your question been "Is this test design flawed", then it would likely have been Closed for being Primarily Opinion Based, which is outside the purview of this forum. I suspect you think this scale should measure something different than what the authors think it should measure. – Arnon Weinberg Feb 22 '17 at 16:12
  • You are right, “this test is flawed” was a bit going too far. Rather, it was: “was this possible effect taken into account?” Anyway, I am not complaining. – Wrzlprmft Feb 22 '17 at 17:07