Development & validation of the Chandigarh autism screening instrument

Sir,

The Chandigarh Autism Screening Instrument (CASI) was developed to screen for autism spectrum disorders (ASD)1. The study was reviewed in electronic Journal Club India (eJCIndia)2, and concerns related to sampling, reporting, interpretation and validation were raised in a discussion across the country.

ASD is more prevalent in males than in females, and the clinical features may change as the child grows up. In this context, no attempt was made by the CASI authors to age match and sex match the ASD and control groups. The authors provided almost no demographic description of the case and control groups. They also did not state how many cases had ASD with versus without mental retardation and how the CASI functioned in these ASD subgroups1. It is therefore, unclear to what population the results of the study can be generalized.

Of considerable concern is that the authors used a retrospectively ascertained chart diagnosis of ASD as the gold standard. In validation exercises, it is necessary for the gold standard to be prospectively established using recognized instruments, such as a diagnostic interview schedule. The gold standard must be truly valid. A diagnostic interview schedule should also have been applied to define the sample in the other groups and, in particular, to confirm the absence of psychiatric disorder in the sample of normally developing children. A teacher's report of the absence of problem behaviour and a parent's report of the absence of concern cannot confirm normal development.

Because the age range of the sample was wide (1.5-10 yr), items in the CASI may have varied in age specificity. It is to be seen whether having separate versions of CASI for different age groups, comprising different items and with different cut-offs, would be a better approach, as with already existing instruments in the field.

The authors validated the CASI in a convenience sample of arbitrarily chosen case and comparison subjects rather than in the population. While their approach may provide an understanding of the sensitivity and specificity of the instrument, it cannot offer an understanding of the positive and negative predictive values (PPV, NPV), both of which are sensitive to the prevalence in the population of the disorder being screened for, and both of which are important to know in the context of a screening instrument. In short, the PPV and NPV values that were reported could be wildly wrong.

With regard to other aspects of study execution, pilot testing was not conducted in the target population. The Autism Behaviour Checklist (ABC), chosen for external validation, has been found to have poor sensitivity at the generally recommended cut-offs3. As additional limitations, the CASI was administered without blinding to the different groups of study participants. This could have resulted in rater bias in scoring and response bias in the ASD caregivers who endorsed CASI items because of familiarity arising from knowledge about the diagnosis. Of further concern, the CASI was in some cases self-administered by caregivers and in other cases, clinician administered; both were inconsistent with the intended method of administration stated in the abstract. Finally, the items comprising CASI Bref were not listed, nor was their selection justified; nor was the sensitivity and specificity of CASI Bref stated for scores of 3 and 4. We also consider that, in their analysis, a cut-off that prioritized sensitivity over specificity might have been better than the one that they chose to optimize sensitivity and specificity. This is because the primary purpose of a screening instrument is not to miss a case (sensitivity), especially when the condition is serious and rare. Perhaps, the authors could have reported the sensitivity and specificity values for different cut-offs instead of for the single cut-off that they selected.

It is surprising that the authors did not present CASI scores that compared ASD in cases with and without mental retardation separately, along with the scores in the different comparison groups. The correlations between CASI and the existing ASD screening instruments are also necessary to examine but were not presented; these are measures of convergent validity, and not external validity, as the authors have mistakenly stated4.

It is hoped that these concerns will be considered in future iterations of validation of the CASI. In this context, it is suggested that other measures, such as test-retest reliability and inter-rater reliability, are also necessary, as is the validation of the instrument in a sample different from the one in which it was derived.