Telepsychiatry clinical decision support system used by non-psychiatrists in remote areas: Validity & reliability of diagnostic module

Integration of mental health with existing general primary care services by enabling primary care workers to identify and treat people with mental disorders has been considered to be the most acceptable and cost-effective way to bridge the gap between unmet needs and scarce mental healthcare resources123. Besides training, there is a need to provide continuing support and supervision from mental health professionals for effective outcome of such measures24. In India, National Mental Health Programme and District Mental Health Programme were launched with such objectives5. However, these programmes largely have been ineffective in practice due to several reasons, such as limited training period, lack of continued monitoring, supervision and technical support from psychiatrists, coverage of only severe mental illnesses and limited treatment options46. Further, diagnostic concordance for mental disorders varies among clinicians, and reliability of diagnoses has posed a serious challenge even to psychiatrists, psychologists and mental health professionals7. As an attempt to overcome these difficulties, a knowledge-based logically linked clinical decision support system (CDSS) for diagnosis and treatment of psychiatric disorders was developed as part of the telepsychiatry project at the department of Psychiatry of the Postgraduate Institute of Medical Education and Research (PGIMER), Chandigarh, India8. The CDSS has two separate decision support systems for adults and for children and adolescents. In each case, the diagnostic system is logically linked to the intervention modules. The diagnostic and management modules have been found to be valid, reliable and feasible in studies conducted at nodal centre91011.

In this study we assessed the validity and reliability of the diagnoses generated by the system when used by non-psychiatrists by comparing it to the well-validated structured diagnostic interview, Mini-International Neuropsychiatric Interview (MINI)12 and comparing with diagnoses generated by the online application when interview was conducted by trainers at the nodal centre.

Material & Methods

The development, basic structure and content of the diagnostic module for adults have been described elsewhere911. Briefly, the diagnostic module assesses for 18 disorders in adults, namely, delirium, dementia, mania (current and past), depression (current and past), dysthymia, psychosis, obsessive-compulsive disorder (OCD), generalized anxiety disorder (GAD), panic disorder, phobias, reaction to severe stress and adjustment disorder, somatoform disorder, dissociative disorder, neurasthenia, sexual dysfunctions, alcohol dependence, substance dependence and mental retardation (MR), which are commonly seen in clinics. Diagnoses are based primarily on ICD-10 criteria, both from the Clinical Diagnostic Guidelines13 and the Diagnostic Criteria for Research14. In certain parts (e.g. for delirium or dementia), Diagnostic and Statistical Manual of Mental Disorders (DSM-IV)15 criteria have been used. Diagnostic assessment includes elicitation of symptoms and other information from the patient as well as the caregiver if available as done in routine clinical case work-up. The diagnostic exercise follows a stepwise approach. The study was conducted at telepsychiatry centres at three remote sites, i.e. Himachal Pradesh (HP), Uttarakhand (UK) and Jammu & Kashmir (JK) (face-to-face interviewing) and nodal centre, i.e. PGIMER, Chandigarh (via video-conferencing) from January 2013 to February 2014. The study was approved by the Institute Ethics Committee at PGIMER, Chandigarh. Written informed consent was obtained from all patients and their relatives, explaining the nature and purpose of study in detail.

On registration of a patient, identification details, socio-demographic profile, presenting complaints and precipitating events were recorded. The core of the diagnostic system consisted of an initial screening sub-module followed by a detailed assessment for specific disorders by separate criteria-wise diagnostic sub-modules. Both the screening and diagnostic sub-modules had a similar structure. The screening sub-module had 18 questions and covered all the disorders, and it was mandatory to answer all the questions. Depending on the responses on screening, only those detailed diagnostic sub-modules opened for which the response to the screening question was positive. For example, if the screening was positive for depressive symptoms, only the detailed diagnostic modules for mood disorders would open. Further, amongst the diagnostic sub-modules that are indicated to be opened, each diagnostic sub-module opens in an order based on the inbuilt hierarchy. To avoid multiple diagnoses, a diagnostic hierarchy giving precedence to diagnostic categories appearing earlier in the ICD 10 was incorporated. In each diagnostic sub-module, there was a second-level enquiry about primary or typical symptoms of that disorder, which proceeded to a third-level enquiry employing other criteria, but only if the specified threshold was met at the second level. Alternatively, it skipped the remaining part of that sub-module and moved to the next sub-module derived from screening. At the end of diagnostic work-up, assessments of symptom-severity (on a five-point scale) and socio-occupational functioning (on a visual analogue scale) were carried out. The diagnostic algorithm or structure for the screening and diagnostic sub-modules consisted of three main components, namely the question item with its serial number; the ‘rater's rule’ for the rater to apply and the ‘decision rule’ for computer automation.

The diagnoses generated by the core diagnostic module of the CDSS were compared with diagnoses generated by the MINI14 to assess diagnostic accuracy of the application. However, some of the disorders included in application are not included in the MINI. Hence, to allow proper comparisons, additional interview questions were asked at the end of the MINI assessment for these diagnostic categories. These included dysthymia, somatization disorder, hypochondriasis, body dysmorphic disorder, pain disorder and adjustment disorder (from the MINI PLUS)12 and delirium and dementia (from DSM-IV)15 and four other disorders which were made on the basis of ICD-10 Diagnostic criteria for Research14.

Consecutive, consenting adult outpatients (>18 yr) attending the remote telepsychiatry centres were recruited for the study. The interviewers who participated at remote sites included general physicians and para-professionals with Masters’ degree in psychology or social work with no clinical experience. Interviews were conducted at nodal centre by three psychologists with Master's level training, and they had been trained by psychiatrists in the use of the diagnostic application and the MINI. For diagnostic accuracy, patients were interviewed face-to-face by non-psychiatrists at remote sites and simultaneously via video-conferencing by two interviewers at nodal centre. The non-specialist at remote site actively interviewed the patient using the online CDSS as the ‘active’ interviewer. At nodal centre, one interviewer served as the ‘passive’ rater who completed the online application based on the responses obtained during the interview conducted by the active interviewer. At the end of the interview, the ‘passive’ rater was allowed to ask any questions if required. In addition, a second interviewer at the nodal centre assessed on MINI in the same sitting through video-conferencing after interview by active rater was over. Interviewers were blind to the results of each other's assessments. Inter-rater reliability of the online application was examined by comparing the assessments of ‘active’ and ‘passive’ raters.

Data analysis: Statistical analysis was carried out using the Statistical Package for the Social Sciences version 18 (SPSS Inc., Chicago, USA). The required minimum sample size for kappa values of 0.4-0.6, at a power of 80 per cent and alpha of 0.05, was estimated to be 100. Analyses were carried out both for the broad ICD-10 diagnostic categories, as well as some individual categories belonging to mood and neurotic disorders. Apart from descriptive analyses, paired t tests were conducted to compare the mean number of diagnosis generated by the online application (active) and MINI. Sensitivity, specificity, positive and negative predictive values (PPV and NPV) were computed for the screening and the diagnostic sub-modules of the tool (ratings done by the active interviewer), compared to the MINI diagnoses. In addition, kappa coefficients were computed to assess agreement between the diagnoses generated by the diagnostic sub-modules (ratings done by the active interviewer) and the MINI diagnoses. Kappa values were computed to assess agreement between diagnosis made using the online application by the active and passive interviewers.

Results

Table I depicts the demographic profile of the 100 patients included in the study. Patients with mean age of 35.10±12.08 yr with more men than women constituted the study sample. They were mostly educated, employed, either married or single and mainly came from joint families. Majority (70%) belonged to rural areas, while 30 per cent belonged to semi-urban areas.

Table I Demographic profile of participants (n=100)

Diagnoses generated by the online application at the remote site and the Mini-International Neuropsychiatric Interview (MINI) at the nodal centre: The mean and median number of total diagnoses made by online application at remote sites were 1.60±0.79 and 1 (range 0-4; 1 in first quartile, 1 in second quartile and 2 in third quartile) and that made by MINI at nodal centre were 1.43±0.74 and 1 (range 0-5; 1 in first quartile, 1 in second quartile and 2 in third quartile). No significant difference was found between the mean numbers of diagnoses made by these two methods. Alcohol use disorders were diagnosed in 19 per cent of patients, drug dependence disorders in 25 per cent, psychoses in 23 per cent, mood disorders (bipolar disorder, major depression and dysthymia) in 49 per cent, neurotic, stress-related and somatoform disorders in 12 per cent, and anxiety (GAD, panic disorder and phobias) in 22 per cent, OCD in eight per cent, somatoform disorder in five per cent and MR in two per cent by the non-specialist at peripheral sites using diagnostic tool. Using the MINI, alcohol use disorders were diagnosed in 28 per cent of the patients, drug dependence disorders in 24 per cent, psychoses in 20 per cent, mood disorders (bipolar disorder, major depression, dysthymia) in 42 per cent, neurotic, stress-related and somatoform disorders in 27 per cent, anxiety (GAD, panic disorder and phobias) in 20 per cent, OCD in five per cent, somatoform disorder in five per cent and MR in one per cent. The time taken for assessment by the ‘core’ diagnostic tool was 19.49±3.0 minutes.

Screening sub-module of the online application at the remote site versus MINI diagnoses at the nodal centre: Table II depicts the sensitivity, specificity, PPV and NPV of the screening sub-module of the online application, with the final MINI diagnosis employed as the standard for comparison. The results showed that the sensitivity of screening part of the tool was high for all disorders, ranging from 80 per cent for OCD to 100 per cent for alcohol dependence, anxiety (GAD, panic disorder, phobic disorders) disorders, stress-related disorders, somatoform disorders and MR. The specificity of screening tool ranged from 48 per cent for any mood disorder to 99 per cent for MR. PPVs were low for most disorders, except for alcohol dependence and drug dependence. This was a reflection of the high rates of false positive diagnoses in these categories. On the other hand, NPVs were consistently high because of the high rates of true negative cases in all categories.

Table II Sensitivity, specificity, positive and negative predictive values of the screening sub-module of the diagnostic tool, compared to the diagnoses obtained from Mini-International Neuropsychiatric Interview (MINI)

Diagnostic sub-modules of the online application at the remote site versus MINI diagnoses at the nodal centre: Table III depicts the Cohen's kappa values, sensitivity, specificity, and PPV and NPV of the diagnostic sub-modules of the tool, compared to the MINI diagnoses. Cohen’ kappa values revealed acceptable level of agreement (>0.4) for all disorders, with substantial (>0.6) to near-perfect agreement for drug dependence, psychosis, broad category mood disorders, bipolar disorder, major depressive and recurrent depressive disorder, dysthymia, anxiety and stress-related disorders (GAD, panic and phobic disorders), somatoform and dissociative disorders, sexual dysfunction and MR. Moderate levels of agreement (kappa from 0.4 to 0.6) were obtained for alcohol dependence and OCD. Sensitivity of diagnoses was moderate to high for all disorders, except for alcohol dependence (57%). Low sensitivity was primarily due to the high rate of false negative cases. The specificity of final diagnoses with the application was high for all the disorders. PPVs were high for all disorders, except that for OCD (0.38) and MR (0.5). Low PPVs were seen because of proportionately higher rates of false positive cases in OCD and MR categories. Similar to the screening sub-module, NPVs remained high because of the high rates of true negative cases.

Table III Frequency of diagnoses and agreement between diagnoses made by diagnostic sub-module and the Mini-International Neuropsychiatric Interview (MINI) (n=100)

Inter-rater reliability of the final diagnoses generated by the application: Inter-rater reliability of the final diagnoses generated by the application was examined by comparing the concordance between diagnoses made by ‘active’ (at remote site) and ‘passive’ (at nodal centre) raters. Table IV depicts the Cohen’ kappa values obtained as a part of this analysis. These revealed substantial to near-perfect agreement (kappa >0.6) for all the disorder categories.

Table IV Inter-rater reliability analysis (n=100)

Discussion

A newer approach of utilizing information and communication technology to develop computer-assisted methods for diagnosing16 and managing psychiatric disorders by non-psychiatrists has been suggested17. Amongst the few existing CDSSs, some have been developed to diagnose either a single disorder (e.g. schizophrenia, attention-deficit/hyperactivity disorder)1819 or common mental disorders16. Other CDSSs have been developed to guide treatment decisions alone2021 for one or more disorders. Most are stand-alone and not comprehensive, and none exists for Indian population. The objective of developing the telepsychiatry application was thus to create a knowledge-based computerized system that can guide the process of diagnosis and logically linked management for psychiatric disorders when used by general physicians and para-professionals at primary or secondary level of care.

For the application to be an effective diagnostic and management tool, validity of diagnoses is critical. Therefore, before the application was used at remote sites, it was validated at the nodal centre in two phases and underwent several revisions. In the first phase, the non-computerized version of the tool was tested against semi-structured clinical interviews in 100 patients9. During the second phase, the online tool was compared with the MINI in 274 patients and inter-rater reliability was also tested11. The results from these studies conducted at the nodal centre suggested that the application had acceptable level of accuracy and reliability and was feasible to use. The present study carried out at all the sites was the third level of validation. The initial screening was logically linked to detailed diagnostic evaluation opening only those diagnostic sub-modules, for which screening was positive. This contributed to the efficiency of entire diagnostic process and was reflected in the finding that unlike the screening sub-module, PPVs were high for most disorders and specificity and NPVs also improved for all disorders with detailed diagnostic evaluation. The kappa values indicated moderate (0.4-0.6) to high (>0.6) agreement between the application and the MINI for all disorders. This signifies acceptable to high diagnostic validity. When compared to concordance studies of other validated diagnostic interviews such as Composite International Diagnostic Interview and MINI122223, the application performed as well or better for most psychiatric disorders. Further, average time required for assessment was 20 min which was comparable to MINI12 and suited the requirements of a general healthcare setting.

In routine clinical practice, low and variable inter-rater reliability has been attributed to individual clinician factors such as clinician experience and knowledge than to problems with nosology24. It has been shown that CDSS is beneficial in improving practitioner performance, adherence to clinical practice guidelines and documentation2526. In the present study, the diagnostic sub-modules showed good inter-rater reliability when used by non-specialists. Further, the software gives treatment recommendations based on the diagnosis and other factors such as age, pregnancy status and medical disorders. The treatment consists of pharmacological and non-pharmacological intervention modules that have been developed based on standard practice guidelines, considerations of population and cultural characteristics amongst others. The accuracy and effectiveness of the treatment have been tested in separate studies2728. While using the telepsychiatry CDSS, it is required that the treatment should be prescribed only by the trained doctor and not the paraprofessional (except non-pharmacological interventions when recommended by the software and selected by the doctor). It also allows the doctor to over-rule the recommendations and prescribe on his/her own. Although the accuracy of CDSS is not 100 per cent, it fares reasonably well when compared with clinicians’ diagnosis and diagnosis made by a structured interview as seen in the previous studies91011 and the present study. By its design of encoded knowledge base and automation, the application attempts to overcome the clinician/individual-related factors such as inadequate or lack of knowledge, training and motivation to a large extent. These findings are significant and encouraging as the tool has been envisaged to enable and assist non-specialists in diagnosing and managing psychiatric disorders. Furthermore, the general physicians can seek a telepsychiatric consultation in case of difficult cases.

As the telepsychiatry application is available online and initial sensitization and training can also be conducted through video-conferencing, the primary and secondary care workers can attend to their routine duties also. Moreover, to ensure effectiveness, it has been emphasized that on-going supervision and monitoring from psychiatrists are extremely important4. The revised NMHP6 reflects that due to scarcity of mental health professionals, continued supervision and monitoring were lacking in the previous efforts. The telepsychiatry programme utilizes technology to overcome this difficulty as the supervision and monitoring are carried out from a distance through video-conferencing. However, in developing countries like India, conventional telepsychiatry models using video-conferencing alone are difficult to implement, as it simply re-distributes and burdens existing resources29. In addition, the use of the application by the primary or secondary care provider directly overcomes the difficulty faced in telepsychiatric consultations via video-conferencing such as issues of duty of care and patient and physician satisfaction29.

There were a few limitations of the present study. Although the sample size was adequate based on targeted kappa values, representation of certain disorders was limited in number. Hence, a larger sample would be required in the future. Further, the assessments on application and MINI were done serially in the same interview setting for the sake of convenience to the patients; interviewers’ bias might have been introduced. It would have been ideal to conduct two separate interviews to test inter-rater reliability. The patient would have to undergo multiple interviews on the same day to avoid any actual change in mental state that could affect the diagnosis. As stated earlier, this would have proven difficult for patients and their caregivers. In future, we need to study the effect of use of telepsychiatry application on performance of non-specialists and on patient outcomes.

In conclusion, the present findings suggested that the diagnostic tool had adequate diagnostic validity and reliability when the telepsychiatry application-based diagnostic interview was conducted by general physicians and other non-specialist professionals. This diagnostic tool had potential to empower non-psychiatrist doctors and paramedics to diagnose psychiatric disorders accurately and reliably in remote sites. Thus, when combined with the management system, the telepsychiatry application can be an effective tool to deliver mental health services through the agency of non-specialists at patients’ doorsteps.