Development & validation of a format for reporting endoscopic colonic biopsies

Literature review to identity relevant questions

Two pathologists (KK and BN) with a special interest in gastropathology conducted a literature search across multiple databases, including PubMed, Cochrane Library, EMBASE, Google Scholar, and the Stanford Library. Based on the compiled diagnostic information, they developed the initial format by identifying relevant questions. The keywords and search terms employed included ‘Colon Biopsy Interpretation’, ‘Algorithm for Colon biopsy reporting’, ‘Inflammatory Bowel Disease’, ‘Ulcerative Colitis’, ‘Crohn’s Disease’, ‘Crohn’s vs. Tuberculosis’, ‘Tubercular Colitis’, ‘Granulomas in colon’, ‘Infectious Granulomas in Colon’, ‘Dysplasia colon’, ‘Histologic Activity index in IBD’, ‘Ischemic Colitis’, ‘GVHD in Colon’ etc. Few references were manually searched. Around 15 articles were found including original articles, reviews, meta-analyses, case reports, etc.

Identification of experts

Informed consent was taken telephonically from all experts before sending them Google form. Their participation was voluntary. Six participants didn’t respond as they believed that they didn’t have enough expertise.

Defining questions for checklist

Initially 27 questions (Items) were framed which were increased to 71 questions after various rounds of suggestions from experts and literature search to make it more extensive and comprehensive. Further suggestions from the participating researcher colleagues included in this study resulted in adding same multiple choice type answers for every questions to make it easy to respond. The final questionnaire was a Google form, and the responses were recorded as grades/scores on a 5 point Likert Scale for relevance. This was repeated for all the 71 items. One point was assigned for strongly disagree, 2 for disagree; 3 for neutral (don’t know/may or may not be included); 4 for agree and 5 for strongly agree. Further review of questionnaire resulted in making sections and adding related points under the relevant section. Then questionnaire was sent to junior and middle level faculty members of Pathology Department of Dr S N Medical College, Jodhpur to look for any spelling mistake, repeat question, question requiring reframing, etc., and the suggestions were accepted and necessary changes were done. The final questionnaire took four months to complete ready to be sent to experts to respond (From March 2024 to June 2024).

Questionnaire validation

The final version of questionnaire (supplementary material) was sent to 20 histopathologists across India, who either had a long experience in the respective subject or belonged to institutes of national importance. The experts responded and reverted after assessing and grading the questionnaire (Google forms shared on mail or WhatsApp). The expected response rate was 50 per cent and we got responses from 14 experts out of 20 (70%).

Content validation

Content validity assesses whether the items in the format sufficiently represent the domain of interest. In developing this format, the two-step methodology described by Armstrong et al⁹ was utilized. In the first stage we did synthesis of sections, followed by questions in each section; following this, the items were evaluated by experts, and their validity was quantitatively determined using the CVI and MKS.

For statistical calculation, Likert response point 4 and 5 (Agree and strongly agree) were considered as agreement and given 1 point, while Likert response points 1, 2 and 3 (Strongly disagree, disagree and neutral) were considered non-agreement and given 0 point. The data were analyzed using Microsoft Excel spreadsheet Mac Version 16.36 (Microsoft Corporation; Redmond, Washington; United States).

The various statistical calculations done as follows with their formulae: (i) Experts in agreement were determined by summing the relevance ratings assigned to each item; (ii) Universal Agreement (UA) An item received a score of ‘1’ if there was complete agreement among all experts, and a score of ‘0’ was assigned if even one expert disagreed. (iii) I-CVI (Item wise-content validity index): represents the fraction of experts who concurred on an item, calculated by dividing agreed items by the total experts.(iv) Scale-wise CVI (S-CVI/Ave): (based on I-CVI): Calculated by averaging the I-CVI scores across all items, i.e., the total of I-CVI values divided by the number of items; (v) S-CVI/Average (by proportional relevance method): obtained by summing the proportional relevance scores and dividing by the total number of experts; S-CVI/UA (Scale wise- content validity index based on universal agreement method): Sum of UA scores/No of items. So, in summary when the number of experts is at least nine, the Item-level Content Validity Index (I-CVI) for each item, as well as the Scale-level CVI (S-CVI) calculated either by averaging the I-CVIs across items (S-CVI/Ave) or by universal agreement among experts (S-CVI/UA), should all exceed the threshold of 0.78. This cutoff is widely accepted in the literature as indicating adequate content validity. Values below this level suggests insufficient expert agreement on the relevance of the items^10,11.

To evaluate expert consensus on each question in all sections, the MKS was utilized.

Kappa was calculated using the following formula:

$\text{K}=\left(\text{I}-\text{CVI}-\text{Pc}\right)/\left(\text{1}-\text{Pc}\right);$

where Pc is the probability of chance agreement on relevance calculated by formula:^6,7;

$Pc=[N!/A!*(N-A)!]*{0.5}^N;$

N= number of experts; A=No of experts in agreement.

Kappa values were interpreted on a scale ranging from 0 to 1, where values between 0 and 0.2 indicated no agreement, 0.21 to 0.39 represented minimal agreement, and 0.4 to 0.59 corresponded to weak agreement. Moderate agreement was reflected by values from 0.6 to 0.79, while strong agreement falls between 0.8 and 0.9. Values above 0.9 suggest almost perfect agreement, with a value of 1 indicating perfect agreement among experts.

In this study expected K value was >0.8 to consider for strong agreement among experts on relevance of various questions/histological features to be looked in for an endoscopic colonic biopsy.

Probability of chance agreement is typically experienced low to avoid inflation of the I-CVI by random consensus. In our study, we obtained Pc values below 0.006, indicating a negligible likelihood of experts agreeing purely by chance. This very low Pc contributed to a modified Kappa value (κ) greater than 0.80, reflecting excellent agreement beyond chance and confirming the robustness of the content validity.