Development, validation & pilot testing of a questionnaire to assess healthcare seeking behaviour, healthcare service utilization & out-of-pocket expenditure of Particularly Vulnerable Tribal Groups of Odisha, India

Healthcare seeking behaviour (HSB) is defined as ‘any action or inaction undertaken by individuals who perceive themselves to have a health problem or ill for the purpose of finding an appropriate remedy’1. HSB is a predominant factor that determines the health status of a country and thus their socioeconomic development2. HSB describes time taken from perceived illness to diagnosis; from diagnosis to receiving treatment and patient compliance with the treatment and type of healthcare services (ranging from informal to formal service providers) a person utilizes for his/her perceived illness3. Studies have highlighted the consequences of delayed or seeking no care, and/or seeking care from informal service providers in terms of risk of increased morbidity, worse health outcomes and mortality4,5.

HSB shapes the utilization of available healthcare services in the community and also drives out-of-pocket expenditures (OOPEs)6,7. Various studies6-10 have emphasized the variation of HSB and its utilization and related OOPEs with geospatial remoteness, culture, belief, literacy, socioeconomic status, type of illness, availability and quality of healthcare services. These factors predominantly affect the HSB of the most vulnerable strata of population, such as tribals in India6.

India is the country with the second largest tribal population in absolute terms constituting 8.6 per cent of the nation’s population11. Among them are the groups who are further marginalized and isolated, also geographically, and officially reclassified as Particularly Vulnerable Tribal Groups (PVTGs). Seventy five such PVTGs reside in India. Odisha harbours the highest number of PVTGs (n=13) in the country12. Their health profile is dominated by high maternal and child morbidity and mortality along with diseases caused by infection and malnutrition13, although disaggregated health data for PVTG as a group, let alone for individual PVTGs, are not available.

Worldwide, there is a handful of studies that examined HSB among the general population and tribals2,3,7-9. However, in a developing nation like India, public health researchers have not delved particularly into the HSB of tribes, except a few6,10, making it imperative to conduct specific studies to elicit information specifically on illness behaviour of the most vulnerable group among tribal PVTGs. Consequently, this genre of study demands PVTGs-specific and culture-sensitive study tools. Nevertheless, to the best of our knowledge, no validated quantitative questionnaire tool exits to measure the HSB specific to symptoms and syndromes. However, there exists pretested National Family Health Survey, 2019-2020 (NFHS-5) women’s questionnaire14, which if customized can help to capture tribe’s healthcare service utilization and related OOPEs of PVTGs of India. Against this backdrop, a parent study was designed to assess the symptom(s) specific HSB of PVTGs of Odisha and to explore their utilization of healthcare services and OOPEs of PVTGs through the lens of maternal and child health service utilization. The present study was aimed to describe the development of the questionnaire in this regard and report the results of the assessment of its validity.

Material & Methods

The present cross-sectional study was conducted by the department of Public Health, ICMR-Regional Medical Research Center (RMRC), Bhubaneswar, Odisha, from January to August 2021, after approval from both the Institute Human Ethical Committee (ICMR-RMRC/IHEC-2020/12) and the Research and Ethics Committee of the department of Health and Family Welfare, Government of Odisha (7033 /MS-2-IV-04/2020(PT).

Describing the sections of the questionnaire: A survey questionnaire was developed with two distinct sections; the first one, dealing with HSB of the respondents, was named HSB-Q. The HSB-Q was administered to individual who had illness or its exacerbation requiring mitigation in the last one year. The second section was concerned with measuring maternal and child healthcare service utilization (MCHSU-Q) and OOPEs related to the same. It was administered to women who had given birth to children (including still birth) in the preceding five years.

Due to the dearth of a validated questionnaire in the domain of HSB of indigenous people, a questionnaire tool was prepared de novo to address the first aim. However, for the second aim, MCHSU-Q was constructed after an existing questionnaire in this domain.

Phases of the questionnaire development process: The questionnaire was constructed in the following four phases, phase 1: questionnaire development; phase 2: assessing the validity of the questionnaire; phase 3: pilot testing; and phase 4: reliability assessment (Fig. 1).

Close

Fig. 1

Describing the phases involved in developing and validating the questionnaire.

Export to PPT

Phase 1: Questionnaire development:

Phase 1a: Development of a conceptual framework and identification of domains: A conceptual framework ensures identification of all the relevant domains of the underpinning research objective and their coverage in the study questionnaire. Therefore, for the HSB-Q section, phase 1a involved an intensive literature review in two databases PubMed and Embase using the following search criteria: ‘health seeking behaviour’, ‘illness behaviour’, ‘healthcare utilization’, ‘care seeking behaviour’, ‘indigenous’, ‘tribe’, ‘tribals’ and ‘India’. The process also included identification of domains – to which the items subsequently developed were mapped.

Phase 1b: Generation of questions (items): Relevant items for the HSB-Q section were generated through critical review of published, unpublished and grey literature, followed by qualitative methods that included focus group discussion (FGD) and in-depth interviews (IDIs), undertaken between January and March 2021. A total of four FGDs were conducted with two PVTGs – Paudi Bhuyan of Sundargarh and Anugul district and Kutia Kandha of Kalahandi district (2 FGDs from each group). In addition, three frontline health workers and two community organizers working with these PVTGs were purposively selected for IDIs. Responses from FGDs and IDIs were coded, then congregated into sub-themes which finally helped in the development of themes.

For the MCHSU-Q section, the NFHS-5 women’s questionnaire14 was referred to for generation of items.

Phase 2: Assessing the validity of the questionnaire: Validity is defined as whether the tool measures what it is intended to15,16.

Phase 2a: Face validity: In this study, face validity was measured using opinions of nine experts (4 public health professionals and 5 anthropologists – all having experience above 10 years), collected through two rounds of face-to-face consultation. They were asked to provide their subjective assessment over clarity, reasonability and unambiguity of each item and the format and construct of the first version of the questionnaire.

Phase 2b: Content validity: In contrast to face validity, the content validity involved quantitative scoring of items in terms of how well each item corresponds and measures what it purports to measure16,17.

The aforementioned nine domain experts scored each item of the questionnaire on the degree of relevance of each item to the measured domains using the following rating scale: (1) the item is not relevant to the measured domain, (2) the item is somewhat relevant to the measured domain, (3) the item is quite relevant to the measured domain, and (4) the item is highly relevant to the measured domain.

The content validity index (CVI) was estimated from the scores assigned by the experts, which are reported to be the most widely used validity measure18. Two CVIs were calculated; item level CVI (I-CVI) and scale level CVI (S-CVI). For the estimation of both the parameters, four point ordered Likert scale was dichotomized. The items scored as quite relevant or highly relevant were recoded as one and items scored as not relevant or somewhat relevant were recoded as zero.

As suggested by Polit et al19, the I-CVI was calculated for each item which was the sum of scores received by each item from the judges divided by the total number of judges. The estimate of I-CVI ranges from 0 to 1. The acceptable cut-off value for I-CVI is based on the number of experts. According to Lynn20, for nine experts I-CVI above 0.79 should be treated as a relevant, value ranging from 0.7-0.79 requires revision or otherwise, less than 0.7 should be eliminated.

Two methods are available for computing S-CVI, one is universal agreement among the experts (S-CVI/ UA) and another is Average CVI (S-CVI/Ave). S-CVI/ UA is estimated by a count of items that received a score of one divided by the total number of items. Meanwhile, S-CVI/Ave is the arithmetic mean of I-CVI (mentioned above). S-CVI/UA and S-CVI/Ave, having values ≥ 0.8 and ≥ 0.9, respectively, are considered as having excellent content validity18 (Fig. 1).

Moreover, Wynd et al21 suggested a combination of CVI as well as multi-rater kappa statistics for estimating the content validity of a tool. As multi-rater kappa statistics, unlike CVI adjusts for chance agreement, Kappa offers a degree of agreement beyond chance and is calculated using the following formula:

K = (I-CVI – Pc)/ (1- Pc)

where, Pc = probability of chance agreement and is calculated using the formula:

Pc = [N! / A! (N-A)!] * 0.5^N

here, N= total number of raters and A = experts in agreement to relevant items.

As described by Cicchetti and Sparrow22, the following kappa scores can be interpreted as Fair (0.4 to 0.59), Good (0.6 to 0.74) and Excellent (>0.74). However, as the number of experts rating the questionnaire increases, the likelihood of chance agreement also reduces and there is a convergence in the estimates of I-CVI and multirater kappa statistics.

Phase 2c: Pretesting of the questionnaire: A cognitive interview was conducted amongst the Saora PVTG in July using the third version of the questionnaire. A total of 15 interviews were conducted in three rounds, nine in the first round, three in the second round and three in the third round.

Phase 3: Pilot testing: A pilot test was conducted in July 2021 among the Saora PVTGs residing in Saora developmental agency, Chandragiri, Mohana block of Gajapati district, Odisha, India. The participants were informed about the title of the study, along with the underlying objectives. Participation was voluntary and signed informed consent was obtained from each participant. 335 and 100 eligible respondents were sampled for HSB-Q and MCHSU-Q sections, respectively.

Phase 4: Assessing the reliability of the questionnaire: The internal consistency reliability of de novo HSB-Q was assessed using Cronbach alpha (α), McDonald’s Omega coefficient (ω)23 in R version 4.2.1 using psych package for a subset of 71 respondents (who perceived their illness/es) out of 315. The scores of α and ω can be interpreted as questionable (0.6 to <0.7), acceptable (0.7 to <0.8), good (0.8 to 0.9) and excellent (>0.9)23.

Results

Phase 1: Questionnaire development:

Phase 1a: Development of a conceptual framework and Identification of domains: A conceptual framework called Axel Kroeger framework which is based on Anderson’s and Newman healthcare utilization model was identified. Axel Kroeger framework fits well with the objectives of the present study. In addition, it is applicable for developing countries like India. Through this framework and qualitative interviews, the current study identified four domains of HSB – predisposing factors, enabling factors, perceived need and treatment seeking options24 (Table). These four domains created a basic architecture of the HSB-Q section. Later, in phase 1b, questions were generated and mapped under these domains.

For the MCHSU-Q section, the study again drew upon the Axel Kroeger framework and eight domains were identified from the NFHS-5 women’s questionnaire - reproductive health, antenatal service utilization, intranatal service utilization, childbirth related OOPEs, postnatal service utilization, child immunization, family planning and menstrual health follow by abundant customization using local contextual issues and knowledge.

Phase 1b: Generation of questions (items): Using literature search, expert consultation and qualitative data, a total of 100 items were finally generated and mapped under four domains of HSB-Q.

For the MCHSU-Q section, eight domains and 100 items were referred from the pretested NFHS-5 women’s questionnaire with considerable customization using local contextual items such as ‘What was the total amount that you received under the MAMATA scheme?’ (Mamata scheme – financial assistance or incentive for delivery care applicable only in Odisha, India); ‘How much did it cost you out of your pocket to; (a) reach to Maa Gruha from home?; (b) transportation cost to reach to hospital from Maa Gruha?; (c) stay for expectant mothers and dependents?; (d) food for expectant mothers and dependents?; and (e) other costs? (Maa Gruha – maternity waiting homes, a special initiative for the promotion of institutional delivery in inaccessible pockets of tribal areas of Odisha, India); and ‘if using sanitary napkin, then from where have you obtained the absorbent?’ (Khushi scheme – to provide good menstrual hygiene care to women and adolescent girls studying in class 6 to 12 Odisha, India).

A first version of the questionnaire was developed containing 12 domains (4 domains of HSB-Q section and eight domains of the MCHSU-Q section) and 200 items and then the entire instrument was subjected to phase 2, i.e., validity assessment.

Phase 2: Assessing the validity of the questionnaire:

Phase 2a: Face validity: After considering the consensus amongst experts, 55 out of 200 items were eliminated from the first version of the questionnaire. The second version comprising 145 items was subjected to the second round of consultation. According to the experts, the final items were clear, succinct and relevant and the questionnaire covered the intended objectives of the study. However, a common concern expressed by the experts was that of the length of the entire questionnaire, even after the removal of 55 items.

Phase 2b: Content validity: Of the 145 items, 27 had I-CVI equal to 0.89 and 106 items had I-CVI equal to 1. Furthermore, 10 items having an I-CVI score of 0.78 were modified according to experts’ suggestions and two items having I-CVI score of less than 0.7 were eliminated (Supplementary Table I). Most of the items came out to be relevant except two under the domain treatment-seeking options; one was ‘Has anybody suggested/referred you to take 4th action for the same episode’ and the second was ‘Has anybody suggested/referred you to take 5th action for the same episode’. Therefore, the third version of the questionnaire was prepared, excluding these two items (Supplementary Table I).

The S-CVI/UA and S-CVI/Ave for 145 items were 0.73 and 0.96, respectively. Overall, there was moderate content validity of the questionnaire according to S-CVI/UA estimates and excellent as per the estimates of S-CVI/Ave.

Only two items had the Kappa value of 0.6, showing a good degree of agreement beyond chance, whereas remaining all the items had the kappa value above 0.74, therefore, presenting an excellent degree of agreement for the rest of the items (Supplementary Table I).

A third version of the questionnaire was developed at the end of phase 2b.

Phase 2c: Pretesting of the questionnaire: First round of cognitive interviews: The response of each of the nine participants for each item were sought, received and analysed immediately. It was found that the respondents understood the broad essence of all the items but faced problems comprehending certain terms in a few items. Using their feedback, the items were modified accordingly. For instance, in section 1: HSB-Q, one more category of response ‘perceived need but took no action’ was added to the item ‘what action did you take for the episode of perceived illness?’. Similarly, one more option ‘don’t remember’ was added to the item ‘Lag time from onset of an episode of perceived illness to 1st consultation’. This process also led to the clubbing of two separate subsections, acute and chronic illness in HSB-Q into one called self-reported illnesses. Subsequently, after necessary modifications, a fourth version of the questionnaire was created at the end of this round.

Second round of cognitive interviews: The protocol of interview was the same as in the first round. This round also brought about certain revisions in the items such as the addition of responses, altering texts of items, reordering positions of some of the items and responses. For instance, while interviewing mothers having an under-5 child, it was observed that majority of them did not keep Mother and Child Protection Card (MCPC) with themselves. MCPC was kept at a child day-care centre (Anganwadi centre). Hence, based on this observation, two more items ‘do you keep your Mother Child Protection Card (MCPC) with you (at your home)?’, ‘if no, then where is it kept usually?’ was added to MCHSU-Q. In addition, it was found that one important question was missing out from MCHSU-Q that is ‘what was your age at first pregnancy?’. Hence, all these alterations at the end of the second round led to the fifth version of the questionnaire.

Third round of cognitive interviews: the subsequent interviews in the third which was the final round resolved other remaining issues and the sixth version of the questionnaire was used for pilot testing.

Phase 3: Pilot testing: The mean age of the respondents in HSB-Q was 24 (±19) years, constituting 51.4 per cent of male. 63.2 per cent of them had no formal education (Supplementary Table II). Of 315 individuals interviewed, only 71 (22.5%) individuals perceived their illness. Among respondents’ having perceived illness, 20.3 per cent had felt the need to seek care. Their treatment-seeking pathway ranged from seeking no care (9.9%) to home remedies (11.3%), advice from elders (2.8%), consulting informal service providers [47.9% (43.7% from Bengali doctor, 2.8% from traditional healer and 1.4% from spiritual healer)] and formal providers (28.1%) (Fig. 2).

Close

Fig. 2

Healthcare seeking behaviour for perceived illness.

Export to PPT

For MCHSU-Q section 54, out of 100 mothers who had given birth to children (including stillbirth) in the preceding five years consented to participate in the study. The mean age of the mothers was 28.5 [standard deviation (SD)±5.2] years. Most of them (64.8%) had no formal education. About 16.7 per cent of the mothers got married before 18 years of age and had at least one under-5 child. Majority (64.8%) had delivered at-home spending on an average ₹ 1377 (SD ±1225.5) from out-of-pocket (Supplementary Table III). Supplementary Table IV illustrates the immunization status of 54 under-5 children, aged between 20 days and 59 months (mean age was 28.1(SD±16.9) months). Amongst them, 61.1 per cent were male. The utilization of their immunization varied at birth as only 31.5 per cent of them received hepatitis B birth dose whereas polio and BCG were utilized by 92.6 per cent and 98.1 per cent.

Phase 4: Assessing the reliability of the questionnaire: The Cronbach alpha (α) and McDonald’s omega (ω) were 0.8 and 0.85, respectively, indicating good internal consistency of the HSB-Q.

Discussion

This study describes the development process of the questionnaire meant for measuring healthcare seeking behaviour, maternal and child healthcare service utilization and OOPEs of PVTGs of Odisha. It also reports the results of the assessment of its validity and reliability. The development of the de novo section of the questionnaire, i.e., the portion concerned with the HSB-Q is the first of its kind to our knowledge that has been designed specifically for studying how the PVTGs seek care in their illness(es). It took 30 min to fill each questionnaire.

The questionnaire was found to have adequate face and content validity. Various validity statistics that included item level content validity index (I-CVI), scale level content validity index (S-CVI)— [universal agreement (S-CVI/UA) and average CVI (S-CVI/Ave)] and multirater kappa statistics proved the adequacy. The questionnaire underwent changes in terms of the inclusion and exclusion of items and their modification in various phases of its development. Further, the questionnaire was pilot tested in a sample of PVTGs, which helped to refine the interviewing technique and interaction modalities with the members of the PVTGs, given that majority of them had little or no formal education and they had their unique cultural practices when it came to seeking care. The pilot also helped us to plan for the logistics required for the main study. In addition, the reliability test performed for the de novo HSB-Q indicated good internal consistency.

The final questionnaire obtained at the end of this study is set to address the envisaged objectives of the parent study. Overall, the newly developed tool can feasibly be replicated to capture variations in symptom specific HSB and utilization of maternal and child health and related OOPEs incurred by the Indian tribal subpopulation. This tool can also be used among nontribal Indian population after context and programme specific customization of tool is conducted.

However, the questionnaire had certain limitations; first, the concerns raised unanimously by experts at the time of content validation were the length of the questionnaire. However, the items that were finally retained in the questionnaire were judged by the same experts to be very relevant to the point of indispensability. Moreover, during the pilot test, it was observed that the respondents did not lose interest while answering the questions because of its length. Second, test-retest reliability was not assessed due to resource constraints. The strength of the present investigation rests with the procedural adaptation and customization of universal frameworks in this domain to suit the context of the PVTGs of Odisha.

To conclude, the process led to the genesis of a valid, reliable, pre-tested and pilot tested questionnaire to assess HSB, service utilization pattern and OOPE of PVTGs of Odisha, India. The final questionnaire encompassed 12 domains and 140 items (HSB-Q section-4 domains, 67 items and MCHSU-Q section-8 domains, 73 items) [Annexure 1].

A cross-sectional survey can be designed for the Indian tribal population using the HSB-Q tool to provide a body of evidence, based on which tailored interventions for PVTGs can be developed.