Practice Question – How can one resolve the issue of reliability and validity in the context of sociological research on inequality? (UPSC 2017)
Approach – Introduction, Write on issue of reliability and validity in social research, Bring in the practical difficulties faced while researching on inequality with respect to validity and reliability, Suggest solutions, Conclusion.
When an instrument of data collection used by the researcher yields a particular set of data, another researcher should be able to derive similar data using the same instrument. Or, the same researcher should be able to derive similar data using the same instrument at another point of time. This refers to the notion called repeatability and consistency which is closely associated with reliability. Such repeatability and consistency may be possible with the instruments to a greatest extent in natural sciences. But, in social science research, there are inherent limitations and it is difficult to talk about reliability in the same sense we talk about it in natural sciences. However, over a period, there has been a steady advancement of tests which qualify the reliability of certain instruments like questionnaire or interview schedule used social research.
Reliability in general sense refers to consistency or repeatability. Consistency or repeatability of results is concerned with the instrument used in data collection, methodology adopted in the study and research design. ‘Reliability is the degree to which a variable or test yields the same results when administered to the same people, under the same circumstances’. The research instrument is considered to be reliable when the results obtained using the same instrument is consistent over time and space. A research study may be said to be reliable if the results can be reproduced using the same methodology.
The reliability of measuring instruments can be improved by two ways.
i) By standardizing the conditions under which the measurement takes place i.e. we must ensure that external sources of variation such as boredom, fatigue etc., are minimized to the extent possible to improve the stability aspect.
ii) By carefully designing directions for measurement with no variation from group to group, by using trained and motivated persons to conduct the research and also by broadening the sample of items used to improve equivalence aspect.
In the test-retest method, the instrument is administered twice, at two different points of time. Important points to remember are: a) the instrument (questionnaire or interview schedule, or a scale) remains same in two occasions, and b) the instrument is administered to the same set of respondents. If the correlation values show consistency between the two tests, then the instrument is said to be reliable.
In parallel forms method, two sets of instrument are prepared. The two sets contain questions which provide same meaning. The researcher has to generate multiple questions aimed at measuring the same variable. So, in parallel forms reliability test, instead of generating questions for one set of questionnaire, sufficient number of questions are generated so that they can be divided into two sets of questionnaire. Multiple questions which address same construct are randomly divided into two sets. These two sets of questionnaire are administered to the same set of respondents. The correlation between these two sets is estimated. The two sets of questionnaires are equivalent measures. Hence, this test is called parallel forms method.
In split-halves method, the instrument is administered once to the respondents. However, the responses are tested for consistency by splitting the instrument (questionnaire) into two halves. Each half consists of questions which are similar to the other half. Then, the correlation between the two halves is calculated. This method is dissimilar to the earlier methods as only one questionnaire is administered at one point of time. It overcomes the problems of cueing and generating multiple questions. Moreover, it is administered only once.
In internal consistency method, reliability is estimated by grouping questions in the questionnaire that measure the same concept. Instead of generating one question to measure the concept, researcher has to evolve two groups of questions, each group consisting of three or more questions that measure the concept. The questions in the two groups aim at measuring the same concept. The instrument is administered only once. The responses to the questions under two groups are correlated. This method enables the researcher to measure the reliability of the instrument by checking the consistency between two groups of questions. Point to be noted is that, it is not just two groups of questions, but the researcher can generate as many questions as possible and group them so that correlation between them is calculated.
Validity is based with the degree of achieving the intended result. A result is valid if it achieves what it was supposed to achieve. Validity determines the success of a study or research. Validity is measured in terms of desired output or goal. When such a goal is not fixed there are problems in ascertaining the validity of a result. According to Alan Bryman, there are four types of validity-
- Internal validity – It affirms the causal relationship
- External validity– it pertains to generalized aspect and the degree to which the results apply to a larger population.
- Measurement validity or construct validity– It is concerned with the fact that whether the measure which is being employed actually measures what it claims.
- Ecological validity– It refers to the fact that how closely a research study mirrors the natural settings of people’s real experiences. A valid research has to be as natural in its settings as it can.
Factors influencing validity
- History, as the time passes, variables also take different values.
- Instrumentation refers to the effect caused by changes in measuring instrument or method during the research.
- Selection bias occurs when the test units are selected in such a way that they are not representative of the population.
Types of validity tests
The most commonly used validity test is face validity test. The instrument (for example, questionnaire or a scale) is accepted as valid if it appears valid for the researcher. Here researcher, as a professional, makes a judgement about the validity of the instrument. It is a casual review of the questions or items incorporated in the instrument. Sometimes, face validity is conducted by individuals who may not have any professional training or formal knowledge. It is the simplest and easiest method of checking the validity of a scale or a questionnaire.
Instruments such as scales are developed to make predictions. For example, the entrance test conducted to select candidates for admission into IITs is an instrument used to make predictions about the academic ability of the candidates. The items (questions) incorporated in the instrument must reflect the larger goal of the instrument. Hence, in content validity, the items are subjected to review by those who are formally trained and have expertise in the subject under study. Usually individuals with considerable domain knowledge are asked to review whether the items used, measure the intended property or not. Consensus opinion is considered in finalizing the instrument. This type of validity test is mostly used by the researchers.
It is conducted to measure the validity of the instrument against the criteria set in the study. Two types of tests are considered in criterion validity test. They are concurrent validity test and predictive validity test. Concurrent validity test is conducted to measure the extent to which the items of the instrument correlate with the ‘gold standard’ available. Generally, standardized, established instruments are used as references to check the validity of the instrument being tested. The predictive validity test measures the extent to which the instrument predicts the expected future observation. For example, instrument developed to measure IQ must help in making the predictions of IQ levels of the respondents.
RELEVANCE OF RELIABILITY AND VALIDITY
The real difference between reliability and validity is mostly a matter of definition. Reliability estimates the consistency of your measurement, or more simply the degree to which an instrument measures the same way each time it is used in under the same conditions with the same subjects. Validity, on the other hand, involves the degree to which you are measuring what you are supposed to, more simply, the accuracy of your measurement. Reliability refers to the consistency or stability of the test scores; validity refers to the accuracy of the inferences or interpretations you make from the test scores.
Reliability and validity of quantitative data, instruments used in their collection is verifiable through the tests established through standard procedures. Reliability of quantitative data and instruments can be estimated. Qualitative researchers, on the other hand, face the challenges of repeatability or consistency and validity of data and instruments of a different kind.
As the researchers are the instruments of data collection, problems associated with personal as well as professional life crop up in qualitative research. Researcher’s bias, effect of the researcher on the setting, theoretical assumptions influence the process and direction of research. These issues pose challenges to reliability and validity of the data, interpretations and conclusions. However, the techniques to overcome such challenges have been developed and available to the researcher. It is the burden of the researcher to consider the suggestions and practice them during the course of research in order to enhance reliability and validity.