Validity

Validity implies that the data collected results in precise and exact outcomes. In technical terms, a measure enables researchers to draw proper and correct conclusions from the sample, which are generalizable to the entire population.

Four Major types:

1. Internal validity: When the relationship between variables is causal.  This type refers to the relationship between the dependent and independent variables, which helps in understanding how changes in the independent variable affect the dependent variable. In this case, the design of the experiment is associated with it, and it is particularly relevant in studies that aim to establish a causal relationship. For example, it can be used for the random assignment of treatments.

2. External validity: When there is a causal relationship between the cause and effect that can be transferred to people, treatments, variables, and different measurement variables which differ from the other.

3. Statistical conclusion validity: The conclusion reached, or alternatively, the inference drawn about the extent of the relationship between the two variables, can, therefore, be significant. Additionally, this can provide valuable insights into their connection, further enhancing the understanding of the underlying relationship. Consequently, understanding this relationship is crucial for further analysis. For instance, we can find it when we aim to determine the strength of the relationship between any two variables that have been under observation and analysis. If we reach the correct conclusion, it is referred to as statistical conclusion validity.

There are two types of statistical conclusion validity.

They are as follows:

a. Type one error: Type one error is when we conclude that there is a relationship between two variables and we reject a true null hypothesis when in reality, there is no relationship between the two variables.  This is in fact very dangerous.

b.Type two errors: If we fail to reject a false null hypothesis that is true it is called type two error.

In statistical conclusion validity, researchers use the method of power analysis to detect the relationship and determine whether the sample size is adequate to identify a significant effect.  Several problems crop up while making a statistical conclusion.  For instance, if a small sample size is used, then there is the possibility that the result will not be correct. In technical terms, a measure enables researchers to draw proper and correct conclusions from the sample that are generalizable to the entire population. Statistical validity is also compromised by the violation of statistical assumptions. The results may become inaccurate if the values in the analysis are biased and the wrong statistical test is applied.

4.Construct validity: Extent that a measurement actually represents the construct it is measuring.  For instance, in structural equation modeling, when we draw the construct, then we presume that the factor loading for the construct is greater than .7.  Cronbach’s alpha is used to assess construct validity. Specifically, for exploratory purposes, a value of .60 is accepted. However, for confirmatory purposes, a value of .70 is accepted, and furthermore, a value of .80 is considered good.  If the construct satisfies the above presumption and expectation, then the construct would be helpful in predicting the relationship for dependent variables. Researchers also use convergent/divergent validation and factor analysis to test construct validity.

Relationship between reliability and validity: There is no way that a test that is unreliable is valid.  Again, any test that is valid must be reliable.  From this statement, we can infer that validity plays a significant role in analysis, as it ensures the conclusion of accurate results.

Overall threats:

1.Insufficient data collected to make valid conclusion
2.Measurement done with too few measurement variables
3.Too much variation in data or outliers in data
4.Wrong selection of samples
5.Inaccurate measurement method taken for analysis

Resources

Bagozzi, R. P., Yi, Y., & Phillips, L. W. (1991). Assessing construct validity in organizational research. Administrative Science Quarterly, 36(3), 421-458.

Brinkman, W. -P., Haakma, R., & Bouwhuis, D. G. (2009). The theoretical foundation and validity of a component-based usability questionnaire. Behaviour & Information Technology, 28(2), 121-137.

Cronbach, L. J. (1971). Test validation. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 443-507). Washington, DC: American Council on Education.

Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281-302.

Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39-50.

Guilford, J. P. (1946). New standards for test evaluation. Educational and Psychological Measurement, 6(5), 427-439.

Krause, M. S. (1972). The implications of convergent and discriminant validity data for instrument validation. Psychometrika, 37(2), 179-186.

Lieberman, D. Z. (2008). Evaluation of the stability and validity of participant samples recruited over the internet. CyberPsychology & Behavior, 11(6), 743-746.

Lozano, L. M., Carcía-Cueto, E., & Muñoz, J. (2008). Effect of the number of response categories on the reliability and validity of rating scales. Methodology, 4(2), 73-79.

Messick, S. (1989). Validity. In R. L. Linn (Ed.), Education measurement (3rd ed., pp. 13-103). Washington, DC: American Council on Education.

Moret, M., Reuzel, R., van der Wilt, G. J., & Grin, J. (2007). Validity and reliability of qualitative data analysis: Interobserver agreement in reconstructing interpretative frames. Field Methods, 19(1), 24-39.

Rosenbaum, P. R. (1989). Criterion-related construct validity. Psychometrika, 54(4), 625-659.

Shepard, L. A. (1993). Evaluating test validity. Review of Research in Education, 19, 405-450.

Administration, Analysis and Reporting

Intellectus Consulting can assist the student or professional researcher in administering the survey instrument, collecting the data, conducting the analyses and explaining the results.
For additional information on these services, click here.

Related Pages:

Structural Equation Modeling

Conduct and Interpret a Factor Analysis