Introduction
One of the biggest challenges when conducting research is developing and using a valid instrument to collect meaningful data. It is not uncommon for researchers – especially those just starting out – to conduct small sample studies using instruments that have not been properly validated. While this can be a learning experience, relying on non-validated measures can seriously undermine the credibility of any findings or conclusions drawn from the study.
This paper will examine a hypothetical sample research study where the researchers developed and used their own non-validated instrument to measure career motivation among college students. Through reviewing the different components of instrument validation that were not addressed, we can analyze some of the limitations and flaws with relying on a non-validated measure. The goal is to illustrate for early career researchers the importance of properly validating any instruments used in empirical research studies.
Background and Methods
For their sample study, two graduate students in counseling psychology developed a 20-item questionnaire to assess factors related to career motivation among college undergraduates. They created items to measure constructs like intrinsic motivation, self-efficacy, career aspirations, and perception of career opportunities post-graduation. The items used a 5-point Likert scale ranging from “strongly disagree” to “strongly agree.”
The researchers administered their non-validated career motivation questionnaire to a convenience sample of 50 undergraduate students enrolled in a large public university. Participants completed the survey either in-person following a class or online through a link distributed via email. Basic demographic information like gender, class year, major, and GPA was also collected.
Once data collection was complete, the researchers conducted analysis on the responses. They ran descriptive statistics to examine response patterns on individual items and mean scores across items thought to measure specific constructs (e.g. intrinsic motivation). Correlations and regressions were also used to examine relationships between career motivation and demographic variables like gender and GPA.
Limitations of Using a Non-Validated Instrument
There are several major limitations to relying on a non-validated instrument like the one used in this hypothetical sample study. These limitations seriously call into question the credibility and meaningfulness of any findings or conclusions:
Lack of evidence of reliability. Reliability testing was not conducted to examine the internal consistency of items thought to measure the same construct (e.g. Cronbach’s alpha). Without establishing reliability, there is no way to know if the items are consistently measuring the intended constructs over time.
Unknown validity. Various types of validity testing were not performed, including content, criterion, and construct validity. Without properly validating the items and mapping them to the theoretical constructs of interest, it is unknown if the instrument is accurately measuring what it is intended to measure.
Potential response bias issues. No pre-testing of items was done to identify potential problems like ambiguous wording, leading questions, or social desirability bias. Response patterns could be influenced by issues not captured by the instrument creators.
Improper scoring. Since valid subscales were not established, simply summing responses to groups of items assumed to measure a construct like intrinsic motivation may not accurately represent scores on that construct.
Overreliance on weak findings. Any “significant” or “meaningful” correlations found between variables must be interpreted with extreme caution given the limitations of the non-validated instrument used. Findings may be spurious correlations rather than true relationships.
Generalizability concerns. Findings from the small convenience sample may not generalize to the broader population given the limitations of the measure. Validity evidence would be needed to extend conclusions more broadly.
No guidance for future use. Without clear documentation of the development and validation process, other researchers have no information on how to properly evaluate or replicate use of this measure going forward.
Addressing Instrument Validation Needs
To properly address these limitations and validate their career motivation instrument, the researchers would need to conduct additional phases of instrument development and testing:
Item generation: Clearly define constructs of interest, perform literature review, develop theory-based items through subject matter experts.
Content validity testing: Expert review of items, revisions based on feedback, establishing linkage to constructs.
Pilot testing: Cognitive interviews to identify comprehension/response issues, revised based on pilot participant feedback.
Reliability testing: Test internal consistency of hypothesized subscales on new sample.
Construct validity testing: Examine factor structure through EFA/CFA, fit to theoretical model.
Criterion validity: Correlate subscales with other validated similar/dissimilar constructs for divergent/convergent evidence.
Large scale validation: Administer validated version to large diverse sample, replicate reliability/validity studies.
Manual development: Document all phases of development/validation/scoring for future researchers.
The researchers of this study would need to conduct multiple iterations of instrument refinement and psychometric evaluation with independent samples before conclusions could be credibly drawn about relationships between career motivation and other variables based on this measure. Proper validation takes significant time and resources but is critical for ensuring research quality and credibility of findings.
Discussion and Conclusion
While this hypothetical sample study provided a learning experience for the early career researchers, the use of a non-validated instrument to measure career motivation among college students significantly limits the credibility and meaningfulness of any findings or conclusions that could be drawn. The major limitations related to lack of reliability evidence, unknown validity, potential response biases, improper scoring, and inability to generalize were not properly addressed.
For any research employing original measures, it is imperative that the instrument first undergoes a rigorous validation process establishing reliability and validity evidence before being used to collect empirical data. Even for small sample studies, some initial phases of validation like expert review and pilot testing should be conducted. Proper validation protects against spurious or misleading findings, enhances the quality and credibility of research, and provides guidance for future instrument use.
With the necessary time and resources invested in multiple phases of instrument development and psychometric evaluation, the career motivation questionnaire created for this sample study could potentially be validated for future research examining related questions. As structured, the study fails to meet basic scientific standards due largely to reliance on a measure lacking any validation. This lesson demonstrates for early career researchers pursuing sample studies the importance of validated instruments for collecting meaningful empirical data.
Andrew Stroud