Writing for Publication: Transitions and Tools that Support Scholars’ Success - Mary Renck Jalongo, Olivia N. Saracho 2016

Validity Issues in Mixed Methods Research
From Mixed-Methods Research to a Journal Article
Conference Proposals and Article Types

All research findings are exposed to threats of descriptive validity (accuracy of explanations), interpretive validity (researchers’ interpretations of the participants’ behavior), internal validity (instrumentation), and external validity (Campbell, 1957; Onwuegbuzie, 2003).

Both the quantitative and qualitative data need to be assessed for data validation/legitimation. Legitimation is the trustworthiness, credibility, dependability, confirmability, and/or transferability of the researchers’ inferences (Onwuegbuzie & Leech, 2007; Leech & Onwuegbuzie, 2011). Lack of legitimation “means that the extent to which the data have been captured has not been adequately assessed, or that any such assessment has not provided support for legitimation” (Onwuegbuzie & Leech, 2004, p. 778).

For qualitative data a detailed description of any threats to trustworthiness, credibility, dependability, authenticity, verification, plausibility, applicability, confirmability, and/or transferability of data (Creswell 2013a, b; Miles, Huberman, & Saldaña 2013) needs to be provided. All verification procedures used need to be discussed. The overall mixed methods research need to have an in-depth discussion of legitimation concerning the quantitative and qualitative analyses. Returning to the study of doctoral students and challenges they faced in reading research, the authors provided this discussion of threats to validity and legitimization of their approach:

Validity of findings from quantitative phase. Threats both to internal validity and external validity prevailed with respect to the quantitative findings (Campbell, 1957; Campbell & Stanley, 1963, 1966). The biggest threat to the internal validity of the quantitative findings was instrumentation because of the relatively low reliability coefficient (i.e., .69) pertaining to the reading comprehension scores, which can affect statistical power (Onwuegbuzie & Daniel, 2004).

With regard to external validity, because the sample represented doctoral students at a single university (i.e., threat to population validity and ecological validity) from whom data were collected at a single point in time (i.e., threat to temporal validity), it is not clear the extent to which the present findings generalize beyond the sample to doctoral students from other institutions in other regions of the United States and beyond.

Legitimation of findings from qualitative phase. The biggest threats to the qualitative findings were descriptive validity (i.e., factual accuracy of the reasons provided by the doctoral students) and interpretive validity (i.e., the extent to which a researcher’s interpretation of the reasons provided represents an understanding of the students’ perspectives and the meanings that they attach to their words and actions)… However, descriptive validity and interpretive validity were enhanced by member checking … all the themes secured endorsement rates that yielded at least small-to-medium effect sizes suggests that data saturation took place.

Legitimation from the mixed research phase. It can be seen that nine threats were addressed to some degree. Nevertheless, despite the extremely rigorous nature of the mixed research design, replications of this inquiry are needed to assess the reliability of the current findings. (Burgess, Benge, Onwuegbuzie, and Mallette 2012, pp. 23—24)

A part of mixed methods research that is sometimes surprising to authors is reformulating the research questions. Based on the results, the goal, objective, rationale, purpose, and research questions are examined to propose new research questions. The mixed methods research report needs to explain how the research questions can be reformulated. Reformulating all research procedures leads to recommendations for future research that will conclude in a validation, replication, or expansion of the study (Leech, 2012; Leech & Onwuegbuzie, 2011). To illustrate, Burgess, Benge, Onwuegbuzie, and Mallette’s (2012) study found five themes that described doctoral students’ reasons for reading research articles. In addition, a series of canonical correlation analyses showed relationships between reasons for reading empirical articles and (a) reading intensity (i.e., frequency of reading empirical research articles, number of empirical research articles read each month) and (b) reading ability (i.e., reading comprehension, reading vocabulary). Based on these findings, Burgess, Benge, Onwuegbuzie, and Mallette (2012) reformulated the mixed methods research question for researchers to use in the future: What is the relationship between doctoral students’ reasons for reading empirical literature and their perceived barriers to reading empirical literature? (p. 28)