Overall Evaluation of a Quantitative Study - From a Research Project to a Journal Article - Conference Proposals and Article Types

Writing for Publication: Transitions and Tools that Support Scholars’ Success - Mary Renck Jalongo, Olivia N. Saracho 2016

Overall Evaluation of a Quantitative Study
From a Research Project to a Journal Article
Conference Proposals and Article Types

Credibility of the study is based on the researcher’s ability to effectively design, execute, and describe the project. Therefore, it is important that researchers evaluate the presentation of their study before submitting it to a journal. Quantitative studies need to be evaluated to determine their contribution to the field. The evaluation process needs to objectively assess the strengths and the weaknesses of a report. Researchers need to consider if the strengths of the study are better than its weaknesses, the results influence practice, and the results suggest future research directions. Evaluating the quantitative research report may initially seem like an overwhelming chore but using a systematic approach can help researchers be more at ease and capable of evaluating their quantitative research reports (Russell, 2005). What if you could get a “report card” on your quantitative study prior to submitting it? Authors are sometimes unaware that there are many such self-evaluative tools in existence. One that we found helpful is in Table 7.4. Going through a set of questions such as these is especially useful if you are writing a quantitative research report with a team and different people are writing various sections of the manuscript.

Table 7.4

Tool for self-evaluation of quantitative research

Introduction and review of the literature

__________Is the problem introduced?

__________Does the problem establish the importance of the study?

__________Is there a discussion on how the study will advance knowledge in the field?

__________Are research questions and research hypotheses well stated?

__________Are relevant theories described?

__________Is there background information about the problem?

__________Is the next step essential to research a problem identified?

__________Is the purpose of the study described based on previous research?

__________Is there a flows from one topic to another?

__________Are headings and subheadings helpful to readers in understanding the major points?

__________Is there a critical analysis of previous research (strengths vs weaknesses)?

__________Is the cited research current and appropriate?

__________Are primary sources mainly cited?

__________Are gaps in the literature identified?

Methods and subjects

__________Was random sampling used?

__________Was stratified random sampling used?

If random sampling wasn’t used:

__________Were subjects selected from the target group?

__________Were subjects from diverse sources included?

__________Were the limitations addressed?

__________Were the subjects well described?

__________Were demographics of the sample discussed?

__________Was an adequate sample size used?

__________Were the guidelines of informed consent followed?

Instruments

__________Were examples of test questions provided?

__________Was the item-response format (e.g., Likert, multiple-choice) specified?

__________Were the testing environment and testing limitations described?

__________Was the selection of the instruments justified?

__________Was information provided on how to obtain the instruments?

__________Was the evidence of instrument reliability and validity described?

Data collection procedures

__________Were subjects randomly assigned to groups?

If random assignment was not used:

__________Was evidence provided that showed the similarity in the groups?

__________Were the procedures in collecting the data well describe?

__________Was a natural setting provided for the experiment?

Results

__________Were the statistical procedures clearly described?

__________Were the appropriate statistical procedures used?

__________Were the results that were statistically significant described?

__________Was the statistical information described in relation to the research hypothesis and research question?

__________Were related statistics presented in a table with highlights discussed in the results?

Discussion

__________Were readers reminded of the study’s major purpose and results?

__________Was information provided about the significant results?

__________Were the interpretation of the results of the study described in detail?

__________Were recommendations provided for future research?

__________Were recommendations provided for practitioners?

__________Were limitations discussed in relation to the study?

__________Were the consistencies of the results from previous studies discussed?

__________Was the information gap from previous studies addressed?

Adapted from Pyrczak (2012) and Hittleman & Simon (2006)

The process of evaluating a research study consists of an in-depth assessment of each stage of the research process. The purpose of evaluating research is to emphasize both strengths and weaknesses. Some researchers are doubtful of their interpretations. These are normal concerns, which can be resolved by reading and discussing research reports. If they practice using the criteria to evaluate research reports, they can improve their critiquing skills (Coughian, Cronin, & Ryan, 2007). Many novice and inexperienced research are unable to understand the concepts and terminology related to research and research critique. When you think about it, quantitative research is like learning another language: it uses words (e.g., vocabulary, key concepts), has specific ways of structuring sentences (e.g., syntax or grammar), and is used to convey meaning (communication). Mastering the “language of science” is every bit as challenging as becoming fluent in a language other than your native tongue. Being able to critically analyze and read research advances a field by promoting evidence-based professional practice (Russell, 2005).