Developing essay test items and accompanying scoring rubrics is an important component of educational assessment. Essay questions allow teachers to evaluate students’ higher-order thinking skills such as analysis, evaluation, and creation in a way that is not possible with multiple choice questions alone. Designing and scoring open-ended essay questions takes more effort than creating selected-response items. This article provides a step-by-step process for developing a high-quality essay test item complete with a detailed scoring rubric.
Step 1) Determine the purpose of the assessment
The first step in developing any test item is clarifying the purpose and intended use of the assessment results. Is the purpose of the test to evaluate students’ mastery of specific course content, or is it meant to measure overarching skills like critical thinking or written communication? Determining the purpose will help guide the creation of an appropriate essay prompt and rubric.
Step 2) Write the prompt
Craft a clear, focused prompt that invites students to thoughtfully respond using their own analysis and interpretations. Effective prompts are concise yet provide sufficient context and direction for students. Avoid overly broad questions that could lead to vague, disconnected responses. Instead, frame the prompt around a specific issue, text, data set, or scenario to anchor the essay. Use action verbs like “analyze,” “evaluate,” “argue,” or “compare” to signal what type of thinking is required. Consider including any context, sources of evidence, or parameters for the response. Pilot test initial drafts of prompts on sample students to spot any ambiguities or areas for improvement.
Step 3) Determine scoring criteria
Before writing can be assessed meaningfully, the elements of quality need to be defined. Develop a list of the knowledge, skills or qualities that exemplary responses should demonstrate. Common criteria might include: thesis/claim, organizational structure, quality and use of evidence, logic/reasoning, consideration of multiple perspectives, command of language conventions, and depth/complexity of thought. The criteria should directly connect back to the purpose and skills targeted by the assessment.
Step 4) Create the scoring rubric
With scoring criteria identified, draft a rubric template with qualitative “levels of achievement” descriptions for each criterion. Most rubrics contain 4-5 levels that range from novice to advanced (example levels: Beginning, Developing, Proficient, Exemplary). Within each criterion level, precisely define the characteristics of a response that would merit that score. Use concrete language so that multiple raters would assign the same score consistently. Pilot the draft rubric by having raters score sample responses and provide feedback on usability, clarity and reliability of scoring. Revise as needed.
Step 5) Train raters on using the rubric
Providing rigorous training on the scoring rubric is crucial for ensuring its reliable and fair application. Explain the rubric’s purpose, criteria, and level descriptors to raters. Model exemplar responses for each rubric level to clarify expectations. Then have raters individually score practice responses, compare results, and discuss any discrepancies to reach consensus. Raters should continue practicing with feedback until consistent and accurate scoring is achieved across the team. Periodic “calibration” sessions help maintain reliability over long grading periods.
Step 6) Administer the assessment
Provide students with all necessary materials, allocate sufficient time for completion based on complexity of prompt, and ensure test integrity. Remind examinees that responses will be evaluated using the published scoring criteria and rubric levels. Monitor the assessment session and collect student work.
Step 7) Score student responses
Each rater should independently assign a holistic score for each response by comparing it to the rubric level descriptors. Raters should not see any identifying student information to reduce potential bias. The team then convenes to cross-check a portion of scored responses, discuss any differences, and adjust scores as needed to resolve discrepancies and ensure agreement. Final scores reported should represent this consensus. Scoring large volumes of open-ended work takes significant time – schedule accordingly.
Step 8) Analyze results and refine
Once all responding have been scored, analyze assessment outcomes to evaluate how well the test measured the intended learning targets. Consider student performance patterns, rubric usage data, and rater feedback. Refine any aspects of the rubric, scoring process, or test content/administration as indicated to strengthen validity and fairness for future administrations. For high-stakes assessments, routinely monitor for reliability of scoring practices over time through ongoing improvement efforts.
Developing quality essay test items requires a thoughtful, rigorous process. Following these steps to craft a clear prompt, define appropriate scoring criteria, create a valid rubric, sufficiently train raters, and refine over time will yield more reliable and meaningful assessment data than an ad-hoc or poorly planned open-ended evaluation. Careful development of essay measures can provide essential insights into students’ higher-order thinking and written communication skills.
Andrew Stroud