Over the past two decades, academic deans, directors, and faculty have struggled to develop an effective peer review process for online courses. Most institutional review processes begin and end with the Quality Matters (QM) standard, the most widely accepted quality standard in the field of online education. But while QM provides an industry standard for evaluating online course design using peer review, it does not address other aspects of online courses, such as course delivery and how an instructor engages with students.
To continue reading, you must be a Academic Leader Subscriber.
Please log in or sign up for full access.
Over the past two decades, academic
deans, directors, and faculty have struggled to develop an effective peer
review process for online courses. Most institutional review processes begin
and end with the Quality Matters (QM) standard, the most widely accepted
quality standard in the field of online education. But while QM provides an
industry standard for evaluating online course design using peer review, it
does not address other aspects of online courses, such as course delivery and
how an instructor engages with students.
The comprehensive review process used in the Washburn University School of Nursing (SON) focuses on three domains: design, delivery, and content. The evaluation is intended to be formative rather than summative. Faculty obtain feedback related to each of the domain areas and use the feedback to improve their courses. Deans use an overall administrative process to coordinate the various reviews. The online peer review process is depicted in Figure 1.
Figure 1. Online peer review model
The design domain is based on Quality Matters standards and interaction design principles. The course design peer review consists of an instructional design consultation and a formal QM review.
Instructional design consultation In the new model, the faculty member works with the school’s instructional designer to ensure that the course meets QM standards and that from an interaction design perspective, the course is engaging, consistent, and easy to use and clearly communicates content and expectations to students.
QM review A formal external or internal QM review with three reviewers is completed. Each of the reviewers has received training to become a peer reviewer from Quality Matters. The final report from the QM review includes specific recommendations for each of the standards included in the QM rubric. The QM rubric is used to evaluate the course in eight areas:
Course overview and introduction
Learning objectives
Assessment and measurement
Instructional materials
Course activities and learner interaction
Course technology
Learner support
Accessibility and usability
The delivery domain focuses exclusively on how well instructors engage with students in the online environment. Each course delivery peer review is completed by a colleague who is given complete access to all areas of a course. The team that developed the model gleaned elements for this domain from a variety of sources. They compiled a list of desirable behaviors in the following areas:
Social presence and availability Social presence and availability is the degree to which students perceive that an instructor is accessible in an online course. It is the difference between an online course being perceived as a correspondence course verses a course resembling a traditional class experience. A strong online presence allows instructors to not only discuss class content and assignments but also to share their expertise, experience, and interests to build a community of learners. Communicating early and frequently with students can improve student retention in the course. Instructors can achieve a strong online presence by posting a picture of and introducing themselves, sending regular announcements or emails, providing introductions to the course and to each unit or module, being active in discussions, scheduling virtual class sessions, and providing feedback on assignments.
Instructor feedback Providing personalized, constructive feedback is critical to the success of an online course. In a face-to-face course, students receive verbal and nonverbal feedback from their instructor in the classroom. In an online course, feedback needs to be more explicit. One criterion in this area is providing personalized feedback to each student. It is often tempting to copy and paste one response to many students, but personalizing the feedback is more meaningful. Constructive feedback guides students on how to improve their performance.
Coaching Students in an online class have a variety of experience levels with the format. Some will have taken several online courses, while for others the course marks their first online experience. In a face-to-face course, students are either present or absent. Student participation in an online course is often more subtle. The instructor needs to monitor student participation to identify students who are struggling and should send personal emails to reach out to students who are missing assignments.
Privacy As with all classes, student privacy must be protected in online courses. For example, instructors need to be aware the replies to discussions posts can be seen by all students.
Class management Class management includes instructor communication regarding student feedback, deadline reminders, and expectations. Timely student feedback is important in general and crucial in an accelerated course. Rather than relying on lectures, many online courses incorporate a variety of activities to engage students. Without timely feedback, students won’t know how to adapt and improve on future assignments. The instructor can use email and announcements to remind the class of upcoming deadlines. Because students access the course at different times during the week, it is essential that the instructor post all materials related to a unit or module when they first make the module available to students. The instructor should also post expectations for email communication, feedback, discussion postings, submitted assignments, and grades to ensure that students understand them.
Virtual class sessions Incorporating virtual class sessions is a great way for instructors to engage with students. They not only increase the instructor’s social presence but also provide students with an alternative means of communicating with the instructor and each other. That said, they add another layer of complexity and a new set of best practices to follow, such as publishing dates well in advance, providing expectations for interaction, recording live sessions for students who were unable to attend, and posting the recordings.
Content
Collaborative faculty review brings together faculty to examine a course as a whole and within the context of how content from the course builds on previous courses or lays a foundation for future courses. Faculty meet to conduct a review of each course and to develop recommendations for improvement. They evaluate the course to determine whether it is as follows:
Complete Is the course content complete? Within the context of the course within the program, are the key topics covered? Does the course build on previous courses, and how does it prepare students for upcoming courses?
Accurate Is the course content free of errors and supported by strong empirical evidence?
Current Does the course content reflect the latest evidence and/or advances in the field?
Relevant Does the course content relate to the program’s expected outcomes and to the roles that students are being prepared to assume.
Grounded in real-world context Is the course content supported by examples that provide a real-world context for understanding course topics?
Appropriately rigorous Does the course challenge students at the appropriate level of rigor (e.g., for a graduate or undergraduate course)?
External expertise review recognizes that community partners and stakeholders can offer constructive insight during the evaluation process. The goal is to include community partners as part of the evaluation team and to involve stakeholders who practice in the role for which students are preparing or are knowledgeable about a specific topic. Community partners and stakeholders include educators from sister institutions, peers from community and professional organizations, and advanced practice nurses. External stakeholders are compensated for reviewing a section in a course matched to their area of expertise. Similarly, the content expert evaluates a selected aspect of the course to determine whether it meets the above evaluation criteria.
The administrative process supports the following:
Budgeting for online peer reviews A budget must be maintained for online peer reviews. There are costs for submitting a course for formal QM review and for contracting with an external expert. Currently, these costs for the SON are $950 per course.
Identification of faculty, courses, and schedules for reviews The dean identifies which faculty and courses will be reviewed each academic year. Tenure-track faculty who teach in online programs are given priority. Each summer, the dean informs those faculty who have been selected for review when specific courses will be reviewed that upcoming academic year.
Faculty implementation of changes Documentation of the online peer review is included in a faculty member’s employment file.
Impact occurs when faculty incorporate feedback
generated by the review process into the courses they teach. Faculty
reflect on the recommendations they receive and consider ways to improve the
design, delivery, and content of their courses. Faculty use the feedback to
identify specific goals, action items, or plans to improve the course. The
administrative team may assign a coach or mentor. After a period of one year, the
faculty member provides a summary of changes they made as a result of the peer
review process to the SON’s instructional designer, associate dean, and dean.
Incorporating an online peer review
process is a valuable strategy for ensuring an online program’s continued
success. Reviewing an online course and instructor in the different domains of
design, delivery, and content from the perspective of a diverse group of people
can be insightful and transformative. The thoughtful and detailed feedback
gleaned from external reviewers has been exceptional and has given SON faculty
a deeper, more nuanced view of how to improve their courses, while the adoption
of regular collaborative faculty reviews has expanded the focus of curriculum
reviews from individual courses to include mapping and threading key themes
throughout the program. Most importantly, it has allowed faculty who teach
exclusively online to collaborate with and learning from their peers.
Linda
Merillat, PhD, is a lecturer and instructional designer, Bobbe Mansfield, DNP,
is an associate dean, Marian Jamison, PhD, is a professor, Monica Scheibmeir,
PhD, is a retired dean, and Jeanne Catanzaro, EdD, is an assistant professor
and director of the RN-to-BSN program at Washburn University.
