Tomorrow's Teaching and Learning
The posting below summarizes the 2018 Professional Organizational Development (POD) Innovation Award Winning project, from the University of California-Merced. The study is the result of a collaboration between the Students Assessing Teaching and Learning (SATAL) Program and the Merritt Writing Program (MWP). This post offers a free online asynchronous video that can be implemented in courses before mid and final SETs. It is by Adriana Signorini, SATAL Coordinator and Mariana Abuan, MWP Lecturer.
UP NEXT: Faculty Development Matters
Tomorrow’s Teaching and Learning
---------- 1,434, words ----------
Students Helping Students Provide Valuable Feedback on Course Evaluations
Students Assessing Teaching and Learning (SATAL) is an assessment service offered by the Center for Engaged Teaching and Learning (CETL) at the University of California, Merced. Following the Students as Partner (SaP) model (Cook-Sather, Bovill, and Felten 2014), faculty collaborate with undergraduate students hired as SATAL interns to assess and enhance teaching and learning. The aim is to gain a better sense of the learning experiences of students in specific courses and/or programs by amplifying the voice and perspectives of students. To this end, SATAL interns receive professional development on protocols for gathering and analyzing different kinds of classroom data. Various methods such as Small Group Instructional Diagnosis (SGID), Classroom Observation Protocol for Undergraduate STEM (COPUS), focus groups, interviews, and surveys are utilized to help instructors receive honest and actionable feedback from students. The “Students Helping Students Provide Valuable Feedback on Course Evaluations” project, a collaboration between the SATAL program and the institution’s Merritt Writing Program, seeks to investigate whether the quality of responses to the open-ended questions on the Student Evaluation of Teachings (SETs) improves when the purpose and importance of SETs, as well as distinctions between “useful” and “not useful” feedback, is explained to students by their peers. SATAL embodies the mindset that student-faculty partnerships are built on mutual respect and responsibility and offer a reciprocal process for meaningful collaboration (Cook-Sather, 2014).
According to Bloom’s Taxonomy, the ability to evaluate is a higher-order skill, which most undergraduates are still developing.However, we often ask students to evaluate instruction without teaching them how or explaining why it is important. For this reason, student responses to the open-ended questions on SETs often do not address course outcomes or do not answer the questions completely. Some studies have shown that students are not the best judges of their own learning, which results in feedback that does not address the questions on evaluations clearly (Weiman 2015). This suggests thatfaculty need to provide students with instruction and practice in leaving actionable feedback.
However, peers are the most potent source of influence on students’ development during their college years (Astin 1993), and engaging in student partnerships is resulting in a variety of beneficial outcomes for students, staff, and faculty, including the field of assessment. Students have been included in pedagogical planning as “co-creators” of teaching approaches, course design, and curricula (Bovill, Cook-Sather, and Felten 2011) as well as pedagogical consultants (Cook-Sather, Bovill, and Felten 2014), and more and more staff and faculty have been engaging in “Students as Partners” (SaP), which differs from just getting the student perspective on pedagogical practices because “ all participants have the opportunity to contribute equally, although not necessarily in the same ways, to curricular or pedagogical conceptualization, decision-making, implementation, investigation, or analysis” (Cook-Sather et al., 2014; Healey Healey, Flint, and Harrington 2014).
Description and Goals
This Professional Organizational Development (POD) grant funded study addressed the need to elicit more actionable student responses to the open-ended questions on SETs by partnering with undergraduate interns and creating a peer-led presentation on the importance of SETs for students captured in a brief video.
The focus of this project was the open-ended questions from the official course evaluation currently used by the MWP. Six instructors volunteered 18 courses and 323 students participated in this study. Student interns developed a thirty-minute presentation that they delivered in these 18 classroomswith the goal of engaging undergraduates in the SET process by explaining the importance of completing SETs and providing a feedback rubric with criteria for actionable responses.
Results and Scope
Live Presentation Results
SATAL interns delivered the peer-led presentation in 11 classrooms to 323 students. 29 courses and 529 SET responses were provided and analyzed with the help of a rubric by 4 participating faculty. SETs from 18 courses from previous semesters were used as baseline and compared to the SET responses from the 11 courses that experienced the peer-led presentation. Students completed a post-presentation survey and faculty assessed the usefulness of SET responses they received from all 29 courses.
Upper division (UD) students provided more useful responses to all open-ended questions in mid- and end of the semester SETs (p< .001, V= .315), and they derived sizeable benefit from the presentation while lower division (LD) students’ responses did not significantly improve. Students found the presentation helpful and reported that their ability to give feedback improved significantly. 92% of students found the feedback rubric useful, and 83% of students recommended the presentation be delivered in more classes. Participating faculty reported an increase in the amount of actionable feedback from students compared to previous semesters, identified concrete ways to enhance their courses, and recommend this presentation to colleagues.
Video Pilot Results
Six faculty volunteered their classes for the video pilot at mid semester. The seven-minute video was presented by SATAL interns in 14 courses to 203 students. Students completed a post-video survey and faculty were asked to reflect on the usefulness of SET responses they received. This pilot is still ongoing and will continue into spring 2019.
74% of students found the video highly effective in helping them understand how to provide course feedback. 87% of students found the accompanying feedback rubric useful. 79% of students mentioned that they recommend the video presentation be shared in other courses, and 55% of students noted that they preferred the information be presented by peers rather than faculty, when asked if they would prefer students.
Four classes (81 students) were showed the 7-minute video at mid semester and then the 5-minute video right before completing the final SETs. These students were asked, Do you think that watching the video for a second time in the semester helped you understand how to leave useful feedback for your instructor better than seeing it once? 63% responded yes. Several students mentioned that it is a good refresher for themselves and their peers. 57% of students recommended that the instructor show the video twice during the semester rather than once. Some students reported that they retain information better when they hear it more than once.
57% of students noted that they would not watch the video on their own if the instructor assigned it before completing course evaluations. Some students explained the reason for this is that they are lazy and forgetful.
Did faculty receive more actionable feedback?
Participating faculty were asked to reflect on how student feedback to the open-ended questions has changed (if at all) and whether or not they would recommend the presentation to other faculty.
“Students were able to articulate their ideas in concrete terms. I learned both the what and why of their feedback. With it, I was able to clearly identify aspects of my class that were working, and precisely those that didn't. In addition, students going through the peer-to-peer presentation wrote much longer comments than other classes I've had in the past. Their comments had meaning, and I liked this aspect of the presentation: it showed students how to become more engaged with their classes in terms of assessment, and it showed them how to meet the level of responsibility of their feedback. “
“... There were fewer ‘no comment’ or statements like ‘The course should stay the same.’ Generally, the students appeared less hesitant about giving responses and as a result, took their time when writing their responses and seemed to know the importance of specific feedback, so the instructor could create more student-centered courses and class activities.”
“I used the peer to peer presentations before both midterm and for final evaluations in lower division writing classes. ... the students did indicate that some of the instructions for some of the activities were unclear. Thus, for the following semester, I rewrote the activities in a more concise way and orderly step-by-step organization, not making any assumptions about prior knowledge the student possessed about the topic under discussion.”
“I would recommend the student presentation. I feel I received more feedback focused on items of importance in regards to content and instruction. ... I decided to allow/change my late policy to be more flexible.”
These seven, five, and three-minute videos are available resources for instructors teaching large or small courses, online or onsite. Videos can be used as a standalone assessment activity and/or to collect mid-course or end of the term feedback. We recommend faculty implement them twice: [E1] [r2] [r3] In the middle and at the end of the course before SETs for optimal results.
We suggest the following three steps when showing the video:
1 Rubric: Complete the rubric with the course learning outcomes and share the rubric with the class.
2 Video: Show the video to the class: https://cetl.ucmerced.edu/SATAL_Video
3 Assessment: Invite students to complete a mid-/final course assessment. This video can be implemented before administering the SETs and / or a customized assessment tool. Once you collect the data, reflect and respond to students’ comments.
Astin, A. (1993). What matters in college? Liberal Education, 79(4), 12.
Bovil, C., Cook-Sather, A., and Felten, P. (2011) Students as co-creators of teaching approaches, course design, and curricula: implications for academic developers. International Journal for Academic Development. 16(2), 133-145.
Cook-Sather, A. Bovill, C. and Felten P. (2014) Engaging students as partners in learning and teaching. A guide for faculty.San Francisco: Jossey-Bass.
Healey M., Flint, A. and Harrington K., (2014) Engagement through partnership: Students as partners in learning and teaching in higher education. York: HE Academy.
Wieman, C. (2015) A better way to evaluate undergraduate teaching,[E6]
Change: The Magazine of Higher Learning, 47:1,6-15.
*Adriana Signorini coordinates the Students Assessing Teaching and Learning (SATAL) Program at the Center for Engaged Teaching and Learning at the University of California, Merced, USA. She facilitates the student-faculty partnerships and her research interests focuses on collaborative approaches to teaching and learning in higher education.
Mariana Abuan is a lecturer with continuing appointment in the Merritt Writer Program at the University of California, Merced, USA, whose research interest includes assessment of teaching and learning in higher education.