Earlier this spring, I worked with a wonderful faculty member to conduct research into a new hybrid version of an introductory Spanish course at our university. He changed some sections of a 4-credit course that typically meets four days each week to so that they only met two days each week with a substantial increase in online activity. I presented a paper on this research at the recent AIR conference with the basic questions: (a) Did students learn more or less in these hybrid sections? and (b) Did students who were more motivated or exhibited better study skills – measured using the Motivated Strategies for Learning Questionnaire (MSLQ) – learn more?
The full details are in the paper but it appears that the answers to our questions are:
- Students didn’t learn any more or less in the hybrid sections. This is consistent with the larger body of research that has found “no significant difference” between courses taught using different media. In fact, this is good news in some ways since we can implement more hybrid sections and courses with some confidence that student learning won’t be negatively impacted. This is particularly beneficial for us as these small four-credit courses require a lot of classroom space.
- The impact of self-regulated learning is unclear. Of the three outcome measures included in this study, performance on the MSLQ was only partially related to two outcomes. This is contrary to our expectations as it seems reasonable that students who are motivated and use better study skills would learn more.
To me, the most interesting part of this study is the role of age in predicting student learning. We created several multiple regression models and age was a negative predictor of student grades but a positive predictor of improved proficiency in reading Spanish. In other words, after we accounted for things such as race/ethnicity and gender, older students tended to earn lower grades but they also seemed to learn more about reading Spanish (but not about listening to Spanish). So older students have learned how to study more effectively and are more motivated to learn, right? No, at least not according to the MSLQ results: Age was not significantly correlated with the MSLQ results.
In addition to the quantitative measures used in this study, we also interviewed several students. At the same time, we also repeatedly interviewed students in some math courses that were also being modified – “flipped” – during the same semester. We were consistently impressed with the older students in our interview sessions and very much enjoyed their maturity and self-reflection. That suggests an interesting hypotheses: Were the older students in this study were simply less concerned with grades and more concerned about learning?
Despite my best intentions, I have become another MOOC dropout. Why am I not continuing to participate in this free course?
- The format isn’t compelling. The course is primarily built around four components: video lectures and notes, weekly quizzes, a discussion board, and two peer-graded assignments. The lectures are alright and although there are many other online R resources it’s nice to have concise descriptions of R procedures specifically linked to data analysis. The discussion board is also helpful but there are many other places to find help with R. As discussed in my previous post, the weekly quizzes are very disappointing as they are the primary means by which students in this course practice what they learn but they offer very, very little feedback.My biggest regret is that I won’t experience the peer-graded assignments. While the idea of requiring students to grade one another’s work is likely driven largely by the logistics of a MOOC, peer-graded assignments can be very powerful and worthwhile even in small classes. That these assignments are the only non-quiz activities in the course is disappointing especially since there are only two non-quiz assignments. Although it will be helpful that each student should receive feedback from several classmates (if it’s possible, I might provide feedback on the reports for some of my classmates even though I won’t be writing my own), it often takes more than two attempts for students to learn and begin to master new skills.
- Except for the peer-graded reports, there seems to be little reason for this course to be on a lockstep 8 week schedule. I might be able to stay with it if the timing were more flexible. Even in the first three weeks of the course I’m having some trouble consistently making time to view all of the videos. I had planned to do this all at work as my supervisor supports this as important and valuable professional development but I’m having trouble doing that because it’s sometimes difficult to carve out the time and I feel guilty watching online videos at work for a non-credit course when I feel like I should be doing something more (visibly and authentically) productive.
- I can’t convince myself to participate in the two peer-graded reports, the only meaningful assignments in this course. This is linked directly to the material of this specific course and is not a criticism of the course itself. I simply can’t muster the will to conduct additional data analysis and write additional reports for this course when those are two of my primary job duties. It’s not that I don’t think that I could learn from the activities, develop new skills, and become a better data analyst and writer. I just can’t bring myself to spend so much time analyzing data and writing reports unrelated to either my job or my research. I am disappointed as I was looking forward to these substantive activities, especially being able to receive feedback from others and seeing how others approached the same activities.
Although I’m disappointed to have decided to not continue with the activities of this MOOC, I am happy to have enrolled and tried it out. I will continue to download the course materials so I can reference them when I am ready to put them into practice in meaningful ways.
I have very mixed feelings about the broader concept of MOOCs. It would take an extraordinary effort for an online course, especially a MOOC, to match the quality of the best face-to-face courses. But the reality is that few face-to-face courses are “the best.” Although the dominant MOOC model seems to mimic much of the worst lecture courses in traditional universities, even the worst course is sometimes good enough especially when the alternative to a crappy, frustrating, and largely self-driven education is no education at all.
I have watched the online videos and successfully completed the quiz for week 2 of the data analysis MOOC in which I am enrolled. I struggled quite a bit with some of the R syntax and that made the quiz a very frustrating experience. I have two observations to share about what I learned this week about the format of the course.
First, I am disappointed that so far the only opportunities for students to practice what is being taught and receive feedback is the weekly quiz. I was able to muddle through things enough to get answers that matched the response options for this week’s multiple-choice quiz but despite answering all questions correctly I’m still very unsure of much of the content – I just know that I happened to somehow end up with answers that matched some of the ones included in the quiz. Some of this is simply due to my lack of experience with R and its high learning curve. But much of it is due to the fact that the multiple-choice quiz was the only opportunity to practice with any semblance of feedback and that feedback was restricted to an anemic “correct” or “incorrect” for each question with no additional feedback.
Yes, I can practice on my own some of the skills taught in this class. This is certainly the case if I want to focus solely on learning how to use R – syntax, configuration, functionality, etc. – as the language provides immediate feedback with error messages or output. But if that is the focus and if that’s sufficient to learn the skills then why do we need an organized course instead just a course packet or list of recommended self-guided topics and exercises?
What distinguishes an organized, well-taught class from a self-taught topic is that a class has an expert who not only make their thinking explicit but also offers targeted feedback for students as they practice the skills they are learning. It’s conceivable that some skills could be taught using sophisticated, automated tools if we have a deep enough understanding of how people typically learn those skills that we can programmatically recognize mistakes and misunderstandings to provide appropriate, specific feedback. Sometimes, this can be done to a (very) limited degree with appropriately designed multiple-choice instruments where the incorrect responses are designed to be diagnostic i.e., wrong answers aren’t merely incorrect but they’re designed to identify particular kinds of mistakes or misunderstandings. That seems to be the case for some of the questions and answers in this MOOC but we’re not provided with any of the related feedback to help us understand what common mistake we may have made, how we might be misunderstanding the issue, and how we can work to correct our thinking.
Second, the size of the course requires innovative ways to provide support for students and this course seems to rely heavily on the course discussion board. This is an observation, not a criticism. I’m quite comfortable using that medium as I’ve been using online discussion boards since the early 1990s when they were one of the primary draws for dial-up bulletin board systems (with the other major draw being online “door” games). I don’t know how well this works for other students, however, as I don’t want to make assumptions about their experiences, skills, and cultures. It’s probably not a big deal; my concern here is very minor and more of a curiosity about how other students experience and use (or don’t use) the discussion board. (In other situations I would be concerned about those who have poor or no Internet access or those who have little comfort and experience with the Internet but it’s reasonable to expect students who enroll in an online course to have sufficient Internet access and skills. I’m not suggesting that everyone has the access and skills to enroll in an online course, merely that those who are already enrolled in one presumably have the required access and skills.)
This semester, I have signed up for a data analysis class being taught in Coursera. This is a massively open online course (MOOC). I’m tech savvy and well educated but it seems like the most responsible way for me to really learn about MOOCs is to gain some firsthand experience. I also hope to learn some new data analysis techniques and ideas in this course. The course will use R to analyze data so it will also be good to expand my (very limited) skills and knowledge with that powerful tool.
Going into this, I am very skeptical about what I understand the typical MOOC model to be with instruction primarily occurring using pre-recorded videos and quizzes with a discussion board as the primary means of communication between students and faculty. I hope I’m wrong either about the model of instruction or about its effectiveness. As an educator, I believe (and am supported by significant evidence) that the best learning occurs when experts make their thinking explicit through demonstration and give learners multiple opportunities for focused practice and feedback. So my skepticism about the effectiveness of videos and quizzes as learning and teaching tools can best be summed up as: “Telling is not teaching.” (Note that this applies just as forcefully to passive lecturing in physical classrooms!)
I’ve just started to get into the material for this course and so far it looks like my low expectations are going to be met: the course is built heavily around pre-recorded videos as the way for the faculty to teach students with weekly online quizzes and two peer-graded assignments as the only opportunities for us to “practice” what we are “learning.” I hope I’m wrong and this proves to be much more enjoyable and rewarding that I think it will be!