Mistaken Goal: Where Student Affairs & Technology Meet

On Friday, a colleague pointed out a new article on Mashable that is titled “Why Tablet Publishing Is Poised To Revolutionize Higher Education.” I don’t trust the claims made in this article. I’m going to explain why I don’t trust the claims, not to convince you that my opinion is correct but to give you an understanding of how I evaluate claims like the ones made in the article. I’ll lay out my thoughts in chronological order.

1. The article is published at Mashable. I removed Mashable from my RSS reader over a year ago because I got tired of their poorly-written articles that make ridiculously overwrought and unprovable claims. This certainly isn’t enough for me to condemn this particular article but it certainly makes me cautious right from the beginning.
2. The title makes a very bold claim. Many people have attempted to “revolutionize” education; few have succeeded. And even fewer have been able to explicitly predict revolutions before they occur or even recognize them as they are occurring. The author has a helluva case to make and he better bring remarkable evidence to support his claim(s).
3. After reading the title, a quick glance through the article indicates that it’s a utopian piece largely based on the idea of technological determinism. In other words, it’s not only wildly optimistic but it also relies on the idea that we can predict and control how people use technologies by the way in which those technologies are designed. Both of these ideas – utopia and technological determinism – have a bit of history in the field of social informatics. The history is mostly negative; these ideas simply don’t work most of the time. So my skepticism continues to increase.
4. The author of the article is an executive at Adobe. In fact, he’s the “director of worldwide education.” That doesn’t mean that his opinions are necessarily biased but it’s another reason for me to be skeptical.
5. The article claims that “[There are] better study habits and performance with tablets.” Only one study is cited to support these sweeping claims: a Pearson Foundation “Survey on Student and Tablets.” For example, the author states that “86% of college students who own a tablet say the device helps them study more efficiently, and 76% report that tablets help them perform better in their classes” and a few other claims. Even if this study were flawless, the author needs a whole lot more evidence to support such a broad claim.
1. To their credit, Pearson offers to share methodological details about and data from their survey if you just ask them; I haven’t asked so I don’t have any more detail than what is provided in that 2-page overview. But we do know that the survey was conducted online. Given that about 20% of people in the U.S. do not have access to the Internet (Dept of Education estimates 18.6% and the Pew Internet & American Life Project estimates 21%), it seems unlikely that an online survey can produce data that is representative of the entire population. It seems particularly problematic to omit non-Internet users when asking about technology since the results will almost certainly be skewed.
2. Even if we accept that the Pearson numbers are accurate or in the right ballpark, I’m still not sure if they’re very informative. I guess it’s interesting that many young people think that tablets will help them study more efficiently and that they will replace textbooks in the next five years. I just don’t think that we can use these data to make any predictions.
3. Let’s ignore the validity issues for some of Pearson’s data (e.g. people are notoriously bad at distinguishing between “what I like” and “what is most efficient/effective) so we can move on.
6. The authors correctly assert that digital textbooks can include more features than printed textbooks, including “video, audio, animation, interactive simulations and even 360-degree rotations and panoramas.” However, the author does not say how we’ll produce all of that additional material. I don’t expect the author to solve every challenge associated with his predicted revolution but it would be nice to at least acknowledge them instead of glossing them over or ignoring them entirely.
7. In the next section of the article, the author claims that “interactive learning leads to better retention.” The only evidence cited is this news article about a study of elementary and high school students using 3D technology in science and math classes. Of course, since I’m an academic snob I think it would be much better to cite a primary source, preferably one that has been peer-reviewed, than to rely on a popular press article. Once again, even if we accept that this study is perfect it’s not even close to being enough to support such a broad claim.
8. Next, the author claims that digital publishing can help us better “[understand] learning effectiveness” using “integrated analytical tools.” I have no issue with this claim as a broad theoretical claim. But it seems to completely bypass the fact that U.S. higher education is in complete disarray in terms of even settling on broad learning objectives much less specific objectives and associated assessment tools or indicators. (Look into the “tuning project,” especially the “Tuning USA” project, to get an accurate view of these issues.)
9. The next claim the author makes is that “digital publishing makes knowledge more accessible.”
1. The author must be using “accessible” in a different way than I commonly use it because it’s hard to take that claim seriously given the (a) lingering digital divide, participation gap, and similar inequities in the U.S. and (b) the immense resistance many digital publishers have exhibited to making their content accessible to the visually impaired.
2. Once again, the author focuses solely on a possibility offered by the technology without giving any thought to the cultures in which the technology is embedded. He writes that “digital publishing allows professors or subject matter experts to self-publish their own educational materials or research findings and distribute the information on tablet devices” without offering even the barest hint about how this will occur without adjusting or overturning the systems that would need to support this. In other words, why would faculty do this? What is the incentive?
3. Similarly, the author claims that “by harnessing interactive technologies, educators can explain even the most complex scholarly or scientific concepts in compelling and intelligible ways.” Once again, I accept this broad claim (ignoring the “even most complex” qualifier because it’s just silly) in theory but balk at it in practice. It takes complex skills to create effective interactive content, skills that are different from those possessed and valued by faculty in many disciplines.
10. At this point I’m just tired of reading these grand claims supported by flimsy or no evidence…

I’m not a Debbie Downer or a Luddite. I agree with the broad proposition that digital publishing has potential to make a huge impact on U.S. higher education. And I agree that tablets are super cool and very useful in some circumstances; I purchased an ASUS Transformer a few months ago to replace an ailing netbook and I’m very happy with my purchase! Fundamentally, I distrust the claims made in this article because the author fails to support them. Even when the author provides cherry-picked examples and studies, they are often of poor quality and always insufficient to support those claims. This is quite disappointing since the author could have easily drawn upon the large and rapidly-growing body of evidence in this area. I expect very little from an article published by Mashable and this article delivered.

Comments(0)

I have been extremely critical of NASPA’s disenfranchisement of graduate student members, especially since that effectively negated the membership’s desire to merge with ACPA. So I was very happy to receive the following message in an e-mail from NASPA:

After a month-long voting period, the NASPA voting delegates overwhelmingly approved a proposal to revise the voting structure of the association to allow associate affiliates, graduate student affiliates, and emeritus affiliates the opportunity to vote in elections for the chair of the NASPA Board of Directors (previously NASPA President) and Regional Directors (previously Regional Vice Presidents).

“As a result of member feedback, the Board of Directors voted unanimously in May to submit this Bylaw Amendment to NASPA’s Voting Delegates,” said NASPA President Patricia Telles-Irvin. “I feel strongly that this was the right thing to do at this point in time, and I am so gratified that the Voting Delegates agreed and voted so overwhelmingly in favor of the change.”

“Graduate students, in particular, have been increasingly active within NASPA and have been its fastest growing membership type over the past year,” said NASPA Executive Director Gwendolyn Jordan Dungy. “I am particularly pleased to see the governance structure adapted to better recognize the contributions our members along the full spectrum of the student affairs career trajectory.”

The expanded voting rights will go into effect immediately with January’s ballots.

That this was necessary and that the organization denied full voting rights for over a quarter of its membership will remain stains on NASPA’s history. But it’s wonderful that the voting delegates have voted to remedy this injustice as we move forward. Well done, NASPA!

Comments(1)

There are many different angles one could take in reporting on the 2011 NSSE Annual Results; it’s a dense 50-page report. I know that every group has its own agenda and every reporter has his or her own personal interests but it’s very disappointing that CBS News chose the snide headline “Business majors: College’s worst slackers?” for their article. In an ordered list, something must be last. In this case, some major must rank last in the number of hours students typically study each week. But to label that group of students “slackers” simply because they fall at the bottom of the list is unnecessarily mean and unprofessional.

Comments(0)

The 2011 NSSE Annual Results were released today. I don’t want to focus on the content of the report in this blog post. Instead, I am briefly noting how fun it is to work on a project with a large impact that regularly receives attention from the press (even if some of the attention is sometimes negative, a very interesting experience itself). It’s gotten more fun each year as I’ve become more involved in much of what we do; this year I directly contributed by writing part of the report itself. Yes, it’s ego-boosting to see my work in print but more importantly it helps address a very serious and difficult problem that vexes many researchers and administrators in higher education: It’s hard to explain to others, especially our parents and extended families, what we do. Instead of trying to convince them that I really have graduated (several times!) and am not wasting my whole life in college, I can send them the report and articles from the New York Times and USA Today and say, “Look – this is what I do!”

Now I get to watch media reports and subsequent discussions to see how they play out and what they will emphasize. This process is unpredictable and it has surprised me in previous years when relatively small bits of information have caught on to the exclusion of other interesting and important information. As The Chronicle of Higher Education notes, this year may be a bit different given recent events but who knows how things will play out.

Comments(0)

I’m buried in work and research but I have two thoughts dancing on my mind and they’re both related to online community:

• I hate when websites or tools list reader comments in reverse chronological order i.e. newest messages first. I finally figured out why I hate that: It makes it very difficult to view the messages as a coherent discussion within a pre-existing social context. Because new participants are not immersed in the context of the ongoing discussion they can easily view the opportunity to comment merely as a way to shout messages without any responsibility to engage with or form a community. Mediated communication is difficult enough without us actively encouraging antisocial behaviors and views.
• Our obsession with tools and technologies leads us to underestimate or ignore the social effects and communities that build up around them. I see this happen all of the time in Wikipedia when new editors leap into articles without having any understanding of the cultural norms of the immense community of users that have used Wikipedia for years. It’s sadly naive to believe that such an immense collection of resources doesn’t have a correspondingly large and complex community with cultural and social norms and expectations.

Comments(0)

Results from three research studies were released late last week. Two of them come from EDUCAUSE; I’m going to their annual conference this week and I’m really looking forward to attending presentations related to these studies.

• EDUCAUSE – or more accurately their research arm ECAR – released results from the 2011 ECAR National Study of Undergraduate Students and Information Technology. As always, the researchers at ECAR have done a great job summarizing the results and making actionable recommendations for colleges and universities. The survey is undergoing changes and two different versions were administered this year. Although the methodological details in the report are not as detailed as I would like, it seems that moving to a third party administration has addressed some of my consistent concerns about non-response bias in this survey and generalizability of its results. EDUCAUSE also commissioned an infographic to summarize some of the results.
• EDUCAUSE also released some data from the Core Data Service (CDS), their annual survey of member institutions. They have not yet released the summary report but they have released other reports including new “almanacs” that summarize data for large aggregations of Carnegie Classifications. The CDS has also been redesigned and several of these reports are also new. However, I am puzzled that they continue to use the outdated 2000 Carnegie Classifications. Not only the actual categories outdated and no longer used, the data on which they are based are well over a decade old.
• Statistics Canada, a government agency roughly analogous to the U.S. Census Bureau, released results from the 2010 Canadian Internet Use Survey. Comparative data are often useful and interesting to me, especially data from Canada and other countries culturally and economically similar to the United States. Unfortunately, only a few summary tables are available; you have to pay for other data. Hopefully Canadians can access these data for free and I am only being quoted a price because I am connecting to the Statistics Canada website from a U.S. IP address.

Comments(0)

The EFF, one of my favorite organizations, has announced a report describing a security vulnerability in Impulse Point’s SafeConnect product. I don’t have any new insight to add regarding the security flaw or SafeConnect. But the announcement is a quick read with a nice little history of Network Access Control (NAC) technology and its important role in managing residential computer networks.

(Off-topic reminiscing: In 2003, college and university campuses experienced massive problems on their student computer networks thanks to the Blaster and SoBig worms. In response, colleges and universities rapidly adopted NAC and similar technologies to curtail those problems. Around that time, a few people from a brand new company visited the campus where I worked to pitch their product; the company was located in Florida and they were visiting nearby colleges and universities to collect feedback and gauge interest. They had a nice product but it didn’t address our needs. If I remember correctly, the product hijacked downloads of copyrighted material – music, movies, etc. – and redirected students to vendors selling the material legally. Again, it was a neat product but one in which we had no interest. Instead, we told them how badly we needed a good NAC, especially after getting our asses kicked by Blaster and SoBig so badly that we shut down the network for several days until we could get a handle on things. Importantly, their product shared a lot with NAC products so our recommendation to develop a NAC was realistic. The nice people from Impulse Point left and when I next heard of them it was about the success of their SafeConnect NAC product. Maybe my memory is faulty or maybe I’m just silly and arrogant but I like to think that I played a teeny tiny role in the success of this company and their popular product. You’re welcome!)

Comments(0)

Right after I posted a screed about how some recent research about Twitter’s relationship with students’ grades has been misunderstood, along came another study that is being mischaracterized. I’m not looking for these things; I don’t want to be some kind of education research watchdog or bully. But this is important and I must speak up.

The demographics of U.S. college students are changing and too many of us are not changing our practices to match. Recently, Complete College America released a report focusing on these changing demographics with a specific focus on part-time students and the continued growth of non-traditional students. It focuses on some very important and often overlooked topics and it should elicit discussion and promote action.

Frustratingly, several of the media reports are misreporting what is in this study, particularly in their headlines and summaries. The study very explicitly says that “4 of every 10 public college students are able to attend only part-time” on its second page. So why are some reporters and commentators summarizing the report with headlines proclaiming that part-time students are the new majority? I can understand a relatively small shop making this mistake, particularly if they’re in a rush to try to get the word out about this important study and happy to make corrections. But why is the Washington Post getting it wrong and letting the error persist for days? And why are higher education professionals passing along this report with incorrect information, blindly repeating headlines and summaries that get it wrong?

(Not everyone is getting this wrong. For example, The Atlantic gets it just right.)

This is so frustrating to me because the topics discussed in this report are so important. Non-traditional students do make up the majority of students. The federal government does a poor job collecting information about these students by often focusing exclusively on first-time, first year students (which, coincidentally, was an issue I wrote about in my qualifying exam). Too many of us have tunnel vision and only focus on the students on our campus or – more accurately – the students we think are on our campus. In the context of student affairs, I worry particularly about the next generation of professionals and whether these demographic changes are being addressed in their coursework. My impression is that they are not; I hope I am wrong!

Comments(0)

I’ve watched with interest over the last several months as media outlets and individuals have discussed, blogged, and tweeted a study conducted by Junco, Heiberger, and Loken. Their study reported that a group of students who used Twitter as part of a class earned higher grades than classmates in sections of the class that did not use Twitter. It’s a nice study that is clearly described and methodologically sound. Like all studies, it has significant limitations and they are concisely and honestly discussed in the study but those limitations have been ignored by too many people who have made the study into something it’s not.

The study concluded that “Twitter can be used to engage students in ways that are important for their academic and psychosocial development” (p. 10). But is that what has been reported and discussed by others? No, of course not; if it were then I wouldn’t be writing this sanctimonious blog post! Mashable, a very widely-read and influential technology blog, reported on the study using the headline “Twitter Increases Student Engagement [STUDY].” A recently-created infographic proclaims that “Students in classes that use Twitter to increase engagement have been found to average .5 grade points higher than those in normal classes.” Another infographic proclaims that “[Students get] grades up half a gradepoint in classes that use Twitter.”

I get that pithy headlines and concise summaries are necessary to grab attention. But by overlooking or ignoring the details of this study, those headlines and summaries get this all wrong. Let’s return to the original study to understand why.

In the study, the researchers assigned some sections of a class to use Twitter. While the entire class used Ning, these sections also used Twitter to complete some received additional assignments. They also received guidance and encouragement to use Twitter to communicate not only with one another but also with instructors. At the end of the semester, these students had earned higher grades than their non-Twittering classmates.

If I understand the study’s methodology (Rey, please correct me if I got anything wrong!), it seems that this study does not show that “Twitter improves grades.” It shows us that students who do more work and spend more time concentrating on class materials can earn higher grades. It shows us that students who have additional opportunities to communicate and collaborate with one can another earn higher grades. It also shows us that students who have greater access to instructors can earn higher grades. It shows us that Twitter can be a viable medium for students to communicate and coordinate with one another and instructors. And, yes, it shows that Twitter can be an effective educational tool when skillfully incorporated into a class with appropriate support and structure. In a critique of one of the infographics, Junco specifically mentions this: “Yes, that’s our study about Twitter and grades. Unfortunately, what’s missing is that we used Twitter in specific, educationally-relevant ways—in other words, examining what students are doing on the platform is more important than a binary user/nonuser variable.”

This illustrates the challenge with testing the efficacy of educational tools and techniques: It’s really, really hard to isolate just the impact of the tool or technique. To test the tool or technique, you almost always have to make other changes and it’s usually impossible to tell if those changes changed the results of your study more than the tool or technique you intended to study. It’s a limitation of nearly every study focusing on the effect of particular media on education and it may be an inherent limitation for this kind of work. (Richard Clark has been pointing this out for decades; look into his writings for more detailed discussions. He’s also been wonderful in creating dialog with his detractors so there are well-documented and substantive discussions between many different scholars with different opinions.)

Hence my frustration with how this study has been summarized and passed around: By ignoring the limitations and nuance of this study, these summaries miss the boat and draw a grandiose conclusion that the authors of the study never attempt to draw themselves. That’s a shame because this is a nice study that is interesting and informative. But like most research, it’s a small step forward and not a giant, earthshaking leap. Summarizing this study by proclaiming that Twitter is a magic ingredient that can be added to classes to increase grades is irresponsible and misleading.

Update 1: Thanks for the clarification about Ning, Liz!

Update 2: Another example of how headlines can distort or misrepresent research has just popped up. Before correcting the headline, Colorlines reported that the majority of college students are part-time students (full headline before being corrected: “Study: Majority of College Students are Part-Timers, Less Likely to Graduate”) But the actual report doesn’t say that. Instead, it says that “4 of every 10 public college students are able to attend only part-time” (p. 2). It’s a shame that the research was initially being reported incorrectly because the changing demographics of college students is incredibly important and very misunderstood and overlooked. I know there is a lot nuance in discussions of demographics – race, ethnicity, SES status, privilege, etc. – but if we cover up or ignore the details then we haven’t made any progress.

To their credit, Colorlines corrected their headline once I pointed this out to them. They made a mistake in their initial headline and it’s great they they’re willing to correct their public mistake!

Comments(6)