Mistaken Goal: Where Higher Education & Technology Meet

Laura Pasquini and I are working on analyzing #sachat data, a follow-up to work I’ve done previously but did not formally publish. Part of our work involves looking at a few other student affairs-related hashtags to help us understand #sachat in context. This figure shows the number of Twitter messages posted with particular hashtags – #highered, #sachat, #sadoc, #sagrad, #sajobs, and #studentaffairs – during the week of June 27, 2011. The #sachat session really stands out here both in the number of messages posted and in how it interrupts an otherwise regular daily and weekly pattern. This isn’t a profound discovery but it’s an easy way of illustrating that #sachat sessions are relatively unique and prominent uses of Twitter among some users.

Comments(6)

It took me much longer than I had hoped but today I finally finished a first draft of a visualization of some of the #sachat data I’ve been working with:

The method of analysis used in this video is dynamic topic analysis (DTA). DTA was developed by Dr. Susan Herring and more information about the method can be found in one of her papers published in 2003. This method of visualizing DTA data was created by Andrew Kurtz and Dr. Susan Herring. Andrew’s original Java tool isn’t working for me any more so I developed the graphs in this video using Excel macros to generate R scripts (which has been an adventure because this is the first time I’ve used R).

The music in this video is Tutto L’Amor Perduto by Giorgio Costantini. It is available under a Creative Common license at BeatPick.com.

I created this and publicly posted it for several reasons. First, this really is a first draft and I would love feedback so I can improve it. I’ve already noticed a few small mistakes that will be corrected in the next version. I’ve also received some feedback and suggestions for possible improvements. If you have some, please let me know!

Second, I hope that some in the #sachat community find even this very rough first draft interesting, informative, and possibly even useful. Even though I’m comfortable studying a group that is so very public with their actions and membership, I still believe that I should give back to that community in ways that are appropriate and helpful. It just seems like a nice thing to do and it’s a small way of showing my appreciation to them.

Finally, I’m interested in seeing if there is interest in helping me continue this kind of work. One of the reasons why this is only a rough draft is that I’m the only one who has analyzed these data. DTA is a specialized form of content analysis and, like any content analysis, it should be performed by multiple persons to ensure the codes are being applied consistently (which is why good content analysis studies report interrater reliability figures to help bolster the credibility of the findings). This analysis – and it should hold up well even when other coders are added – shows that this particular use of Twitter is moderator-led discussion with coherent threads of discussion. I need to analyze a few other #sachat sessions to ensure this is consistent for other sessions. I also need to analyze some other Twitter data so I have some useful points of comparison.

I think this use of Twitter is fairly unusual and it would be great to be able to publicly discuss that with confidence. This is a wonderful example of a group of people using a very limited tool to do very good things that transcend (my) expectations and it should be represented in the research literature.

Comments(4)

The editors of Oracle: The Research Journal of the Association of Fraternity and Sorority Advisors are putting together an issue dedicated to “empirical research on technology.” Examples of such research may include:

• Technology’s effect on fraternity/sorority recruitment
• Studies regarding the ways alumni(ae) connect online
• Relationship of technology use and fraternity/sorority involvement
• Impact of email/twitter/facebook and other social networks for Greek organizations

Kim Nehls, Executive Director of ASHE and Visiting Assistant Professor at UNLV, is the guest editor of this issue. Please contact her at kim.nehls@unlv.edu if you’d like to contribute to this issue or have questions.

Comments(2)

A colleague and I are presenting a paper at ASHE in a few months discussing the content of Wikipedia college and university articles.  The most common comment the reviewers made of our paper proposal was that we did not quite answer the “So what?” question.  In other words, we didn’t quite convince them that our topic is important and interesting.  Part of the answer lies in convincing you that U.S. college and university Wikipedia articles are (a) very common and (b) very popular.

First, let’s see how common U.S. college and university Wikipedia articles are.  To do this, I need to figure out how many institutions have a Wikipedia article.  I randomly selected 10% (732 units) of the 2008 IPEDS universe, a listing of every Title-IV-participating institution (e.g. virtually every accredited institution in the United States and its territories).  I then checked to see if these units have Wikipedia articles.  Broken down by sector and control and ignoring the handful of system offices and unclassified institutions pulled into the sample, here is what I found:

Considering that most people in the U.S. think of 4-year institutions when they think of “college” or “university,” Table 1 shows us that it’s fair to say that college and university Wikipedia articles are very common.  Not only are they ubiquitous for public 4-year institutions, they’re very common for private 4-year institutions and community colleges.  The primary types of institutions for which they are uncommon are private 2-year institutions and all types of less than 2-year institutions, institutions typically associated with specialized technical training and usually omitted when talking about colleges and universities.

Next, we need to figure out the popularity of U.S. college and university Wikipedia articles.  In this context, I am defining “popular” by examining where the top three search engines – Google, Yahoo!, and Bing – place U.S. college and university Wikipedia articles.  To do this, I selected a random sample of these Wikipedia articles; the sample is also stratified, including 12 articles from each major quality classification assigned by the Wikiproject Universities (Featured, Good/A, B, C, Start, and Stub).

As shown in Table 2, when you search for these institutions in each of the three leading search engines, Wikipedia articles are not only among the very first results but they’re usually the first result that isn’t controlled by the institution.  Google seemed to struggle with providing accurate results for the institutions who do not have unique names (i.e. Southwestern College, Sierra College), listing several other similarly-named institutions above the Wikipedia article.  Yahoo! and Bing did not have this problem, almost always listing the Wikipedia article immediately after the institution’s official website or immediately after the institution’s official website and the official athletics website (of course, Yahoo! and Bing provided the same results since they use the same search technology).

Based on a random sample of the accredited colleges and universities in the United States, Wikipedia has articles for the majority of institutions.  This is particularly true when considering 2- and 4-year institutions, especially public ones.  Further, those Wikipedia articles are placed very highly in search results, usually immediately proceeding the institution’s official website.  Not only are U.S. college and university Wikipedia articles very common, they’re extremely popular.

(The data are available here:

• IPEDS population
• IPEDS sample
• Wikipedia population
• Wikipedia sample

A few of the spreadsheets are rather large for Google spreadsheets so they’re a bit sluggish.  Sorry!)

Comments(2)

(This is largely a note for myself.  I had an epiphany while showering this morning and I don’t want to forget it!)

I haven’t touched it for a while but for a few years I’ve been working on historical research focused on entertainment and communications technologies in American college and university residence halls. As is often the case, I began this research as it was a topic of interest to me; I placed only superficial thought on practical applications and implications. In other words, I did it only because I liked it and it interested me. But that won’t convince others to care about this research, to listen to me discuss it, or allow me to publish it.

This morning I finally found my hook. This will be the first time I’ve written it down so let’s see how it looks in print:

Understanding the history of entertainment and communications technologies in residence halls provides us with a means for understanding the tapestry of forces that have shaped not only residence halls but academia in the United States. These technologies provide rich examples of innovations motivated by economic competitiveness, cultural expectations, and academic experimentation.

Not only does this provide me with a much-needed organizational framework for this work but it also provides others with a motivation for understanding and supporting this historical research.

Comments(0)

(This is a very brief summary of a paper a colleague and I presented on Monday, March 31, at the AIR Forum in Chicago, IL.  Both the paper and the presentation are available on the NSSE website; please consult the paper or contact us for more detailed information.  We hope to further develop this paper and submit it for publication very soon so your comments and questions are very much appreciated!)

In the paper Allison BrckaLorenz and I presented earlier this week, we used data collected with the 2009 administrations of the National Survey of Student Engagement (NSSE) and Faculty Survey of Student Engagement (FSSE) to examine how often these two populations – students and faculty – use academic technologies.  We added several questions about technology to the surveys administered to some institutions.  In this paper, we examined the responses to those additional questions from senior undergraduate students and faculty who teach them at 18 institutions who participated in both NSSE and FSSE.  Specifically, we (a) compared student and faculty responses and (b) explored responses across academic disciplines.  However, to keep this blog post a manageable and readable length, I will omit most of the discussion of disciplinary differences; I encourage you to read the full paper if you are interested in those findings.

The survey question on which we focused was multi-part and asked respondents how frequently (Very often, Often, Sometimes, or Never) they used some academic technologies:

1. Course management systems (WebCT, Blackboard, Desire2Learn, Sakai, etc.)
2. Student response systems (clickers, wireless learning calculator systems, etc.)
3. Online portfolios
4. Blogs
5. Collaborative editing software (Wikis, Google Docs, etc.)
6. Online student video projects (using YouTube, Google Video, etc.)
7. Video games, simulations, or virtual worlds (Ayiti, EleMental, Second Life, Civilization, etc.)
8. Online survey tools (SurveyMonkey, Zoomerang, etc.)
9. Videoconferencing or Internet phone chat (Skype, TeamSpeak, etc.)
10. Plagiarism detection tools (Turnitin, DOC Cop, etc.)

The average responses to this question are shown in the figure below.

A few things are apparent from this figure and the responses that it displays.  First, the only technology that students and faculty really use is course management systems (CMSs); most respondents never used the other technologies.  Second, except for Plagiarism detection tools, students are reporting more frequent use of these technologies than faculty.  This is particularly noticeable for collaborative editing software, a technology that students probably use outside of class to collaborate much more often than they use it during class or when specifically assigned to use it.

Another way to make sense of these survey responses is to use cluster analysis to group respondents together.  For students, a 4-means cluster analysis made the most sense:

The students in the High and Medium Use clusters used multiple technologies with relatively high or medium frequency (remember that most students never used most of these technologies).  Students in the Low Use cluster only used CMSs.  And students in the No Use category didn’t really use any of these academic technologies.

A 3-means cluster analysis was most appropriate for the faculty respondents:

As with the student clusters, faculty in the High Use cluster used multiple technologies with some frequency.  Faculty in the Low Use cluster only used CMSs and faculty in the No Use category didn’t really use any of these academic technologies.

From these figures, it is clear that most students and faculty are making little use of academic technologies except for course management systems like Blackboard and Sakai.  Given the resources campus have invested in these particular technologies, it is probably good that faculty and students are making frequent use of them.  However, we stop short of making a subjective judgment based on these responses as there are certainly many instances in which technology is neither helpful nor appropriate in classwork and assignments.  A better approach – one that may be impossible using self-administered surveys – would be to understand not just how often students and faculty use technologies but how often they use them appropriately and well.

More interestingly, students are reporting a higher usage of these academic technologies than faculty.  Most likely, these technologies are not be required by faculty but used by students on their own initiative to complete their work and collaborate and communicate with one another.  The differences between student and faculty responses might be a result of the methodology of this study.  But these differences are probably real and point to genuine differences in how frequently students and faculty use these and other technologies, differences that may result in tension and other differences between these two groups.

Comments(4)

(I started to write an e-mail to some colleagues outlining my current and upcoming projects and the e-mail was getting a bit long.  So I’m writing it all out here as perhaps some of you will be interested in one or more of these projects.)

Here are my current and upcoming projects, listed in no particular order…

• Continue editing and submit for publication (EDUCAUSE Quarterly?) the paper (A Comparison of Student and Faculty Academic Technology Use Across Disciplines) I just presented with Allison BrckaLorenz at the AIR Forum.
• Finish preparing for my ResNet 2010 assessment preconference session.
• Continue working with the ResNet 2010 hosts to schedule and conduct attendee focus groups to supplement the survey data we recently collected regarding the current state and future direction of the ResNet organization.
• Two potential AERA proposals:
  • Discourse analysis of #sachat.  I wrote a solid paper for the discourse analysis class I took in the spring but Rey Junco will be helping me to redo some of the analysis and edit the paper.
  • Historical analysis of student affairs and technology.  I have a solid draft of this paper already done (another class paper) but it’s very long and needs to be edited down to a more manageable, readable length.  Additionally, I’ve recently discovered that we have in the library stacks at Indiana University proceedings from NASPA and ACPA meetings held during the first half of the twentieth century.  I need to spend time in the library with those proceedings as I haven’t yet incorporated them into my study (I didn’t know where I could find them; I certainly didn’t expect to find them at my home institution!).
• Begin a new project analyzing the demographics of student affairs professionals.  I wanted to use these data in my Twitter research but no one has done this work in 15 years so I’ll have to do it (I hope that I’m wrong and that I simply haven’t found a current or recent source!).
• Wait to hear back from ASHE to know if our Wikipedia proposal has been accepted.  If so, then we need to do more work on it to update it and get it into shape for the conference later this year.

Of course, I have other things going on and coming up: quals in 2 months, ongoing projects at work, and beginning data collection for my dissertation.  I thought that summer – especially the summer after you finish coursework – was supposed to be quiet and relaxing?

Comments(0)

A few weeks ago, a colleague (Chris Medrano) and I submitted a paper to the 2010 ASHE conference. The paper is a content analysis of Wikipedia articles covering American colleges and universities.  Chris and I believe that we – higher education scholars, administrators, and policy makers – can learn a lot about what the general public believes is important and interesting in higher education by analyzing Wikipedia articles about individual colleges and universities.

I hope this paper is accepted (otherwise I wouldn’t have submitted it!) but I know it’s a bit “out there.”  Despite my apprehension, I firmly believe that we must be mindful of how the public perceives higher education and the explosion of information available on the Internet provides an incredibly rich source of information if we can figure out how to harness it (In this vein, I am extraordinarily happy and grateful to have had the opportunity to study web content analysis and computer-mediated discourse analysis, giving me some of the necessary background and tools to study these data!).  And given that (almost?) every significant college and university in the United States has a Wikipedia article that (theoretically) lies largely outside the control of the institutions, these articles are a rich source of public opinion.

I know what some of you are thinking: Wikipedia editors don’t represent the general public!  I’m not entirely convinced that is true – especially without data – but I’ll concede the point anyway.  Even if those editing the articles are not representative of the general public, surely we can agree that the information placed in these articles clearly indicates what the general public is learning about these institutions from Wikipedia.  So it’s still important to know what’s going on in these articles.

Since we submitted our paper, Wikipedia articles have gotten another boost in visibility and importance: Facebook is making heavy use of Wikipedia articles in Community pages.  This has already raised a discussion within Wikipedia (full disclosure: I’m one of the participants in the discussion) about the role (or lack thereof) Wikipedia should play given that articles are being displayed in Facebook.  More specifically, at least one institution has objected to the graphic that is being displayed in Facebook.  The topmost graphic in nearly all of these Wikipedia articles is the official seal or crest of the institution.  But most institutions have graphic identity standards that mandate the use of another set of graphics (their “wordmark”) and limit the use of the official seal or crest.  Of course, Wikipedia is not required to honor those standards and it’s pretty clear that fair use allows Wikipedia to use official seals and crests without the permission of the institutions.  This is the kind of interesting complexity about which higher education administrators and scholars should know and in which they should appropriately participate.

Love it or hate it, Wikipedia is an immense force in today’s information societies.  We don’t yet know exactly what role it plays in the college choice process but we can be certain that many people are learning about our institutions via Wikipedia.  We can not and should not control the information in Wikipedia but we should be aware of it and the communities that create, edit, and even vandalize that information.  And we should be eager to use that information to develop a better understanding of how the public views higher education and our institutions.

[August update: The proposal has been accepted.  I look forward to sharing the final paper here and at ASHE this fall.]

Comments(1)

I just received IRB approval to begin conducting research on the weekly student affairs-related discussions being held on Twitter.  The initial round of research is being conducted for Susan Herring‘s Computer-Mediated Discourse Analysis class but I plan to expand the research and present and publish it more broadly once I’m done with the class.

For those who are unfamiliar with #sachat, here is how I described it in my first paper for this class:

Beginning in the fall of 2009, a group of American higher education administrators began using the micro-blogging tool Twitter to communicate, collaborate, and connect with one another.  Each week for at least one hour, these professionals employ Twitter as a public synchronous mass communication medium by marking each of their messages with the #sachat hashtag and discussing a predetermined topic of professional interest.

Each Wednesday, student affairs professionals use Twitter to vote on a topic of discussion.  On Thursday, these same professionals discuss this topic (and others) for at least one hour.  These discussions are loosely coordinated and moderated by one user associated with the TheSABloggers.org website.  Although the participants are highly-educated professionals and many of the topics are related to their professional interests, the tone of the discussions is informal and often playful.

Using Twitter for these conversations imposes particular properties and restrictions.  First, Twitter is nominally an asynchronous medium; by collectively participating at a prearranged time, these users are effectively using Twitter as if it were synchronous.  Second, to coordinate all of their discussions, including the voting and discussion outside of the established hours, participants must include in their messages the phrase “#sachat.”  This phrase – a Twitter “hashtag” – allows Twitter users to search for and categorize these messages.  Third, Twitter restricts messages to 140 characters.  Finally, although Twitter users can address particular users in their messages there is no threading or other advanced addressing functionality.

Since this class is focused on computer-mediated discourse, I’ll be analyzing patterns in these online conversations in terms of features such as participation, message complexity, speech acts, topic development, and politeness. I’m initially focusing on the discussion that occurred on January 21 so I can learn and begin to understand these methods used in discourse analysis.  Later in the semester, I’ll expand my analysis to also include January 14 and January 28 (daytime only; I can’t seem to locate an archive of the evening conversation) for my final paper in this class.  Eventually I would like to expand the analysis to include more discussions and to include content analysis in addition to discourse analysis so I can write a fully-formed paper for publication or presentation (I’m thinking maybe AERA 2011 if I can meet their submission deadline in late summer).

I am interested in conducting this research not because it focuses on Twitter but because it focuses on a grassroots community that has found a unique way to connect and communicate with one another.  It’s especially interesting because their method of communication is free and this is a time of financial stress with reductions to or eliminations of professional development budgets prominent at many institutions.

Many of the methods I’ll be using have been pioneered or extensively used by Susan Herring.  It’s terribly exciting to learn from and with her as she is probably the world’s foremost expert in these methods!  This is the second class I’ve taken with her and it’s a lot of fun to learn from someone who not only intimately knows the topic but is also still really excited about it and super supportive of new, young researchers.

If any #sachat participants have questions, concerns, or suggestions, please share them with me!  Although the data are all publicly-available, I will be using pseudonyms in all of my public presentations and papers so hopefully that will allay any privacy concerns.  Additionally, I imagine that I’ll eventually file an IRB amendment so I can officially talk to you about your experiences and opinions (because a study on this topic seems incomplete without actually talking to the participants).  But in the meantime I’m definitely open to informal discussion, especially if you have concerns about this research.

(And can someone throw a link to this post out there in Twitter and tag it with #sachat?  I would do so myself but I am trying to retain some distance as I study this phenomenon.  More importantly, I just don’t have time right now to jump into Twitter, at least not this month as I prepare for quals and begin preliminary work on my dissertation.  There are only so many hours in the day…)

Comments(4)

This post is a rehearsal of part of a presentation in which I’m participating in a few weeks at ELI.  The presentation is entitled “Using NSSE and FSSE to link technology to student learning and engagement” and I’ll be giving it with one my colleagues here at Indiana University’s Center for Postsecondary Research, Amy Garver.

The relationship between student engagement and technology is a hot topic right now.  The current issue of EDUCAUSE Quarterly focuses on this relationship.  Both the National Survey of Student Engagement (NSSE) and the Community College Survey of Student Engagement (CCSSE) have focused on technology.  NSSE most recently published technology-related findings in 2008 and 2009 Annual Results (CCSSE followed suit in 2009) but we’ve poked at this topic several times in the past ten years.

In general, every time we’ve examined this relationship we find it to be positive.  The relationship isn’t always terribly strong but it’s positive and significant*.  More importantly, this relationship appears to persist no matter what we throw into the mix.  We’ve tried many different things (“controls”) to see if there is something tricky going on, such as a complex relationship with other variables.  For example, it’s possible that students from more affluent backgrounds both use technology more often and score higher on our measurements of engagement because they had better schooling.  But that doesn’t appear to be the case.  At the moment, however, it appears simply that “technology is good.”

That conclusion is neither satisfying nor likely.  It’s not satisfying because it seems very shallow and not at all explanatory (e.g. it doesn’t tell us what it is about “technology” that encourages more engagement and better learning).  It’s not likely because several decades of research has told us that it doesn’t matter which medium we use to deliver education (Clark, Yates, Early, & Moulton (2009), available as a pre-print, is an excellent overview of this body of research).

So if we don’t accept the overly-simple statement that “technology is good,” what do we do?  We did two things.  First, we focused on some specific technologies so we could move beyond broad conceptions of technology and look at some tools currently in use.  Despite the excellent research that tells us that technology itself should not have an impact, we must keep an open mind and explore that possibility, especially as technology advances and becomes more complex and ubiquitous.  Second, we asked faculty participating in the Faculty Survey of Student Engagement (FSSE) a nearly identical set of questions as we were asking the students participating in NSSE in the spring of 2009.  We even convinced 18 institutions to administer both sets of questions!  We wanted to draw faculty directly into the mix because the most likely explanation for our repeated finding of “technology is good” is that use of technology is associated with good teaching.  (That hypothesis also seemed to tentatively arise from one of our studies of distance learners, a study that didn’t seem to do much to cut through the clutter despite using sophisticated methodology.)

We presented some of our results at POD’s 2009 conference in Houston.  As mentioned above, we’ll be presenting some more at ELI’s 2010 conference in Austin.  And we’ll be presenting again at AIR in Chicago in a few months.  These are all different presentations focusing on different aspects of our data.  And there is still data we haven’t yet analyzed and presented!

I’m sorry that I haven’t give you any answers in this blog post.  We’re still working to find them and so far it’s been devilishly difficult.  It’s probably hard for us because our tools – voluntary, self-administered surveys administered to massively large groups of students and faculty – are blunt objects with limited capabilities.  And every answer we find raises more questions.  But it’s clear that there is positive relationship between student use of technology and student engagement, even if the relationship is more complex than it appears on the surface.

* – Statistical significance is tricky for us.  Our data sets are enormously large and since significance is sensitive to sample size a whole lot of things are significant.  So we often turn to other measures such as effect size and other contextual indicators to make sense of our data.

Comments(0)