Additional (older) #SAchat data: Participation, Geography, and Gender

In a comment to my previous post sharing some of my thoughts about #sachat in advance of their “State of #SAchat” discussion tomorrow, Gary Honickel asked about demographics of #sachat participants.  In our forthcoming chapter (I’m not trying to advertise it – honest! Just trying to explain why I have all of this information. I’m a researcher, not a stalker!), Laura Pasquini and I analyze #sachat and we include some information about the participants.  We didn’t include the specific information Gary asked about: gender and geographic location of participants.  But I did collect that data and although it’s for three sessions that occurred last year maybe this is still useful or helpful.  My sense is that these things haven’t changed much in the past year.

Keep in mind that these data come from three 2011 chat sessions:

Date Topic Participants Messages Average messages/participant Standard deviation messages/participant
March 10, 2011 Beyond the Conference: Networking When You Aren’t Attending a National Conference 70 442 6.3 6.5
June 2, 2011 Intentional Recruiting to the Field: Responsibilities and Liabilities 83 442 5.3 5.3
June 30, 2011 Creative Orientation Approaches and Ideas 45 323 7.2 10.2

The thing that jumps out at me in the table above are the average number of messages per participant and the standard deviation of that number.  There is immense variance in the number of messages posted by each participant and that makes me wonder about the pattern(s) of participation for each session.  The histogram below showing how many people posted a particular number of messages in each chat helps us understand these numbers (click on it to view a larger version).

This histogram is a classic “long tail” distribution, showing us that most participants in these three #sachat sessions posted very few messages and only a handful of participants posted many messages; the participant with the most messages is, of course, the moderator.  This is a very typical situation and an unsurprising finding.

This gives us a broad understanding of #sachat participation but let’s look a bit deeper and explore two different ways of classifying participants: gender and geography. First, a few words of caution: these data were inferred from the Twitter profiles and messages posted by these participants.  Geography was the easier datum to capture for each participant as most participants associated themselves with a particular college or university, either in their profile or in their introduction during one or more #sachat sessions.  Gender was much more difficult and I present these data with trepidation because there was a significant amount of guesswork involved in classifying participants as male or female.  If this were anything more than a one-off blog post or if gender were a central concern for this or any other analysis, I wouldn’t even share or use these data because inferring gender from name and photo obviously lacks rigor.

This chart shows the geographic locations of the participants in these three #sachat sessions (I used the U.S. Census geographic regions to aggregate the data).  Nothing surprising here.  #SAchat is indeed U.S.-dominated but even that isn’t a surprise.  Nothing particularly interesting is discovered if you look at the number of messages posted by participants from each region; the numbers get very small very quickly when slicing the data this many ways so it’s not worth trying to display.

 

What about gender?  For at least these three sessions, the gender breakdown seems to be about even.  Like geographic region, nothing terribly interesting happens if you slice these numbers in different ways.

So what do we make of all of this?  I think it shows that – for these three sessions – there was considerable diversity among #SAchat participants, at least in two ways we can measure. Of course, these are coarse (and in the case of gender, potentially problematic) measures and there are many other ways in which we might examine the makeup and diversity of this population.  Functional area and role (student, entry-level professional, faculty, etc.) are two measures that jump to mind as interesting and useful.  (Incidentally, I tried to classify participants using those two measures in a previous study; it was difficult, time-consuming, and very incomplete since those data are not spontaneously volunteered by all participants.)

Are #sachat participants diverse enough?  I don’t know.  How do we define “diverse enough?”  Should we be concerned about how well the #sachat population matches the larger student affairs population?  A quick glance shows some alignment between these populations but I have not done any definitive work in this area, partially because it’s very hard to obtain data about the larger student affairs population.

Of course, all of this does not and can not include anything about lurkers.  I agree that there is value in #sachat even for those who do not directly or visibly participate but we’d have to make a concerted effort to identify those people if we want to know anything about them.

I hope this is helpful or interesting!  I wish I had more up-to-date data but I don’t.  I’m job searching, working, and trying to finish a dissertation so I don’t have time or plans to gather additional data right now.  This is data that I had at hand and I am happy to share it in the hopes that it’s useful for someone.

Reflections on #sachat

Tomorrow, the members of the #sachat community will be engaging in introspection and discussing “The State of #SAchat” instead of their usual weekly discussion of topical student affairs topics.  I have been conducting research on the #sachat community for a couple of years now so I thought it might be helpful for the community if I could organize and share some of my thoughts.

I won’t spend time describing the basics of #sachat; if you are interested in this particular conversation, I assume that you are familiar with the community and its tools.  If I wrong and you are not familiar with #sachat, the official overview is here.  An annotated visualization of one chat session – a February 10, 2011 discussion about job searching – is below (my original blog post discussing this visualization has some of its background details).

The chart below shows Twitter message traffic from six hashtags – #highered, #sachat, #sadoc, #sagrad, #sajobs, and #studentaffairs – during the week of June 27, 2011.  This illustrates how #sachat differs in that it not only has consistent traffic everyday (although not as much as #highered) but it spikes during the scheduled chat session on Thursday afternoon.

In a book chapter Laura Pasquini and I have in press, we examine #sachat as a case study of informal learning using technology.  One of our conclusions is that #sachat is doing several things right to overcome the significant limitations of Twitter by:

  • Allowing participants to direct the discussions as much as practical.  For example, potential participants vote on each week’s topic and do not have to register to participate (in the voting or the actual discussion).
  • Using other tools to supplement the core use of Twitter.  Most of these tools reside on the SA Collaborative website.  One of the most important may be the chat archives that give the chats a sense of continuity and history beyond the typically ephemeral nature of Twitter.
  • Employing a well-prepared and clearly identifiable moderator in each discussion.  This account helps impose order on the Twitter chat, allowing conversation to run for a bit before drawing it back to the core topic by using clearly marked, pre-prepared questions.

We also identify several specific concerns and challenges:

  • Can the participants continue to overcome the inherent limitations of Twitter, especially its (a) short message length, (b) lack of threading, and (c) ephemerality?  Although some participants attempt to overcome the first limitation using multipart messages, this is not very successful; the 140 character limit of Twitter is one of its core features and unlikely to be overcome.  The second limitation has been addressed with some success with the use of MOD messages and Q# replies.  The third limitation has been partially overcome by regularly making transcripts of chats publicly available.
  • Is the small community of volunteers that run the chats – those who use the moderator account and the SA Collaborative website – sustainable?  These volunteers and the tools they provide and maintain are essential to the success of the community.  For how long will these volunteers sustain their energy and will there be a smooth transition as members come and go?
  • How representative of the larger student affairs community is the #sachat community?  Is that important?
  • How diverse are the members of the #sachat community?  In what ways are they diverse and in what important areas is diversity lacking?

Limitations and Lost Nuance: Twitter Does Not Improve Grades

I’ve watched with interest over the last several months as media outlets and individuals have discussed, blogged, and tweeted a study conducted by Junco, Heiberger, and Loken. Their study reported that a group of students who used Twitter as part of a class earned higher grades than classmates in sections of the class that did not use Twitter. It’s a nice study that is clearly described and methodologically sound. Like all studies, it has significant limitations and they are concisely and honestly discussed in the study but those limitations have been ignored by too many people who have made the study into something it’s not.

The study concluded that “Twitter can be used to engage students in ways that are important for their academic and psychosocial development” (p. 10). But is that what has been reported and discussed by others? No, of course not; if it were then I wouldn’t be writing this sanctimonious blog post! Mashable, a very widely-read and influential technology blog, reported on the study using the headline “Twitter Increases Student Engagement [STUDY].” A recently-created infographic proclaims that “Students in classes that use Twitter to increase engagement have been found to average .5 grade points higher than those in normal classes.” Another infographic proclaims that “[Students get] grades up half a gradepoint in classes that use Twitter.”

I get that pithy headlines and concise summaries are necessary to grab attention. But by overlooking or ignoring the details of this study, those headlines and summaries get this all wrong. Let’s return to the original study to understand why.

In the study, the researchers assigned some sections of a class to use Twitter. While the entire class used Ning, these sections also used Twitter to complete some received additional assignments. They also received guidance and encouragement to use Twitter to communicate not only with one another but also with instructors. At the end of the semester, these students had earned higher grades than their non-Twittering classmates.

If I understand the study’s methodology (Rey, please correct me if I got anything wrong!), it seems that this study does not show that “Twitter improves grades.” It shows us that students who do more work and spend more time concentrating on class materials can earn higher grades. It shows us that students who have additional opportunities to communicate and collaborate with one can another earn higher grades. It also shows us that students who have greater access to instructors can earn higher grades. It shows us that Twitter can be a viable medium for students to communicate and coordinate with one another and instructors. And, yes, it shows that Twitter can be an effective educational tool when skillfully incorporated into a class with appropriate support and structure. In a critique of one of the infographics, Junco specifically mentions this: “Yes, that’s our study about Twitter and grades. Unfortunately, what’s missing is that we used Twitter in specific, educationally-relevant ways—in other words, examining what students are doing on the platform is more important than a binary user/nonuser variable.”

This illustrates the challenge with testing the efficacy of educational tools and techniques: It’s really, really hard to isolate just the impact of the tool or technique. To test the tool or technique, you almost always have to make other changes and it’s usually impossible to tell if those changes changed the results of your study more than the tool or technique you intended to study. It’s a limitation of nearly every study focusing on the effect of particular media on education and it may be an inherent limitation for this kind of work. (Richard Clark has been pointing this out for decades; look into his writings for more detailed discussions. He’s also been wonderful in creating dialog with his detractors so there are well-documented and substantive discussions between many different scholars with different opinions.)

Hence my frustration with how this study has been summarized and passed around: By ignoring the limitations and nuance of this study, these summaries miss the boat and draw a grandiose conclusion that the authors of the study never attempt to draw themselves. That’s a shame because this is a nice study that is interesting and informative. But like most research, it’s a small step forward and not a giant, earthshaking leap. Summarizing this study by proclaiming that Twitter is a magic ingredient that can be added to classes to increase grades is irresponsible and misleading.

Update 1: Thanks for the clarification about Ning, Liz!

Update 2: Another example of how headlines can distort or misrepresent research has just popped up. Before correcting the headline, Colorlines reported that the majority of college students are part-time students (full headline before being corrected: “Study: Majority of College Students are Part-Timers, Less Likely to Graduate”) But the actual report doesn’t say that. Instead, it says that “4 of every 10 public college students are able to attend only part-time” (p. 2). It’s a shame that the research was initially being reported incorrectly because the changing demographics of college students is incredibly important and very misunderstood and overlooked. I know there is a lot nuance in discussions of demographics – race, ethnicity, SES status, privilege, etc. – but if we cover up or ignore the details then we haven’t made any progress.

To their credit, Colorlines corrected their headline once I pointed this out to them. They made a mistake in their initial headline and it’s great they they’re willing to correct their public mistake!

More #sachat analysis: One Illuminating Figure

Laura Pasquini and I are working on analyzing #sachat data, a follow-up to work I’ve done previously but did not formally publish. Part of our work involves looking at a few other student affairs-related hashtags to help us understand #sachat in context. This figure shows the number of Twitter messages posted with particular hashtags – #highered, #sachat, #sadoc, #sagrad, #sajobs, and #studentaffairs – during the week of June 27, 2011. The #sachat session really stands out here both in the number of messages posted and in how it interrupts an otherwise regular daily and weekly pattern. This isn’t a profound discovery but it’s an easy way of illustrating that #sachat sessions are relatively unique and prominent uses of Twitter among some users.

How I (Don’t) Use Social Media

This is an uncomfortable post to write. I’ve never wanted to use this blog to discuss personal issues and it feels very vain and self-important to describe some of my own personal habits and practices. But every time I’ve mentioned the things below people are intrigued and interested. Some people are even relieved to find someone else with some of the same practices. So here goes…

My personality strongly shapes my use of social media. I am introvert and an intensely private person. I am also learning in very profound ways what kinds of relationships I want in my life and I am working very hard to find and nurture them.

Specific ways in which my personality and interests shape my social media practices:

  • Facebook: I don’t use Facebook. Like most people my age (33), I was an avid Facebook user for several years. But I don’t use use it anymore unless I specifically receive an e-mail message or a personal request of some sort. I don’t dislike Facebook or people who use it. I simply reached the conclusion that it was not meeting my needs. I realized several months ago that I didn’t like reading about my friends’ and colleagues’ lives because it was unfulfilling. I don’t want to read about their lives – I want to be part of them. For me, it feels cheap and even a bit hollow to read about and see pictures from someone’s life when I want to be part of that life. Maybe it’s selfish but it’s important to me that we reinforce our relationships in substantial ways. I want to hear about your weekend over coffee, not Facebook. (And I never got anything out of Facebook as a scholar, student, or professional; maybe I just never looked in the right places for substantive information or support.)
  • Twitter: I don’t follow anyone. I typically use Tweetdeck and I have it set up to search for several hashtags and subjects of interest to me. It’s how I try to avoid the banality of Twitter: I don’t care what you had for breakfast but I do care if you have something to say about a passion we share.
  • LinkedIn: I don’t have a LinkedIn account. The idea of pure networking – meeting and “connecting” with people just to use them – is morally offensive to me. People are not means to ends and I refuse to use them in that manner. Yes, I’m sure that I’ve got a very skewed and probably incorrect perception of LinkedIn and how it’s used (e.g. I know some people love the discussion forums and get quite a bit of professional knowledge and support there). But I’m okay with that and with those who use LinkedIn; I just don’t think it’s for me.
  • FourSquare: I don’t have a smartphone so naturally I don’t use FourSquare or other similar tools. Even if I had a smartphone I don’t think I’d be comfortable broadcasting my physical location (although it would simply alternate between “work” and “home” most of the time). I don’t agree that “privacy is dead” but I think that we’re (often unwittingly) doing our damndest to kill it.

I’m not a Luddite or an antisocial recluse. I just have a very good idea what I want out of life and my relationships with others and I don’t care to use tools that don’t contribute to my life in the ways that I believe are positive. I know there is a price to be paid for a refusal to use these tools or an unusual usage of them. I’m okay with that.

Maybe you think I’m wrong or misguided. I’d love to hear from you! And I’d love it even more if we could spend time together substantively addressing and appreciating one another. So let’s not discuss this on my Facebook wall. Let’s discuss this over coffee, drinks, or dinner.

Yes, I know that’s unrealistic and we’re destined to have most of our conversations in blog comments, Twitter messages, e-mail, and – if we’re lucky – Skype. But a guy can dream, right?

Visualizing #sachat Data (First Draft)

It took me much longer than I had hoped but today I finally finished a first draft of a visualization of some of the #sachat data I’ve been working with:

The method of analysis used in this video is dynamic topic analysis (DTA). DTA was developed by Dr. Susan Herring and more information about the method can be found in one of her papers published in 2003. This method of visualizing DTA data was created by Andrew Kurtz and Dr. Susan Herring. Andrew’s original Java tool isn’t working for me any more so I developed the graphs in this video using Excel macros to generate R scripts (which has been an adventure because this is the first time I’ve used R).

The music in this video is Tutto L’Amor Perduto by Giorgio Costantini. It is available under a Creative Common license at BeatPick.com.

I created this and publicly posted it for several reasons. First, this really is a first draft and I would love feedback so I can improve it. I’ve already noticed a few small mistakes that will be corrected in the next version. I’ve also received some feedback and suggestions for possible improvements. If you have some, please let me know!

Second, I hope that some in the #sachat community find even this very rough first draft interesting, informative, and possibly even useful. Even though I’m comfortable studying a group that is so very public with their actions and membership, I still believe that I should give back to that community in ways that are appropriate and helpful. It just seems like a nice thing to do and it’s a small way of showing my appreciation to them.

Finally, I’m interested in seeing if there is interest in helping me continue this kind of work. One of the reasons why this is only a rough draft is that I’m the only one who has analyzed these data. DTA is a specialized form of content analysis and, like any content analysis, it should be performed by multiple persons to ensure the codes are being applied consistently (which is why good content analysis studies report interrater reliability figures to help bolster the credibility of the findings). This analysis – and it should hold up well even when other coders are added – shows that this particular use of Twitter is moderator-led discussion with coherent threads of discussion. I need to analyze a few other #sachat sessions to ensure this is consistent for other sessions. I also need to analyze some other Twitter data so I have some useful points of comparison.

I think this use of Twitter is fairly unusual and it would be great to be able to publicly discuss that with confidence. This is a wonderful example of a group of people using a very limited tool to do very good things that transcend (my) expectations and it should be represented in the research literature.