Thoughts on Backward Design

 This post will be less organized than most posts; some of these thoughts and ideas are still a little raw.

Backward design – the method by which one begins with the desired end result(s) of an educational program, determines acceptable evidence showing that the result(s) has been achieved, and then creates a plan to teach the skills and content that will lead students to provide that evidence – has been on my mind lately.  It’s one of the core concepts of a college teaching and learning course I co-teach but that’s not why I’ve been thinking about it.

For me, backward design is a “threshold concept;” it’s an idea that changed how I think about teaching and I can’t go back to how I thought prior to this change.  So although I learned and most often use and teach backward design in the context of designing or redesigning a single college course, I’ve been thinking about the role of backward design in different contexts.  For example:

  • I know that backward design has been and is used to develop curricula and not just individual courses.  Today was the first time I got to see firsthand how that plays out with a group of faculty to develop a full 4-year curriculum for this discipline.  I was most struck by how difficult it was to keep true to the backward design philosophy and not get mired down in content coverage and the limitations imposed by the current curriculum.  It was difficult even for me to remain on course as I tried to help facilitate one of the groups of faculty engaged in this process.  I underestimated the increased complexities involved in scaling up the process from a single course to an entire curriculum; it’s not a linear function.
  • There has been quite a bit of discussion lately among student affairs professionals regarding their conference presentations (e.g. this Inside Higher Ed blog post with 30 comments).  Put bluntly, many people are unsatisfied with the current state of these presentations.  Just as backward design can scale up from a class to a curriculum, it can also scale down to a single class session.  And shouldn’t a good 50 minute conference presentation resemble a good 50 minute class session?  So why not systematically apply backward design to conference presentations?  Many conferences seem to try to push presenters in that direction by requiring them to have learning outcomes for their sessions but that isn’t enough.
  • Unfortunately, pedagogy and good teaching practices are not formally taught and emphasized in most student affairs programs so I expect that most student affairs professionals have not been exposed to backward design as a formal process.  That’s a shame because it seems like such a good fit for what student affairs professionals do!  And it fits in so well with the ongoing assessment movement because it so firmly anchors design in measurable outcomes and evidence-based teaching!

Would any student affairs professionals out there want to learn more about backward design and try to apply it to some of your programs?  Please let me know because I’d love to help!  I’m positive this would work out well and I’d love to test these ideas!

“Best” Practices?

In a recent blog post releasing a (very nice!) infographic about “Best Practices in Using Twitter in the Classroom Infographic,” Rey Junco writes:

I’d like to point out that I’m a real stickler about using the term “best practices.” It’s a concept we toss around a lot in higher education. To me, a “best practice” is only something that has been supported by research. Alas, most of the time that we talk about “best practices” in higher ed, we’re focusing on what someone thinks is a “good idea.”

I agree and I’m even more of a stickler. There have been several specific situations in which I have been asked or encouraged to write a set of best practices for different things but I always got stuck asking myself: What makes this particular set of practices the “best?” I share Rey’s dislike of “good things I’ve done” being presented as best practices. But my (relatively minor) frustration extends a bit further because to me the adjective “best” implies comparison between different practices i.e. there is a (large) set of practices and this particular subset has been proven to be better than the rest.

I’d be perfectly happy if people were to stop telling us about best practices and just tell us about “good” practices until we have a large enough set of practices and data to judge which ones really are the best. If you’ve done good work, don’t distort or dishonor it by trying to make it bigger than it is. After all, even Chickering and Gamson (1987) presented their (now-classic and heavily-cited) ideas as “Seven Principles for Good Practice in Undergraduate Education” and not “Seven Best Practices in Undergraduate Education.”

Growing Beyond an Established Digital Identity

I have run into an unexpected and interesting issue.  Although I am not as far along with my dissertation as I would like to be, I have decided to hit the job market.  Some of the jobs to which I am applying are not directly related to student affairs and technology, the primary topic of this blog and the tagline of this website.  That makes me a bit nervous.

It’s natural for people to want to make career changes, large and small.  But I never considered how to handle making such a career change when I have a strongly established digital identity that is not directly aligned with the desired career.  This is particularly tricky because I have a diverse skillset and I am applying to a diverse set of jobs from faculty development to student affairs assessment.  How will potential employers handle an apparent disconnect between my established digital identity – the topics I’ve regularly discussed and the areas in which I have publicly proclaimed expertise – and the jobs to which I am applying?

I am not misrepresenting myself in my application materials.  There are many skills I have acquired and interests I have developed that I simply haven’t discussed here, especially some that don’t seem to be on-topic.  But will potential employers take my claims of competence and experience seriously when they weigh these “new” and undiscussed skills and interests against those I have repeatedly and publicly discussed?

I don’t have answers for these questions right now.  But I will soon because this is not a theoretical issue but one I am actively confronting right now.

What can I do?

  • Scour my materials to ensure that anything I already have online that is relevant is accurately tagged, perhaps even highlighting those collections of materials somehow.
  • Quickly begin to build up a (larger and more visible) body of blog posts related to these other topics (e.g. Scholarship of Teaching and Learning, faculty development, assessment).
  • Tweak the tagline of this website so it’s aligned with a broader set of my professional interests.
  • Create alternative expressions or evidence of competence and experience with these other topics (e.g. e-portfolios).

If I were always completely open and transparent about all of my interests and experiences, I wouldn’t have this problem because these facets of my identity would already be visible.  But I think it’s healthy and even necessary to consciously practice some level of self-censorship and selection, at least for me.  I just need to figure out how to present multiple facets of my identity with integrity now that it has become necessary for me to do so.  And hope that others can perceive that I am acting with integrity and understand what has happened.

Little Things DO Matter

I’ve never liked the trite phrase “don’t sweat the little things.” I have no argument with the general idea that you should spend most of your time on the large, important things. But I reject the implication that the little things aren’t important and not worth spending time on. It offends my passion for detail and belief that details are important. More importantly and more defensible is the idea that “little” is relative; what is little to one person is large to another.

Let me offer an example.

One of the projects at my research shop, the Law School Survey of Student Engagement (LSSSE), focuses on law schools and law students in the U.S. and Canada. I don’t have any formal responsibility beyond general collegiality and professionalism to work with the project and its staff. However, I work on LSSSE projects when they need assistance and my schedule permits because (a) the work they do is important and interesting and (b) I love working with the LSSSE staff. A few months ago, the LSSSE folks needed some help preparing their latest Annual Results and I was very happy to help. They surprised me a few weeks ago by letting me know that in return for my assistance they gave me “top billing” in the Annual Results by including me in the LSSSE staff listing on page 1 of the report.

In many ways, this was literally a little thing. It costs the LSSSE staff virtually nothing to do this. It’s less than half a line of text that few people will ever read (even if you’re interested enough to read the LSSSE Annual Report I doubt that you’ll read through the staff listing, too!). And it only took them a few second to include my name in the document.

But to me, it’s not so little. How wonderful that the LSSSE staff thought enough of me to claim me as one of their own! What a kind and unexpected gesture of thanks!

That is why I think it’s important to spend a little bit of time “sweat[ing] the small stuff:” You never really know what is small. So spend some time working on the little things because they may unexpectedly grow into big things.

I Don’t Trust This Article – And Here’s Why

On Friday, a colleague pointed out a new article on Mashable that is titled “Why Tablet Publishing Is Poised To Revolutionize Higher Education.” I don’t trust the claims made in this article. I’m going to explain why I don’t trust the claims, not to convince you that my opinion is correct but to give you an understanding of how I evaluate claims like the ones made in the article. I’ll lay out my thoughts in chronological order.

  1. The article is published at Mashable. I removed Mashable from my RSS reader over a year ago because I got tired of their poorly-written articles that make ridiculously overwrought and unprovable claims. This certainly isn’t enough for me to condemn this particular article but it certainly makes me cautious right from the beginning.
  2. The title makes a very bold claim. Many people have attempted to “revolutionize” education; few have succeeded. And even fewer have been able to explicitly predict revolutions before they occur or even recognize them as they are occurring. The author has a helluva case to make and he better bring remarkable evidence to support his claim(s).
  3. After reading the title, a quick glance through the article indicates that it’s a utopian piece largely based on the idea of technological determinism. In other words, it’s not only wildly optimistic but it also relies on the idea that we can predict and control how people use technologies by the way in which those technologies are designed. Both of these ideas – utopia and technological determinism – have a bit of history in the field of social informatics. The history is mostly negative; these ideas simply don’t work most of the time. So my skepticism continues to increase.
  4. The author of the article is an executive at Adobe. In fact, he’s the “director of worldwide education.” That doesn’t mean that his opinions are necessarily biased but it’s another reason for me to be skeptical.
  5. The article claims that “[There are] better study habits and performance with tablets.” Only one study is cited to support these sweeping claims: a Pearson Foundation “Survey on Student and Tablets.” For example, the author states that “86% of college students who own a tablet say the device helps them study more efficiently, and 76% report that tablets help them perform better in their classes” and a few other claims. Even if this study were flawless, the author needs a whole lot more evidence to support such a broad claim.
    1. To their credit, Pearson offers to share methodological details about and data from their survey if you just ask them; I haven’t asked so I don’t have any more detail than what is provided in that 2-page overview. But we do know that the survey was conducted online. Given that about 20% of people in the U.S. do not have access to the Internet (Dept of Education estimates 18.6% and the Pew Internet & American Life Project estimates 21%), it seems unlikely that an online survey can produce data that is representative of the entire population. It seems particularly problematic to omit non-Internet users when asking about technology since the results will almost certainly be skewed.
    2. Even if we accept that the Pearson numbers are accurate or in the right ballpark, I’m still not sure if they’re very informative. I guess it’s interesting that many young people think that tablets will help them study more efficiently and that they will replace textbooks in the next five years. I just don’t think that we can use these data to make any predictions.
    3. Let’s ignore the validity issues for some of Pearson’s data (e.g. people are notoriously bad at distinguishing between “what I like” and “what is most efficient/effective) so we can move on.
  6. The authors correctly assert that digital textbooks can include more features than printed textbooks, including “video, audio, animation, interactive simulations and even 360-degree rotations and panoramas.” However, the author does not say how we’ll produce all of that additional material. I don’t expect the author to solve every challenge associated with his predicted revolution but it would be nice to at least acknowledge them instead of glossing them over or ignoring them entirely.
  7. In the next section of the article, the author claims that “interactive learning leads to better retention.” The only evidence cited is this news article about a study of elementary and high school students using 3D technology in science and math classes. Of course, since I’m an academic snob I think it would be much better to cite a primary source, preferably one that has been peer-reviewed, than to rely on a popular press article. Once again, even if we accept that this study is perfect it’s not even close to being enough to support such a broad claim.
  8. Next, the author claims that digital publishing can help us better “[understand] learning effectiveness” using “integrated analytical tools.” I have no issue with this claim as a broad theoretical claim. But it seems to completely bypass the fact that U.S. higher education is in complete disarray in terms of even settling on broad learning objectives much less specific objectives and associated assessment tools or indicators. (Look into the “tuning project,” especially the “Tuning USA” project, to get an accurate view of these issues.)
  9. The next claim the author makes is that “digital publishing makes knowledge more accessible.”
    1. The author must be using “accessible” in a different way than I commonly use it because it’s hard to take that claim seriously given the (a) lingering digital divide, participation gap, and similar inequities in the U.S. and (b) the immense resistance many digital publishers have exhibited to making their content accessible to the visually impaired.
    2. Once again, the author focuses solely on a possibility offered by the technology without giving any thought to the cultures in which the technology is embedded. He writes that “digital publishing allows professors or subject matter experts to self-publish their own educational materials or research findings and distribute the information on tablet devices” without offering even the barest hint about how this will occur without adjusting or overturning the systems that would need to support this. In other words, why would faculty do this? What is the incentive?
    3. Similarly, the author claims that “by harnessing interactive technologies, educators can explain even the most complex scholarly or scientific concepts in compelling and intelligible ways.” Once again, I accept this broad claim (ignoring the “even most complex” qualifier because it’s just silly) in theory but balk at it in practice. It takes complex skills to create effective interactive content, skills that are different from those possessed and valued by faculty in many disciplines.
  10. At this point I’m just tired of reading these grand claims supported by flimsy or no evidence…

I’m not a Debbie Downer or a Luddite. I agree with the broad proposition that digital publishing has potential to make a huge impact on U.S. higher education. And I agree that tablets are super cool and very useful in some circumstances; I purchased an ASUS Transformer a few months ago to replace an ailing netbook and I’m very happy with my purchase! Fundamentally, I distrust the claims made in this article because the author fails to support them. Even when the author provides cherry-picked examples and studies, they are often of poor quality and always insufficient to support those claims. This is quite disappointing since the author could have easily drawn upon the large and rapidly-growing body of evidence in this area. I expect very little from an article published by Mashable and this article delivered.

Thumbs Down for CBS News NSSE Article

There are many different angles one could take in reporting on the 2011 NSSE Annual Results; it’s a dense 50-page report. I know that every group has its own agenda and every reporter has his or her own personal interests but it’s very disappointing that CBS News chose the snide headline “Business majors: College’s worst slackers?” for their article. In an ordered list, something must be last. In this case, some major must rank last in the number of hours students typically study each week. But to label that group of students “slackers” simply because they fall at the bottom of the list is unnecessarily mean and unprofessional.

Fun Time of Year: NSSE Annual Results Released

The 2011 NSSE Annual Results were released today. I don’t want to focus on the content of the report in this blog post. Instead, I am briefly noting how fun it is to work on a project with a large impact that regularly receives attention from the press (even if some of the attention is sometimes negative, a very interesting experience itself). It’s gotten more fun each year as I’ve become more involved in much of what we do; this year I directly contributed by writing part of the report itself. Yes, it’s ego-boosting to see my work in print but more importantly it helps address a very serious and difficult problem that vexes many researchers and administrators in higher education: It’s hard to explain to others, especially our parents and extended families, what we do. Instead of trying to convince them that I really have graduated (several times!) and am not wasting my whole life in college, I can send them the report and articles from the New York Times and USA Today and say, “Look – this is what I do!”

Now I get to watch media reports and subsequent discussions to see how they play out and what they will emphasize. This process is unpredictable and it has surprised me in previous years when relatively small bits of information have caught on to the exclusion of other interesting and important information. As The Chronicle of Higher Education notes, this year may be a bit different given recent events but who knows how things will play out.

Two Quick Observations Regarding Online Community

I’m buried in work and research but I have two thoughts dancing on my mind and they’re both related to online community:

  • I hate when websites or tools list reader comments in reverse chronological order i.e. newest messages first. I finally figured out why I hate that: It makes it very difficult to view the messages as a coherent discussion within a pre-existing social context. Because new participants are not immersed in the context of the ongoing discussion they can easily view the opportunity to comment merely as a way to shout messages without any responsibility to engage with or form a community. Mediated communication is difficult enough without us actively encouraging antisocial behaviors and views.
  • Our obsession with tools and technologies leads us to underestimate or ignore the social effects and communities that build up around them. I see this happen all of the time in Wikipedia when new editors leap into articles without having any understanding of the cultural norms of the immense community of users that have used Wikipedia for years. It’s sadly naive to believe that such an immense collection of resources doesn’t have a correspondingly large and complex community with cultural and social norms and expectations.

Quick Update: NSSE/EDUCAUSE Partnership

(I’m working on a longer post but I keep getting interrupted by life so this short post will have to do for now.)

I’m super excited that I’m going to the 2011 EDUCAUSE Annual Conference next month in Philadelphia to work with EDUCAUSE staff and members to develop potential questions for the next version of NSSE! I’ve always been a huge fan of EDUCAUSE and the work they do so I’m very hopeful that this collaboration will be fruitful and help us figure out the right kinds of questions to ask about technology. Over the past four years I’ve been involved in several efforts to address technology in NSSE and it’s very difficult so I’m really excited that we’ll be able to tap into the experience and expertise of technology experts.

I’m also a bit trepidatious about this collaboration. It’s young and in many ways undefined. I am hopeful that it bears fruit but it may fizzle out or even backfire since there is so much ground we have yet to cover and these are two large, complex organizations. Like many such efforts, it also feels like it is very dependent on a small number of people. While we’re all very talented and dedicated, we’re also incredibly busy and it may turn out that our interests are incompatible.

I’m also very thankful that this collaboration has even made it this far. It’s very gratifying that my colleagues are still willing to take risks on public ventures like this even as we continue to experience sharp public criticism. It’s more incredible for me to know that my supervisors have been supportive of this effort even though it has largely been championed by one graduate student. Of course, I haven’t done this or anything here by myself; I’ve had wonderful support from many people in nearly everything I’ve done here, especially from my current supervisor Allison BrckaLorenz who has been an enthusiastic supporter and wonderfully capable advisor from day one. Despite all of her other important responsibilities, Allison is neck deep in this EDUCAUSE/technology-thing with me and I’m so happy that she is involved!

So even though I’m a little fearful that this particular effort could fizzle out or even publicly blow up (which seems extraordinarily unlikely but I’m always a bit paranoid), I go into this knowing I’m not alone and I’m working with and for people as supportive as they are brilliant. I really want this collaboration between two of my favorite organizations to work. If this all works out well – and it will be a couple of years before we really know – it could be very powerful in helping U.S. higher education better understand and use technology to teach and communicate with undergraduates. I know that’s a very lofty aspiration but these two organizations are more than capable of fulfilling it.

Confessions of an Uninvolved Student

This is a further development of thoughts that occurred to me as I read and responded to John Gardner’s latest post.

I have worked in student affairs and I have a Master’s degree in that field. I am a PhD Candidate in one of the world’s best higher education programs. I work at the National Survey of Student Engagement. These experiences and education have firmly drilled into me the benefits of being engaged and active in campus groups, events, and activities. I see and hear from my colleagues and my students the incredible impact of these activities, especially the acquisition of lifelong friends.

Here’s my secret confession: I was involved in virtually nothing as an undergraduate and a Master’s student. I can only name two fellow students from my undergraduate alma mater; I’ve scarcely exchanged Facebook messages with them and haven’t spoken to them since I graduated from the University of Tennessee (with a 2.48 GPA; “C is for cookie, that’s good enough for me!“). The story isn’t much different for my Master’s classmates; with one exception, I only keep in touch with them through coincidental attendance at professional conferences.

My lack of campus involvement was my choice, for good or ill. It’s part of who I am and I can’t envision my life any differently. And I don’t think anyone could have convinced me to act differently or be different.

I make these confessions because I know there are many other students who are making the same decisions and I don’t think those students and their decisions are understood by or respected by many of my colleagues, especially those in student affairs. I get the impression that sometimes those students are viewed with pity and even scorn because they choose not engage in our favored activities in our chosen environment. And that saddens me, especially because we preach the benefits of diversity and choice. Many of us believe those students need to be “saved” but that seems very disrespectful of those students and their choices.