Dissertation Journal: Defended, Edited, Submitted, Accepted

It’s been about a year-and-a-half since my last post about my dissertation.  Two weeks ago, I defended my dissertation NON-RESPONSE BIAS ON WEB-BASED SURVEYS AS INFLUENCED BY THE DIGITAL DIVIDE AND PARTICIPATION GAP.  I’ve included the abstract below if you’re interested in its content but I’ll focus here on some of the process.

I originally intended to write a lot more in this blog about my dissertation-writing process but my posts eventually petered out as I got further and further behind schedule.  After a while, I refused to write about it not only because I had nothing new or interesting to say but more importantly because I was simply ashamed to even bring up the topic.  I don’t know why I stopped writing.  It took me about three years longer to finish this than it should have taken and I can’t help but wonder how different my life and career would be if I had finished in a timely manner.  I’m not sure why I avoided working on this for so long but I know that all of the obstacles were internal and emotional.  And I can’t tell you that I had any miraculous breakthroughs that let me finally finish except for that fact that I was almost out of time.

The defense itself turned out almost exactly as expected.  My committee requested only very, very minor edits that required the addition of only a few sentences.  I had set aside the two days immediately following my defense to make edits and the final submission but I only needed a few hours to make those edits and a second round of (minor typographical) edits requested by my graduate school.  They’ve accepted the document and forwarded it on to ProQuest for permanent archival so I think I’m just waiting for a few random bits of paperwork to work their way through the systems before everything is completely, totally, and finally done.

I’m not sure what my next steps will be.  I worry that the data are too old – Internet access and use data collected in 2010 – to be publishable.  Of course, I have many ideas about how to conduct further data analysis and push this particular set of ideas further but I don’t think that anyone can be surprised that I’m a little bit burnt out on these specific ideas right now.  I’ll be sure to write more here if I do any further work with this study.

I think that my colleagues, family, and friends are surprised that I’m not more celebratory about finishing my doctorate.  The dissertation itself  – conceptualization, collection of data, analysis, and writing – was pretty easy for me and it doesn’t feel much different from other studies I’ve completed.  But the emotional drain of living with this immense self-imposed and emotionally puzzling weight for so long was so soul-sucking that I’m  more relieved than happy or excited to finally be done.  I’ll try to learn to celebrate later but for now I’m enjoying just living without the shame and embarrassment I’ve hidden from everyone for several years.

Now that I’m done, I can begin to chip away at my large backlog of video games.  I’ve tackled the problem of non-response bias on a Web-based survey but now I’m going to save humanity from aliens.


Abstract

Higher education scholars, policy makers, and administrators know little about the experiences of undergraduate students who matriculate with minimal experience with technology. It is often assumed that all students, particularly traditionally-aged students, have significant experience with, knowledge of, and comfort with technology. Although that assumption is correct for many students, it is false for others. Despite the enormous increase in the use of Web-based assessment surveys and the increasing importance of accurate assessment and accountability data, those efforts may not be collecting adequate and accurate data about and from all students.

This study explores the non-response bias of first-year undergraduate students on a self-administered Web-based survey. First, data were collected with a supplemental survey added to the Beginning College Survey of Student Engagement (BCSSE). K-means clustering was used with this newly constructed Internet Access and Use survey to classify students according to their Internet access and use experiences. Second, demographic data from BCSSE and the Internet access and use data were included in a logistic regression predicting response to the subsequent National Survey of Student Engagement (NSSE).

The Internet Access and Use instrument proved to be a viable way to classify students along lines of their previous Internet access and use experiences. However, that classification played no meaningful role predicting whether students had completed NSSE. Indeed, despite its statistical significance the final logistic regression model using provided little meaningful predictive power.

Generalizing the results of this study to all Web-based surveys of undergraduate college students with random or census sampling indicates that those surveys may not introduce significant non-response bias for students who have had less access to the Internet. This is particularly important since that population is already vulnerable in many ways as being disproportionately composed of first-generation students, underrepresented minority students, and students with lower socioeconomic statuses. This reassures assessment professionals and all higher education stakeholders that cost- and labor-efficient Web-based surveys are capable of collecting data that do not omit the voices of these students.

Dissertation Journal: Less Time and More Pressure Makes Kevin a Productive Boy

Although I have not finished my dissertation, I began a full-time job a little over a month ago.  I know that this is a dangerous move and that many people who leave school before completing their dissertation never complete it.  I also know that even in the best circumstances this will delay my progress.  This is a move motivated by the reality of five years of graduate student pay and loans, however, not by academic concerns.

So far this is working out well.  For over a year, I was stalled and made no progress at all.  I was paralyzed by indecision and fear and always eager to find other interesting and worthwhile projects.  I was also very good at dodging or redirecting questions from friends and colleagues.  But I knew that I wouldn’t be able to dodge questions from potential employers so I had to buckle down and get back on task – I didn’t have a choice.  Backing myself into a corner seems to have been the right choice as it forced me into action.

As I entered the job market, I began writing again so I could honestly tell interviewers that I was making substantive progress.  Even then I wasn’t writing as much and as often as I should have been doing.  Once I had a job offer, however, I knew that my days as a full-time student with lots of discretionary time were quickly coming to an end.  I finally got off of my ass and wrote with the effort and work ethic that I should have employed a year ago so I could finish my first three chapters and submit them to my chair.  I knew that for about two months I wouldn’t have any time to devote to my dissertation so I did as much as I could before moving and starting a new job.  I finished new drafts of my chapters and submitted them to my chair the day before I began packing up and moving to Delaware.  It was a huge relief to have made substantial progress so I could move with a clear conscience and start a new job without this looming over me.

As I have settled into my new job, I have learned that I have been extraordinarily lucky by landing a job where my supervisor, director, and colleagues are extremely supportive of me completing this terminal degree  When I was offered this job, I wanted to negotiate a pay raise dependent on completion of my dissertation to incentivize it.  That wasn’t possible as my supervisor negotiated the highest pay she could get for me regardless of my doctorate or lack thereof.  But my supervisor wants me to finish my doctorate for my own benefit; when we discussed my goals for the year, she asked me to place this at the top of the list.  Today, she asked if I would like to carve some time out of my work schedule to work on my dissertation on a regular basis as a form of professional development.  I couldn’t ask for more and I now feel a responsibility to justify the support I have been given.

I was also very fortunate in that one of my faculty members reached out to me to offer advice about completing the dissertation while working full-time but I will post that advice in a separate post because it may be more interesting to a larger audience than news about my personal journey.

Dissertation Journal: Chapter 2 Draft

It’s taken me about a year to reach this point but I finally completed and submitted to my chair a draft of chapter 2, the literature review.  It’s pretty solid but it’s still a draft.

  1. There are a handful of places that I know I could expand but I’ll wait until I hear back from my chair before doing that.
  2. I need to add a couple of new sources to the digital divide section but nothing significant and nothing worth delaying this draft any longer.
  3. I think that I need to add a brief section – to this chapter or another one – summarizing the assumptions of the study.  Several of them are spread throughout the lit review where I discuss and justify them but it seems that it would be more organized if I would summarize them all in one place.

Why did this take so long?  I don’t know.  It’s certainly not because I struggle with this kind of writing.  It’s definitely not because I don’t know what I want or need to write.  I imagine that it has to be some kind of emotional block, some sort of fear of failure perhaps.  That is completely uncharacteristic of me but it’s the only thing that makes sense.

Why did I finally buckle down and get this draft completed?  The shame of not having done this yet became overwhelming and I simply had to finish this so I could look my chair in the eye.  I am also running out of time; I’m already behind where I wanted to be right now and hitting the job market ABD.  I’m not very pleased to be on the market as an ABD but now my energy needs to be focused on making as much progress as possible before I land a job because the more progress I have the better the odds that I’ll finish.

Dissertation Journal: Progress – Finally!

Initial outline of chapters one and twoI am having a very hard time keeping myself on track with my dissertation. I’m having trouble maintaining focus. First, there are so many other interesting and worthwhile things to do that I keep getting distracted. I am trying to keep the rest of my research agenda afloat and other projects on track. I am trying to stay somewhat visible and active within my professional communities.  And I am still working with faculty developers to gain experience in that field and some of that work is very interesting and extremely satisfying (I am really excited to be working with our History Learning Project as a survey and assessment consultant!). Second, there seem to always be personal issues that threaten to distract or even overwhelm me, whether they be as complex as intensely personal issues inappropriate for this blog or simple as a dying car that needs to be nursed along until I can replace it. All of this has pushed me off-course quite a bit and I’m now several months behind my self-imposed schedule.

But I’m finally making progress! I sent to my chair-to-be a nearly-complete draft of chapter 3 (Word document with many comments) and he seemed to like it. It’s incomplete but I wanted to get feedback on the structure and any glaring omissions or problems. Nothing major was found lacking and that is very heartening. It was also encouraging that my chair-to-be agreed with many of my comments and observations.

I’m setting this aside for now to back up and work on chapters 1 and 2*. I’ve been told that it’s good to have chapters in different stages of work as you’re initially writing and editing them so when you get burned out on one you can move on to something different and still be productive. I am very grateful to Meredith Adams for sharing her dissertation as it’s really helping me structure the first two chapters because our topics are very similar. Based on her dissertation, previous drafts of my lit review, and a lot of thought and consideration, I’ve decided to use this as a an initial guide to the structure of the first two chapters. I have much of this already written in other documents, particularly my draft lit review, so I hope I’ll be able to get these two chapters drafted quickly.

If you’re reading this blog, you might also be interested in knowing that I’ve been accepted to the Oxford Internet Institute’s Summer Doctoral Programme. It will cost a pretty penny to attend the program but it should be more than worth it.

 

* – In a typical social science dissertation (such as this one), chapter one is the initial introduction that argues for the importance of the study.  Chapter two is the literature review where you convince readers that you are an expert on all of the topics in your study.  Chapter three is the methodology where you describe how you have studied the problem.  Chapter three seemed to be the easiest place for me to start as it’s the most well-defined and limited in scope.

Dissertation Journal: Quals, Candidacy, and Data

Last week, I defended my qualifying exam and was advanced to candidacy. Today, I finished creating, checking, and uploading reports and data files for the 11 institutions who administered my survey to their students. So now I’m officially done with everything except my dissertation proper. I plan on spending the long weekend holiday (Thursday is Thanksgiving here in the U.S. so I’ll have Thursday and Friday off) working on my proposal so I can get a few drafts and outlines of chapters to my chair so I can move towards defended my proposal.

Thank you, Wen, for checking the numbers in my reports! You’re the best!

Dissertation Journal: Data and Distractions

Unless there are some surprises in store for me, I have all of the data for the first “half” of my dissertation. Before I can organize and analyze it for my own purposes, however, I want to get it organized and analyzed for the institutions that so graciously asked their students to participate in this survey. I’ve had a lot of fun creating all of the (SPSS and Excel) macros that generated the necessary reports and data files. Once someone else double-checks a few of the reports by independently generating and checking the numbers, I’ll send them out to the institutions so I can move on.

But moving on will be tricky, at least for the next couple of weeks. As much as I’d like to dive in to the proposal and related data analysis (which will be crucial to the chapter about methodology seeing as how these data will determine part of my methodology; more about that later), I have quite a few distractions. In the next three weeks I will be:

  • Presenting a keynote address at a state financial aid administrator conference. I’ll be talking about networking with colleagues in the 21st century (i.e. using social media and other technologies). I’ll post some notes about this soon because preparing for this talk has helped me crystallize some of my thoughts and ideas in this area, particularly the role of intentionality and goals in effectively using technology to communicate and network.
  • Leading a concurrent session at the same conference. It will be a follow-up to the keynote but focusing on networking and communicating with students. I think this session will have a different flavor compared to the talk because this is an area where there is more solid research and data.
  • Defending my qualifying exam. This should be straight forward and low key judging by the feedback I received on my written responses. The primary challenge is that this is sandwiched between two conferences so I’ll have to work hard to maintain my focus and give this the attention it deserves.
  • Presenting a paper at a national conference. Luckily, the conference is in my backyard (Indianapolis) and I’m already very familiar with the topic. We still have to finish the paper, however, and nail down the details. Thanks goodness it’s a roundtable presentation so (a) we have had more time to write the paper and (b) it will be low key and low stress.

All of these things are worthwhile and arguably necessary. But their timing is awkward and I can already see my hoped-for dissertation proposal defense slowly moving further and further away. Hopefully I can hit it hard once I’m through with these major commitments. A turkey-fueled Thanksgiving write-a-thon may be in my future!

Dissertation Journal: Scanning Survey Instruments

Last week I finally began scanning survey instruments to turn pencil and pen marks into data. Using the setup in the photo, I’ve scanned most of the surveys. The scanner I’m using is quite old but it’s a beast that still scans very quickly. It even scans surveys printed on longer sheets of paper and on both sides; it’s what is used to scan paper versions of NSSE and BCSSE. And the fine folks in IU Center for Survey Research (CSR) are experts in this entire process so they know the hardware, software, and how it’s all used and they’re guiding me with patience and professionalism.

Verifying the surveys – systematically double-checking responses the software thinks are missing and resolving ambiguous situations – takes much longer than scanning them. I can only spend two afternoons a week scanning or verifying but I hope to have this set of surveys done so I can send data back to participating institutions very soon.

I’m still worried that there may not be much variance in the data. It sure seems like that as I glance at each survey while I’m batching, scanning, and verifying them. But that might be selection bias so I just have to wait until I have data in a usable format.

It was also interesting to learn that one of the steps of the whole scanning-surveys-to-import-data process doesn’t work like I thought it did. In fact, the step – “monitoring” – doesn’t take place at all. I thought that there was a step after verifying where some survey instruments (1 out of every 10) were compared to the extracted data. Apparently CSR no longer monitors scans as the scanning process is so accurate that as long as the verification process is carried out accurately the data are accurate. In fact, monitoring not only makes the entire process longer but it may even introduce more error than it reduces. I thought that monitoring was an important part of the process but I trust the experts in CSR and their guidance.

Finally, I really need to get back to working on my dissertation proposal. I’m having trouble getting myself into a productive routine. I hoped that my time would be more open now that I’m done with coursework but it seems like I have even more demands on my time. I’m pulling back from some things (I recently resigned from the ResNet Applied Research Group and I stepped down from my regional NASPA leadership position, for example) but I still feel it necessary to continue some non-dissertation-related activities. I need to find a better balance. Or move to a planet with longer days and more hours to get things done.

Dissertation Journal: From Surveys to Data

Completed surveys from two of the eleven institutions participating in the first wave of data collection have arrived.  Now I’m working with my colleagues in IU’s Center for Survey Research (CSR) to transform these from a stack of completed surveys into an SPSS data file.  One of my colleagues in CSR likened this process to alchemy and I think he’s right!

One of the final steps in creating my survey instrument was to send it down to CSR for them to review and reformat it so their scanners can read it.  The main part of that process involved setting up their scanning program to read this instrument.  Not only did they have to indicate where to look for responses but also what the responses mean (i.e. a mark in this specific area is response number 3 to question 1).  This also involves telling the program how to record the responses (i.e. response number 3 to question 1 generates a value of “4” for the “compuse” variable).  As can be surmised from the previous example, this setup process also includes naming and defining all of the variables that will eventually end up in the SPSS file.

Just as interesting and important as the automated processes are the manual processes that must be created, documented, and enacted.  Most of these are quality assurance or error checking processes.  For example, after a batch of surveys is scanned someone must manually review the places where the program is unsure (i.e. a large checkmark that spans multiple response boxes) or the response was too faint for the scanner to properly record (all “missing” values are checked to ensure they are actually missing and not a scanning error).  There are also a few points in the process where results are manually double-checked to provide quality assurance.

When the instruments are scanned, the data are inserted into a database.  Then the data have to be extracted from the database and inserted into an SPSS file.  Once the SPSS template is created (and checked and double-checked), inserting the data is fairly trivial.  It can get a bit tricky, however, if you’re merging in data from other sources.  In this instance, we’re merging the results from this survey with the results from these students’ BCSSE surveys but I’ll do that on the back end using SPSS instead of doing it on the front end with a database query; that will make it easier for me to merge these data into the institution-specific data files we return to participating institutions.  It’s also something I can do myself which gives me more control over and understanding of things (I don’t touch the database; that is all CSR).

There are a lot of small details not described in the above overview and I’m really enjoying learning about this entire process.  It’s nice that my survey is a relatively small one: ~1600 one-page instruments.  That allows me to be very hands-on which (a) ensures that I understand the whole process and (b) saves me money because I don’t have to pay someone else to do these things.

There are still some unanswered questions, mostly those surrounding what to return to participating institutions and when to do so.  I wish I had an answer to some of those questions but I don’t.  Part of this is caused by the fact that I spend almost all of my time working on NSSE or occasionally FSSE where data are collected and reports generated at pre-determined and coordinated times.  BCSSE, on the other hand, uses a rolling schedule where we generate reports and return data to institutions as we receive their data.  That might not sound like a big difference but it’s not just a different process but a different mindset, one I had not fully anticipated or appreciated.

Finally, it’s tremendously exciting to finally see data!  We’re going through several test runs to ensure everything is set up properly and I understand how everything works.  I’ve been able to glance at a handful of surveys during testing but it was finally real to me when I received the first (test) SPSS file with MY data from MY survey instrument.  It sounds silly to admit that a screenful of numbers is exciting and even exhilarating but it’s true.  I have quite a ways to go but through the haze I’ve glimpsed the light at the end of the tunnel reflecting off something shiny in the far, far distance.

http://www.indiana.edu/~csr/

Dissertation Journal: Updated Synopsis and Other Updates

I’m a few weeks away from my qualifying exam so I took an opportunity to update my dissertation synopsis so my advisor can use it to write my second question.  It includes my final survey instrument and a few notes related to the pilot administration.

In the meantime, I wait for surveys to be administered and returned.  I’ve mailed about 5,500 surveys to 11 institutions.  I was very fortunate in being able to get a fairly diverse group of institutions to participate.  However, I’m a bit nervous that a handful of institutions that have not yet committed to participating in NSSE in 2011 may decide (or be forced) to not participate.  That wouldn’t kill my study but it might require significant changes.