Last week, I ceased piloting my survey instrument and finalized it.Â I submitted it to IRB as an amendment to BCSSE at the same time as sending it to my colleagues in IU’s Center for Survey Research who are formatting it for their scanners.Â IRB approved the amendment and I will immediately begin soliciting institutions to participate.
The pilot did not go nearly as well as planned.Â I was hoping to conduct 5 cognitive interviews and a pilot administration with about 50 students.Â I feel far short of both of those goals.Â I think that the biggest factor that contributed to this failure is the timing: I was so late in soliciting participants that we were nearly in finals and that is a bad time to get students to do anything.Â I also limited my recruiting to one (large) residence hall in the belief that my requests were so simple and easy that I wouldn’t have to recruit in other places.Â Finally, I think the advertisements were too plain and too prominently focused on the Internet, perhaps causing students to believe that I was recruiting only computer experts or geeks.
Despite the dismal participation in my interviews and pilot, I am pressing ahead.Â First, my timeline does not have any slack time.Â Second, the data I was able to collect was all very, very positive in terms of the construction of my survey instrument.Â Finally, even if this stage in designing this new survey instrument did not go as well as hoped, all of the preceding stages went very well and were performed very thoroughly.Â I take some reassurance from the idea that all of the stages that were within my control were done well and thoroughly (I hear my dad’s voice echoing in my head: “Control the controllables”), in part because there are some stages that I can not control.Â I will, however, conduct more cognitive interviews this summer even as my instrument is in the field; it would be too late to make changes if I were to discover any problems but (a) I do not expect to discover any problems and (b) if there are remaining problems then I need to be aware of them.
So now my focus turns to recruiting institutions to participate.Â Ideally, all participating institutions would meet several criteria:
- Registered for the paper version of BCSSE
- Administering BCSSE in on-campus events (orientation sessions, FYE classes, etc.)
- Intending to participate in NSSE next spring
- Possessing a diverse student body
The first criteria is non-negotiable as it is critical to my research design.Â More than any other criteria, this one immediately and dramatically narrows the pool of potential participating institutions.Â The second criteria is desirable as those institutions meeting it should be those institutions that have the highest response rates.Â The third criteria is also non-negotiable as it is critical to my research design.Â The fourth criteria is important because if I have a homogeneous sample – especially an affluent one – then I may not have enough variation to perform some of the statistical tests I would like to perform (in statistical terms I would not have enough “power” for operations such as logistic regression).
Getting all of these criteria to align – including the unlisted logistical one of “we haven’t mailed their BCSSE surveys yet” – is challenging.Â In fact, I worry about the fourth one considerably.Â I can’t change the BCSSE registrants to make them meet my criteria but I am placing a small safeguard in my study regarding the fourth criteria: I will be recruiting a few institutions who have diverse student bodies even though they may not be participating in NSSE next spring.Â I will not be able to analyze NSSE non-response data from those institutions but by collecting demographic and Internet access information I will at least be able to compare their incoming student bodies with those of the other institutions in my sample.Â If there are no important differences then I know that I’ll have nothing to worry about with respect to diverse student populations.Â If there are important differences then either I will be able to explore it in my study or note it for future studies.
Since I am pursuing an opt-in strategy in recruiting institutions (i.e. I am asking if they want to participate and they must say “yes”), there is still potential for this study to fall apart if institutions ignore my requests or say “no.”Â I was hoping to use an opt-out strategy (i.e. telling institutions that they will be participating unless they say “no”) but some of my colleagues are uncomfortable with that strategy.Â In any case, this recruitment is another critical link in the chain that is this study.Â From my viewpoint, in the middle of this mess and uncertain how or if it will all turn out, it seems like yet another fragile link in a long chain of fragile links.