Survey of Undergraduate Research Experiences (SURE): First Findings

Attempts to determine an empirically established set of benefits generated by undergraduate research experiences in the sciences are fairly recent (Lopatto, 2003a; Seymour et al., 2004). Lopatto surveyed science undergraduates at four liberal arts colleges over a period of three summers. The quantitative results of these surveys fit well with qualitative data drawn from student interviews at the same four institutions by Seymour et al. (2004). Having grounded the benefits of undergraduate research in both quantitative and qualitative data in these pilot studies, it remains for researchers to establish that the findings apply to a broader range of institutions. The opportunity for a more extended study was presented by the Howard Hughes Medical Institute (HHMI), which funds grant activity for undergraduate science education at a variety of institutions.

The undergraduate research experience is widely touted as an effective educational tool for enhancing the undergraduate experience (Mogk, 1993; Tomovic, 1994) with multiple benefits (Lopatto, 2003a), the most instrumental of which is an increased interest in a career in the science, technology, engineering, and mathematics workforce (Fitzsimmons et al., 1990; Zydney et al., 2002). Undergraduate research experiences are associated with increased persistence in pursuit of an undergraduate degree (Nagda et al., 1998); increased levels of pursuit of graduate education (Hathaway et al., 2002; Kremer and Bringle, 1990); and alumni retrospective reports of higher gains than comparison groups in skills such as carrying out research, acquiring information, and speaking effectively (Bauer and Bennett, 2003). Several studies have supported the hypothesis that undergraduate research helps promote career pathways for members of underrepresented groups by increasing the retention rate of minority undergraduates (Nagda, et al., 1998) and increasing the rate of graduate education in minority students (Hathaway et al., 2002).

This study was motivated by three research questions, primarily concerning but not limited to HHMI-funded undergraduate research experiences in the sciences. These questions are: 1) Is the educational experience of undergraduates being enhanced? 2) Are undergraduate research programs attracting and supporting talented students interested in a career involving scientific research? 3) Are undergraduate research programs retaining minority students in the “pathway” to a scientific career?

In terms of this survey, question 1 was answered by investigating both the general response to the undergraduate research experience and the specific gains reported on 20 potential benefits. Question 2 was examined by asking students their plans for postgraduate education. Are students in undergraduate research programs declaring that they intend to continue their science education or to seek a science-related career? Furthermore, can we identify differences in the experiences of those students who intend to continue in science compared with those students who assert that they have lost interest in science careers? Question 3 was examined by looking for differences among ethnic groups in proportion to each group that intends to continue in science. Included in this examination were differences between women and men because women are an underrepresented group in some sciences.

METHOD

Description of the Respondents

The survey was completed by 1,135 undergraduates representing 41 universities and colleges in late summer and fall of 2003. A preliminary survey of HHMI program directors at participating institutions indicated that a total of 1,526 students participate in summer undergraduate research programs. Thus, the overall response rate was 74%. The 41 institutions included 19 universities (Carnegie Classification Doctoral/Research Universities—Extensive), 15 colleges (14 Carnegie Classification Baccalaureate College—Liberal Arts, 1 Carnegie Classification Baccalaureate College—General), and 7 master's level institutions (Carnegie Classification Master's College and Universities I). Demographic characteristics of the respondents are given in Table 1.

As is true of every tabular or statistical presentation in this report, there are missing cases. Students might have failed to indicate their institution, declined to specify personal characteristics, or left an evaluative question unanswered because it did not apply to them. Table 1 shows the categories of institutional type, sex, and ethnicity. Overall, including cases missing from Table 1, 258 respondents attended a college, 59 respondents attended a master's university, and 787 respondents attended a research university. Thirty-one respondents did not specify their institution. Analysis of the data in Table 1 indicates that the distribution of men and women is uniform across institutional types. The distribution of men and women is also uniform across ethnic categories. The distribution of ethnic categories across institutional types is nearly uniform. A statistical analysis of ethnicity by institutional type yields a significant chi square (χ² = 34.4; df = 12; p, .05), with higher than expected frequencies of Asian American and Hispanic students at Research Universities.

Respondents were asked to cite the source of the funding that supported their research. Fifty-five percent of the respondents reported being funded by HHMI programs. Local university or college funding accounted for 12% of the respondent support. Other sources of funding were each less than 10% of the total. Non-HHMI-funded students were eligible to take part in the survey because institutions with HHMI grant programs often fund other students for similar activities in the same time period, so it was deemed sensible to invite these students to participate.

Approximately 59% of the respondents are women. Table 2 shows the research fields of the respondents crossed with the sex of the respondent. As might be expected from national trends, women outnumber men in biology, chemistry, and biochemistry, but men outnumber women in physics, mathematics, computer science, and engineering.

About 48% of the respondents reported their graduation year as 2003 or 2004. Juniors graduating in 2005 made up 32.6% of the total, whereas sophomores made up 16.3%. The remaining 2.7% reported their graduation year as 2007, implying that they were entering first-year students. Specific ethnic categories conformed to this pattern, with the exception of Hispanic respondents, who had a lower proportion of rising seniors (34.5%) and a higher proportion of rising juniors (40%). Only about one-third (35%) of the entire group reported no prior experience in undergraduate research. Older students tended to have more prior experience than younger students.

The Survey

The Survey of Undergraduate Research Experiences (SURE) consisted of 44 items, including demographic variables, learning gains, and evaluation of aspects of summer programs. Items regarding learning gains were suggested by previous survey research. Before the main survey was conducted, a brief checklist survey was sent to HHMI program directors. A list of learning gains concepts was presented, and program directors were asked to report whether each was primarily taught explicitly (e.g., in a seminar or workshop) or implicitly (e.g., a behavior modeled by the student's supervisor).

The SURE survey was located online on server at Washington University in St. Louis, MO.

Procedure

Notices of survey availability were sent to each program director. Participating program directors (PDs) were asked to specify the number of students from their school eligible to take the survey and the date on which they would be asked to do so.

The target date was immediately after the end-of-program symposium or other“ summing up” activity. Two weeks after that date, PDs were informed how many students had participated, giving the PDs the option to contact their students to remind them to participate in the survey. Students were provided with a name and password for access to the survey. Within the survey, students identified their school and provided demographic information, but anonymity was maintained. Student names were collected for a raffle that awarded gift certificates to the winners, but the names were separated from the survey material. Students answered items on the survey by either selecting from a pull-down menu or choosing a number on a rating scale. A “no answer” option was available. At the end of the survey, students were provided with a text box for written comments, which were directed to the PD for that institution by e-mail. After the site closed, all PDs received the aggregate results for their school.

RESULTS

Interesting Science Careers

Students reported their plans for postsecondary education. Their responses are categorized in Table 3. Most of the respondents had some plan for further education; the leading categories are medical school (22%) and biology related (20%). Respondents were asked how their research experience influenced their plans for postgraduate education. These responses are presented in Table 4. Of those who responded, almost 91% reported that their research experience sustained or increased their interest in postgraduate education. Only 4.7% reacted to their undergraduate research experience by changing their plans away from postgraduate education.

Reported Learning Gains and Overall Evaluation of the Undergraduate Research Experience

To evaluate the educational experience of undergraduate researchers, 20 evaluative questions on specific learning gains, drawn from previous research, were presented to the respondents. Table 5 summarizes the results by listing the means for each set of responses. These means are listed in descending order. The highest rated item is “Understanding of the research process in your field” (X = 4.13), followed by “Readiness for more demanding research” (X = 4.03), “Understanding how scientists work on real problems” (X = 4.0), and “Learning laboratory techniques” (X = 4.0). The lowest rated item is “Learning ethical conduct in your field” (X = 3.15), followed by “Skill in science writing” (X = 3.32), “Skill in how to give an effective oral presentation” (X = 3.42), and“ Clarification of a career path” (X̄= 3.42). A measure of interitem consistency, Cronbach's alpha, on these 20 items is .92, indicating a high degree of consistency. Students who reported being funded by HHMI also reported higher means on all 20 items compared with the overall survey means (Table 5).

In addition to the 20 specific learning gains, respondents were asked to evaluate five general questions about their experience. Student attitudes toward their research supervisor, peers, and overall sense of summer research as a learning experience were decisively positive. The frequency distributions of responses to these questions are found in Table 6. The responses reveal a high degree of satisfaction with the undergraduate research experience. Eighty-seven percent of the respondents rated their experience as good or better than they expected. Seventy-eight percent of the respondents rated their supervisors as above average or outstanding, an impressive finding. The supervisors included faculty, postdoctoral students, and graduate students. Sixty-three percent of the students rated their experience with other students in a positive way; 19% of the respondents did not answer this question, presumably because they did not work with other undergraduates. Eighty-four percent of the respondents indicated they were likely or very likely to choose another research experience if they could, and a similar percentage gave a positive overall evaluation of their summer research as a learning experience. These five overall evaluative questions are correlated with each other (Table 7). For example, the overall sense of summer research, perhaps the most global of these questions, is significantly related to prior expectations (r_s = .47), to supervisor ratings (r_s = .39), to student ratings (r_s = .20), and to choosing another research experience (r_s = .36). Thus, the responses are generally consistent.

To investigate the question regarding the attraction of talented undergraduates to science, the data were analyzed for students who reported positive or negative experiences. One variable for such exploration concerns the influence of the research experience on plans for postgraduate education given that one of the goals of undergraduate research is to promote graduate education in science. As described earlier, the undergraduate research experience sustained or confirmed the student's plans for graduate education. Over 83% of the students continued to plan for postgraduate education in the sciences following their undergraduate research experience. Two groups of students did change their plans (see Table 4). Forty students initially had no plans for postgraduate education, but the research experience changed their minds. Fifty-one students initially had plans for postgraduate education, but their research experience convinced them otherwise. In what ways did their experiences differ? To find out, the 20 learning gains questions were analyzed. When the mean scores for these two subgroups are inspected, the students who now plan to further their education have the highest means of any group displayed in Table 5 on 18 of the 20 items. In contrast, the respondents who decided not to pursue further educations have the nominally lowest means of any group displayed in Table 5 on 15 of the 20 items. When the two groups are directly compared to each other via independent group t-tests, 13 significant differences emerge at the a = .005 level, all in the direction of higher learning gains by the group that is now planning graduate education.

The two groups that were influenced by summer research to change their plans for postgraduate education also differed on some of the overall evaluation questions. An inferential statistic, the Mann-Whitney U-test (evaluated at a = .01), was employed to analyze the data. Students who changed their plans in favor of graduate education had higher ratings about summer research meeting or exceeding their expectations, evaluated their supervisors more favorably, were more likely to choose another experience, and had a more positive overall sense of summer research than students who changed their plans to forego graduate education. The two groups did not differ in their evaluation of working with other students. The two groups did not differ in the type of institution at which they had their experience. Although the results can be interpreted in various ways, the SURE survey nevertheless appears to capture the different experiences of students who become attracted to, compared with those who turn away from, science careers.

Differences between Women and Men

Women constituted almost 60% of the sample of respondents. As noted in Table 2, women are the majority in the biological sciences, whereas men are the majority in the physical sciences, math, computer science, and engineering. This distinction is repeated when students were asked about their postgraduate education plans. A higher percentage of women than of men plan to continue in biology (21.9% of all women vs. 19.3% of all men) and medicine (24.7% of all women vs. 21.1% of all men), whereas a lower percentage of women than of men plan to continue with a Ph.D. in the physical sciences (9.5% of all women vs. 21.6% of all men). Women and men did not differ overall in prior experience, in the influence of the research experience on their future plans, in their evaluation of their supervisor, or in whether they would choose another research experience. Women reported higher gains than men on 14 of the 20 learning items. These results, however, are entangled with institution (there are two women's colleges in the group), research area (biological vs. physical science), and previous experience indicating that women have a positive response bias when compared with men.

On overall evaluation questions, women reported higher ratings than men on two items. When asked to compare their experience to their prior expectations about the experience, more women than men rated the research experience as“ Much better than I expected” (35.5% of women vs. 26.2% of men;χ ² =13.7; p, .05). Women also rated their experience with other students more favorably than men did (45.1% of women vs. 37.6% of men rated working with other students as “One of the best parts of the summer research experience” (χ² = 14.0; p, .05).

Differences among Ethnic Groups

To address the issue of retention of minority students in a pathway to a scientific career, the reported experience of minority students was analyzed. Nearly half of the respondents in the sample were not members of the Caucasian majority. The better represented minority groups include Asian-American (14.9%) and African-American (10.0%). Hispanic students made up 4.9% of the total, whereas foreign national students constituted 6.1%. The memberships of some ethnic groups are so small (e.g., 1 Native American) that statistical comparisons are not meaningful.

Ethnic groups did not differ in their distribution of women and men, with women constituting the majority in every ethnic group. The research fields chosen by each group were dominated by the biological sciences (Table 8), as might be expected because the major source of student funding in this study was from the HHMI. Ethnic groups did not differ in prior experience. Asian-Americans showed relatively less interest in postgraduate education in biology-related Ph.D. programs (15%) and relatively more interest in medical school (35%) than comparison groups (see Table 9). There were no statistically significant differences among ethnic groups in the influence their summer research experience had on their postgraduate plans (Table 10), an indication that the undergraduate research experiences are retaining minority students in the pathway to a science career as well as they retain Caucasian students.

An analysis of the five general satisfaction questions revealed no differences among ethnic groups in their expectations of summer research being met, their evaluation of their supervisors, their evaluation of their student coworkers, their inclination to have another research experience, or their overall sense of research as a learning experience. Analysis of the 20 learning gains questions indicates that ethnic groups statistically differed on three items (“Learning ethical conduct”, “Skill in oral presentations”, and “Becoming part of a learning community”), however, with Hispanic students reporting scores that were higher than at least one other group.

Institutional Type

The means and standard deviations for each institutional type for the 20 learning gains items are presented in Table 11. Although some ratings of the institutions differ significantly on some items, the pattern of the differences is difficult to determine. Analyzing differences in student learning with reference to Carnegie Classification categories does not reveal clear differences among institutions.

DISCUSSION

This study provides some answers to the three research questions raised initially. First, the educational experience of undergraduates is enhanced, both in terms of their general satisfaction and in terms of learning gains on 20 specific items. Of these items, those related to the research process, scientific problems, and lab techniques are rated as the highest gains. These are followed by gains in personal development (such as tolerance for obstacles and working independently). Institutional type does not determine the level of enhancement. Second, the 41 institutions represented in the data are attracting and supporting students interested in a career in science, as represented by their plans for postgraduate education in science. The majority of the student respondents planned to continue in the sciences, and health-related education, including medical school, was also a popular choice. The undergraduate research experience sustained the science career plans of a large majority of the students. A few students (about 4%) no longer planned for further science education; about the same number were inspired to change their plans toward further science education. Finally, the patterns of postgraduate plans were similar across gender and ethnic groups. Over half the students responding to the survey are women; almost half the students are from minority groups. The undergraduate research programs are providing a pathway to a scientific career for minority students, and the data indicate that most of these students intend to continue on this path.

The present findings replicate pilot work on assessing undergraduate research (Lopatto, 2003b; Seymour et al., 2004). Lopatto surveyed 384 science undergraduate researchers working on summer research programs at four liberal arts colleges. As in this study, students rated their gains on potential benefits on a scale of 1 (no gain) to 5 (very large gain). The results showed that “Learning laboratory techniques” and “Understanding the research process” were among the highest rated benefits. Personal development items, including “Readiness for more demanding research” and“ Tolerance for obstacles,” were also rated highly. The overall pattern of rated benefits was similar to that seen in this study. Seymour and her colleagues (2004) analyzed the results of 76 interviews with students participating in undergraduate research experiences at the same four sites examined by Lopatto (2003b). A qualitative analysis, analyzing the text of the transcripted interviews, revealed the same pattern of benefits reported by students as are seen in the results presented here.

This study does not include a control group and does not sort out all the influences, including precollege career intentions, general academic achievement, and family influence, that might account for the choices students make to pursue careers in science. A control group is selected with the goal of creating a comparison group that is like the treatment group in all characteristics except the treatment. This goal is elusive in the in situ study of undergraduate students. Undergraduate research programs are affected by both institutional selection procedures, which filter the best applicants for research positions, and self-selection by students motivated to explore science. An attempt to create a quasi-control group, such as a group composed of students who applied for research positions but were turned down, does not escape the criticism that the control subjects were not identical to the treatment subjects before the undergraduate research experience commenced. Differential academic credentials or interpersonal skills might have sorted the undergraduate researchers and unselected students into nonequivalent groups. An attempt to create a true control group by traditional means, such as random assignment, would be unlikely to succeed in the face of ethical concerns regarding withholding education from talented students and regarding traditional student autonomy in choosing courses of study.

Should a reasonable control group be provided for the study of the undergraduate research experience, a second obstacle renders comparisons between treatment and control groups difficult. Undergraduate research experiences contain features that are not exclusive; laboratory courses,“ research-like” experiences and other pedagogies can share some of the features of undergraduate research. The presence of non-exclusive features means that treatment and control subjects are likely to differ in amount of treatment rather than to differ by the presence or absence of treatment. Furthermore, the undergraduate research experience is a molar treatment (Campbell, 1986), consisting of an array of treatment components, including the mentoring skill of the supervisor, the social interactions of the working group, the sophistication of the instrumentation related to the research experience, and so on. Therefore, this study cannot meaningfully compare the learning gains of students who experienced undergraduate research relative to those who did not, except for the small group that left the science career pathway as a result of their experience. The results do show, however, that most respondents report an enhanced educational experience through a variety of learning gains. The experience sustains the interest of many students already planning science careers and attracts some students who did not previously plan a science career. These outcomes are consistent across gender and ethnic group.

Further information is needed to support these findings. The current research includes a 9-month follow-up survey, currently in progress, that will contribute more information to the robustness of student plans and to the specific enhancement of later educational experiences following the summer research experience. Ideally, a further research effort should include tracking the cohort of student respondents to determine how many continued on pathways to science careers.

FOOTNOTES

ACKNOWLEDGMENTS