Observing populations and testing predictions about genetic drift in a computer simulation improves college students’ conceptual understanding

Genetic Drift and Bottlenecked Ferrets module

The SimBio Virtual Labs^® module Genetic Drift and Bottlenecked Ferrets (Herron et al. 2014) is a learning module built around a series of interactive simulations. Instructions, tables for recording data, and questions are in an accompanying workbook. The module is designed to demonstrate and explore the causes and consequences of genetic drift, including conservation implications, using the example of endangered black-footed ferrets (Mustela nigripes). The interactive simulation models a population of ferrets that vary in coat color, a fictitious single-locus trait with two selectively neutral alleles.

The module has four exercises that guide students through simulations to explore (1) the relationship between sampling error and the founder effect; (2) the gene pool as a source of sampling error; (3) the effect of population size on the magnitude of changes in allele frequency and the likelihood of an allele becoming fixed in a population; and (4) the factors contributing to inbreeding depression. Throughout the module, students are asked to make predictions, record data and observations from repeated runs of the simulations, draw inferences from their observations and data, and construct explanations to explain why their predictions were or were not supported. In a final exercise, students are asked to apply their knowledge on an open-ended challenge of creating a ferret reintroduction plan that is likely to maintain the genetic diversity of the newly founded wild population. They can design reserves of various sizes, with or without connecting corridors, and run simulations to assess their designs using ferrets of known genotypes from a zoo. Readers wishing to see the Genetic Drift module can request a review copy from info@simbio.com.

Genetic drift inventory (GeDI)

The GeDI is a concept inventory designed to measure undergraduate students’ understanding of key concepts in genetic drift and to diagnose misconceptions around that topic (Table 1; Price et al. 2014). The test has 22 true/false items, phrased as agree/disagree, that relate to vignettes describing scenarios in which genetic drift took place. For example, an item asking about the target concept that “The processes leading to genetic drift tend to cause a loss of genetic variation within populations over many generations” (Price et al. 2014: 71) asks students to evaluate a vignette and then to indicate whether a biologist would agree or disagree with the statement that “The island population likely has fewer alleles—that is versions of genes—than the mainland population” (Supplementary Materials in Price et al. 2014). Seven of the items target key concepts of genetic drift, while the remaining 15 items target misconceptions (Table 1).

Treatments

We recruited courses for the two treatments separately, calling on our network of colleagues who have some familiarity with the literature on evolution education for the control treatments or our colleagues who already use the SimBio Virtual Labs^® for the module courses. In the control treatment, students received traditional instruction (control courses); in the experimental treatment, students completed the Ferrets module (module courses) in addition to traditional instruction. For both treatments, students completed the GeDI (Price et al. 2014) before instruction on genetic drift began, and completed it again after instruction ended. We excluded students who did not complete both the pretest and the posttest, students who took the assessment three times, and courses that allowed students to work in groups on the test. In all of the module courses and three of the five control courses, the version of the GeDI used in this study was altered slightly from the one published by Price et al. (2014) by adding the phrase “State whether you agree or disagree” to Stem E; the course in the control group from the Research University, Midwest and the Moderate Research University, West were the only courses that used the original wording. Permission to use data from students was granted by Institution Review Board Approval 42,505 from the University of Washington, New England Independent Review Board Protocol 14–131, and California State University, Fullerton Institution Review Board Approval 13_0473. All students in the study consented to participate and consented to have de-identified data published.

Data analysis

We calculated item difficulty on the pretest across both treatments by dividing the number of correct responses for each item by the total number of responses for that item (Crocker and Algina 1986) in order to compare our populations with those reported previously (Price et al. 2014). We used generalized linear mixed models (GLMM), using the glmer function from the lme4 package (Bates et al. 2014) in R 3.1.3 (R Core Team 2014) for the rest of our analysis.

Next, we compared how students performed on the posttest with a model that included Pretest score (fixed), Treatment (fixed), Course (random) and OLRE (random) as predictor variables (Table 4). Because each item in the GeDI targets either a single key concept or a single misconception, we used the same model to compare how students performed between treatments on the items in the GeDI that targeted key concepts essential to understanding genetic drift and items that targeted misconceptions about genetic drift (Table 1). We used Holm-Bonferroni corrections to account for the fact that we conducted multiple analyses of the same sets of data (Holm 1979), resulting in an alpha of 0.025 for the first comparison and an alpha of 0.05 for the second comparison of posttest performance.

We conducted additional analyses to investigate sources of variation between control and module courses that could have affected the outcome of our main analysis. Because the form of credit (completion vs. correctness on GeDI posttest) varied among the module courses, but not the control courses, we constructed two additional GLMMs. The first compared posttest performance between module courses in which students received credit for completion of the GeDI (8 courses) to module courses in which students received credit for correctness on the GeDI (11 courses), controlling for course differences. We found no significant difference in posttest performance. We then re-ran the full model but excluded those module courses that gave credit for correctness on the GeDI. There were no differences in the results of that analysis when compared to the analysis of the full dataset, so we continued with the full model.

We had relatively large differences in mean class size between the control (mean 63, SD 57) and module (mean 27, SD 13) courses. We investigated the impact of class size on our results in two ways. First, we constructed an additional GLMM that included only the module courses with class sizes of 30 or greater. This reduced the number of module courses to eight, and narrowed the difference in mean class size between the treatment groups. The results were largely similar to the analysis with the full dataset, with the exception that the effect of Treatment on the performance of students on questions about misconceptions, which was small in the GLMM using the entire data set (p = 0.018; Table 4), was no longer significant (p = 0.065). Second, we divided the sample of module classes in half to compare the class size of the nine lowest performing classes to the ten highest performing classes, which were 26.8 and 26.9 respectively. Taken together, the results of these two tests suggest that class size is not responsible for the differences we found between treatments, so we continued our analysis with all of the classes.

We used Cohen’s d, a standardized measure of the differences between the means, to calculate the strength of the effect of instruction within each courses, as well as the effect of treatment across courses (Cohen 1998; Sullivan and Feinn 2012). Cohen’s d, in a sense, is a measure of the degree of overlap between the comparison groups. A Cohen’s d of 0 indicates complete overlap between treatments (i.e., the module students performed equivalently to control students), while a Cohen’s d of 3 would indicate that all members of one treatment scored above the mean of the other (i.e., module students far outperformed control students). The larger the absolute value of Cohen’s d, the less overlap between groups, and the stronger the effect of treatment. However, a negative Cohen’s d would indicate a change in the opposite direction as a positive Cohen’s d.

We calculated effect size in two ways: by looking at the effect of instruction on GeDI performance within each course and by looking at the effect of treatment on change in performance on the GeDI. To determine the effect of instruction within each course, we calculated the Cohen’s d [(average posttest − average pretest)/(pooled SD_{pre- and posttest}]. We used Cohen’s d, instead of normalized learning gains, because the calculation for normalized learning gains does not account for the students who have perfect scores on both pre- and posttest, nor students whose scores decrease on the posttest (Miller et al. 2010). We then averaged the Cohen’s d across all courses in each treatment to estimate the average effect of instruction within each treatment.

We then estimated the strength of effect of treatment on student learning by calculating Cohen’s d from the average change in test scores across treatments [(average change in score_module − average change in score_control)/pooled SD of change in score_{all courses}]. This approach allowed us to account for the change in student performance from pretest to posttest for students when estimating the magnitude of effect of treatment on student performance on the GeDI.

To compare pre- and posttest scores visually, we calculated the proportion of items about key concepts and misconceptions that students answered correctly in each class. We then graphed the average proportion of correct answers by treatment. To highlight how students in the two treatments differed on each item of the GeDI, we calculated the change in percent correct for each item by course, and then graphed the averages of those differences by treatment.

Students’ performance on the GeDI before instruction

Students in both treatments performed similarly on the GeDI prior to instruction (Fig. 1; Table 4). The mean proportion of correct responses on the GeDI pretest across all students in the five control courses and 19 modules courses was 0.58 (SD 0.09), which falls on the low end of the range of GeDI performance reported earlier (Price et al. 2014). Item difficulty ranged from 0.30 to 0.80, closely matching the range of difficulty seen during GeDI development (Price et al. 2014).

Effect of treatment on students’ performance

Students who completed the Ferrets module showed significant increases in performance on the GeDI after instruction. The mean proportion of correct responses on the GeDI increased from 0.60 (SE 0.02) to 0.70 (SE 0.03). This increase is due to improved performance on items about both key concepts and misconceptions (Fig. 1). Furthermore, students who completed the Ferrets module significantly outperformed (p < 0.001) those in the control courses when accounting for pretest scores and random effects associated with each course (Figs. 1, 2; Table 4). This outperformance occurred both on items about key concepts (p < 0.001) and on items about misconceptions (p < 0.05). In the control courses, mean performance on items about misconceptions improved from 0.49 (SE 0.03) to 0.56 (SE 0.03) (Fig. 1). However, the mean performance on items about the key concepts was significantly worse after instruction, falling from 0.62 (SE 0.04) to 0.43 (SE 0.11) (Fig. 1). Together this indicates that mean student performance in the control group was poorer on the GeDI after instruction, dropping from 0.53 (SE 0.03) to 0.51 (SE 0.03) (Fig. 1).

For control courses, the average Cohen’s d was −0.04 (SD 0.2); the negative value indicates that, on average, the pretest scores were slightly higher than posttest scores (Additional file 1: Table S1). Therefore, instruction in control courses had a minimal impact on student performance. In comparison, the average Cohen’s d across courses in the module treatment was 0.63 (SD 0.59), demonstrating that instruction with the module had a sizable positive effect on student performance (Additional file 1: Table S1). Moreover, the effect size of treatment on the change of scores was quite large: 1.63 (Cohen’s d). This result indicates a strong positive effect of the Ferrets module on student performance relative to control course students.

The improvement of students in the module courses was not limited to content explicitly covered in the Ferrets module. Key concepts 4B and 4C (Fig. 2a; Table 1) are not incorporated into the Ferrets module workbook or simulations. However, students in the module courses improved on items aligned with those concepts, while fewer students in control courses answered those items correctly. A similar pattern occurred with items about misconceptions 7, 8, and 10; these are not covered in the Ferrets module (Table 1), but students in the module courses improved their performance on most of these items, and outperformed students in control courses on many of them (Fig. 2c; Table 1).

Limitations of the experimental design

It is surprising that the scores of students in the control courses dropped after instruction. Here we consider aspects of our experimental design that may have contributed to this finding. Ultimately, we conclude that these aspects did not bias our results toward this unusual discovery.

Performance in the control courses

Although performance among students in the control and Ferrets module courses did not differ significantly on the pretest, it did differ between treatments on the posttest (Figs. 1, 2; Table 4). Students in the module courses improved on the posttest, but students in the control courses performed significantly worse on the posttest because their performance on items about key concepts decreased (Fig. 1). Students in the control courses showed a decrease in their understanding of three of the key concepts associated with genetic drift (as defined by Price et al. 2014; Table 1; Fig. 2a): (1) that genetic drift can lead to a loss in genetic variation, (2) that the effect that drift can have is governed by a population’s effective population size, and (3) that genetic drift works simultaneously with—and can overwhelm—other evolutionary processes. However, students in the control classes increased their performance on the key concept that “random sampling error happens every generation” (Price et al. 2014: 71).

We know of only one other study that looked specifically at students’ understanding of genetic drift before and after instruction (Andrews et al. 2012). In that study, introductory students answered an open-ended question that required them to consider whether genetic drift could explain a shift in genotype. In the pretest, students referred to genetic drift infrequently: only 1 % of the 85 students referred to genetic drift; even within this 1 %, their comments were so vague that they could not be evaluated. After instruction, 21 of the 122 students referred to genetic drift, but only 13 indicated some knowledge of what genetic drift actually does (Table 2 in Andrews et al. 2012).

Andrews et al. (2012) used their results to propose a framework that describes how students acquire knowledge about genetic drift through three stages, one of which is learning to recognize genetic drift as distinct from other evolutionary processes, such as natural selection, mutation, and migration (Stage 2 in Fig. 3). For example, Item 6 on the GeDI asks students whether a biologist would agree or disagree with the (incorrect) statement that “The fact that individuals that were best suited to the environment had a higher rate of survival contributed to genetic drift” (Supplementary Materials in Price et al. 2014). In our study, we find a big increase in performance on this item for students in both the control and module treatments (Fig. 2b), indicating that they are in Stage 2. Because it is so challenging for students to recognize that evolution encompasses more than natural selection (Price and Perez 2016), the fact that the students in the control courses are making this change is noteworthy.

We postulate that students in the control courses are still in Stage 2, because, even though they recognize the existence of different evolutionary processes, they continue to be confused by the distinctions between them, and they are often distracted by vocabulary. For example, students might confuse genetic drift with gene flow, perhaps because they confound the word drift with the idea of migration (Andrews et al. 2012); on Item 18 of the GeDI, students are asked whether a biologist would agree or disagree with the (incorrect) statement that “Since there was no migration there could be no genetic drift” (Supplementary Materials in Price et al. 2014). Although students in the module courses improved on this item, performance on it did not change after instruction among the students in the control classes (Fig. 2c). This indicates that some confusion over vocabulary persists through instruction.

Students in the control group may be performing worse on the items about key concepts because their understanding of genetic drift is only just developing; inaccuracies are possibly being incorporated into or already existing in their conceptual frameworks (Stage 3 in Andrews et al. 2012). The items in the GeDI about key concepts predominantly focus on how genetic drift works and the effect that it has on a population. For example, Item 1 in the GeDI asks students whether a biologist would agree or disagree with the (correct) statement that “Genetic drift is more pronounced in the [founding] island population than the [larger] mainland population in these first few generations” (Supplementary Materials in Price et al. 2014). We suggest that what students typically learn during instruction is that genetic drift has a powerful effect in founding populations. This focus on small populations, however, can lead to an incorrect conclusion that genetic drift occurs only in small populations, and students often fail to recognize that drift occurs in all real, finite populations. The fact that a misconception like this could emerge from instruction may be a natural consequence of students making sense of new ideas. Indeed, it is unlikely that students think about the situations in which genetic drift occurs before they fully understand what genetic drift is.

Price et al. (2014) suggest that items about key concepts are less difficult for students than items about misconceptions. In this study, the opposite pattern appears to hold true. One key difference between our study and theirs that might explain the opposing findings is that the testing in Price et al. (2014) was completed before instruction. It may be that, for this challenging topic, misconceptions are most difficult prior to instruction, but they are nonetheless easier to dispel then key concepts are to acquire. Moreover, all of the courses used for final testing in Price et al. (2014) were upper division courses, in which students had previously been exposed to genetic drift. It is therefore conceivable that students were already in Stage 2 of the learning framework when they took the GeDI. The students in our control courses are primarily—but not exclusively—in general biology courses (Table 2). Future work exploring the pre- and post-instruction difficulty of items about misconceptions and key concepts could help inform a new model of student learning in genetic drift.

Performance in the module courses

The effect of the Ferrets module on student learning was substantial for a short intervention (average Cohen’s d within module courses = 0.63). Students in the Ferrets module courses significantly outperformed students in the control courses because they did better on items about key concepts, suggesting that they had a better understanding of genetic drift (Figs. 1, 2; Table 4). Their performance even improved on items about key concepts and misconceptions that were not explicitly covered in the Ferrets module, generally to a greater degree than did control course students (Fig. 2c).

The Ferrets module was designed to engage students by guiding them through the construction of their own concept of genetic drift by making observations, collecting data, and making and testing predictions. While this approach is not unique to the Ferrets module, the combination of observations and experimentation with simulations is what makes the module treatment different from the controls. Our experimental design does not allow us to determine how much each of the individual practices contributed to learning. Instead, we can only offer the evidence that the multi-faceted approach to instruction in the Ferrets module supported learning better than classroom instruction alone.

Although we cannot attribute the gains in GeDI scores to specific elements of the Ferrets module, we suggest that computer simulations support learning for topics such as genetic drift. Simulations allow students to investigate population-level phenomena that span generations, like genetic drift, which are otherwise not amenable to investigation in the classroom given the time and spatial scales involved. The visualizations available in the Ferrets module enable students to observe and experiment with several aspects of drift, including the random changes in allele frequency due to sampling error, that these changes occur every generation, and that drift occurs in populations of any finite size. In support of the suggestion that visualization may help teach random processes, Meir et al. (2005) demonstrated that simulations of osmosis and diffusion decreased misconceptions about those molecular phenomena because they allowed students to directly observe the random movement of molecules. Within the Ferrets module, students can set parameters such as population size and initial allele frequency; repeatedly test the effect of varying these parameters; and make and test predictions. Since drift is a phenomenon where the starting conditions impact the outcome in a probabilistic way, repeated testing and varying of parameters may help students build understanding in a way that is difficult to do with reading, lecture, and static representations. As suggested by Windschitl and Andre (1998), simulations that allow for exploration can be effective tools to overcome misconceptions and effect conceptual change. Separating the impacts of computer-based simulations and experimentation would be a topic for future research.

Some intriguing aspects of students’ performance suggest that learning during this activity is particularly sophisticated. In the learning framework hypothesized by Andrews et al. (2012), students begin to recognize different mechanisms of evolution (Stage 2) before they learn content specific to genetic drift (Stage 3); this is the pattern that we observed in students in the control courses. Students in the module courses were increasing their understanding of both vocabulary and genetic drift during the module (Figs. 1, 2; Table 4). Students in the module courses also improved on both key concepts and misconceptions, including some that were not directly addressed in the Ferrets module’s instructions or simulations (Table 2; Fig. 2). This approach was clearly effective.

Revised hypothetical framework for learning genetic drift

As described above, Andrews et al. (2012) hypothesized three stages for learning genetic drift: (1) undeveloped concepts of evolution and genetics at the broadest level, (2) undeveloped and overlapping concepts of different evolutionary mechanisms, and (3) developing understanding about genetic drift in particular. Our results lead us to suggest to revise this framework to incorporate multiple learning pathways, rather than a linear progression through the stages (Fig. 3). The results from students in the module classes suggest that students can move from Stage 1 to either Stage 2 or Stage 3, or to both Stage 2 and Stage 3 simultaneously.

We note that Stage 1 and Stage 2 are about students’ general understanding of evolution, not specifically genetic drift. Our interpretation is that, when students, such as those in the control courses, are moving through Stage 2, they are actually expanding what they know about evolution by recognizing that many different mechanisms of evolution exist. This realization in itself is quite challenging (Price and Perez 2016). Thus, there is misalignment between what students are learning and what instructors intend to be teaching. Instructors think they are teaching genetic drift, but student thinking is revolutionized by a more basic concept that there is more to evolution than natural selection.