Parametrizing evolution acceptance
The purpose of this study was to examine the measurement properties of the items contained in three quantitative evolution acceptance instruments, each of which captures evolution acceptance differently in terms of dimensionality and context. The MATE was designed as a unidimensional measure which used macroevolution and human evolution contexts in its items (Rutledge and Warden 1999). The I-SEA was developed to capture these contexts in separate constructs; it was designed to measure acceptance of microevolution, macroevolution, and human evolution as three respective dimensions (Nadelson and Southerland 2012). The GAENE was designed to provide a unidimensional measure of evolution acceptance which is independent of knowledge of evolution and religious orientation (Smith et al. 2016). Given the high similarity of wording between many of the items on the three instruments and the fact that they share a common Likert measurement scheme, we hypothesized that these instruments may share more similarities than differences and actually provide similar quantitative information about evolution acceptance. We found that this is the case. Putting the 57 items from the three instruments together to form an instrument-independent scaling results in useful unidimensional and two-dimensional parametrizations of evolution acceptance.
Given the two parametrizations, a big-picture question that comes forth immediately is: should evolution acceptance be treated unidimensionally as it has in the vast majority of quantitative work on evolution acceptance, or should it be treated as a two-dimensional construct which is apparently delineated by the positive or negative orientation of the wording of the items (also found in a recent study by Romine et al. 2017)? Our position is that a unidimensional usage of these items may provide a useful measure for evolution acceptance. Evidence for this includes: (1) the high variance of the first dimension in the EFA (28.9 items of variance out of 57 total items) in comparison to the other dimensions (Fig. 1), (2) the closeness of the first eigenvalue from PCA on Rasch residuals to 2 for a unidimensional construction (2.11 items of variance), and (3) the moderate relationship between acceptance of truth and rejection of incredible ideas (r = 0.73). In fact Metzger et al. (2018) found that much of the apparent two-dimensionality of the MATE found by Romine et al. (2017) disappears in more advanced science students. However, deciding not to recognize evolution acceptance as a two-dimensional construct may also result in a missed opportunity—CFA and Rasch analysis demonstrate that a unidimensional construction does not capture the entire story regarding how these three instruments measure evolution acceptance.
Our analyses collectively (Tables 1, 2, 3) suggest that observing a conceptual distinction between positively- and negatively-worded items is empirically useful, and we argue that recognizing the distinction between acceptance of truth and rejection of incredible ideas is supported by conceptual work around evolution acceptance (Deniz et al. 2008; Ha et al. 2012). For example, Deniz et al. (2008) propose a model in which evolution acceptance is constructed by an integration of cognitive, affective, and contextual factors. Further, they nest evolution acceptance within the theory of conceptual change (Strike and Posner 1992) in their suggestion that acceptance of evolution is necessary in order to find new ideas about evolution plausible and fruitful (Deniz et al. 2008). Our findings in this study not only support this assertion, but also allow us to expand upon it in proposing that the dimensionality by item valence aligns with specific elements of conceptual change theory. The theory of conceptual change is rooted in the idea that students construct new ideas through agreement based on their existing knowledge, experience, and social interactions (Posner et al. 1982). This implies that different individuals may interpret evolutionary phenomena in different ways based on their prior knowledge, metaphysical beliefs, and sociocultural background (Hewson 1981). It follows that one mechanism for learning involves agreement to abandon previously-held ideas and replace them with scientifically acceptable ideas (Posner et al. 1982). For a learner to agree to do this, he/she must find dissatisfaction in a previously-held idea and henceforth find the new idea more intelligible, plausible, and fruitful than the previously-held idea (Posner et al. 1982; Treagust and Duit 2008). Our first sub-construct, acceptance of the truth of evolution, aligns with the intelligibility and plausibility conditions of conceptual change, similar to that suggested by Deniz and colleagues. A student’s measure for rejection of incredible ideas about evolution aligns directly with a student’s dissatisfaction with non-scientific ideas and in turn the likelihood that he/she will find a scientifically-acceptable idea fruitful. Rejection of incredible ideas therefore more directly influences the likelihood that instruction will be successful in facilitating conceptual change. Although we did not measure knowledge of evolution in this study, previous work (Romine et al. 2017) supports this assertion in its finding that the correlation of acceptance of the credibility of evolution (aligned closely with rejection of incredible ideas in this study) with knowledge of macroevolution was significantly higher than the correlation between acceptance of facts about evolution (aligned closely with acceptance of truth in this study) with knowledge of macroevolution.
We also find that the two-dimensional parametrization aligns well with the more recent model of Ha et al. (2012), which describes evolution acceptance as manifesting through both conscious and unconscious neurological pathways. Our acceptance of truth dimension aligns most closely with the unconscious pathway which is based upon a student’s intuitive feeling of certainty. For example, evolution is a good explanation for how humans first emerged on the earth (from the GAENE) solicits an appraisal based on one’s feeling of certainty built from extracurricular experience, and not necessarily analysis of the credibility of an idea based on logic. On the other hand, the rejection of incredible ideas dimension aligns most closely with the student’s conscious, reflective thinking supported by his/her process of understanding and logical reasoning around conceptual ideas underlying evolutionary theory. For example, expressing disagreement with the statement, species were created to be perfectly suited to their environment, so they do not change (from the I-SEA) requires a student to first consider the specific idea and then decide to reject that idea. Although the decision to reject an idea is affective in nature, it is in itself a comparatively logical and reflective process.
In addition to understanding how our current measures of evolution acceptance fit with conceptual work, this study also yields insight into the empirical implications for parametrizing evolution acceptance. Various parametrizations have been explored. Nadelson and Southerland (2012) utilized a three-dimensional model for evolution acceptance in their construction of the I-SEA, suggesting that the dimensions of evolution acceptance should be delineated by the type of evolution: microevolution, microevolution, and human evolution. What this study suggests is that, from a quantitative perspective, topic is a determinant of the difficulty of an item along the same sub-construct (Figs. 3 and 4), but it does not seem to serve as the key delimiter in terms of the unique sub-constructs. In other words, while acceptance of macroevolution, human evolution, and microevolution may be distinct in their difficulty, it may not be necessary to treat them as distinct sub-constructs. Rather, the data show that differences between students’ responses on items across contexts are accounted for by the expected difficulty hierarchy imposed by the Rasch model (Boone 2016), making it unnecessary to define new sub-constructs to account for the different response patterns across contexts.
Previous literature supports the idea that acceptance of evolution varies depending on context (i.e. Scott 2005; Nadelson and Hardy 2015). Figures 3 and 4 suggest that this comprises a progression of acceptance similar in structure to what we have seen in research applying Rasch models to learning progressions (Romine et al. 2016; Todd et al. 2017; Todd and Romine 2016). From Figs. 3 and 4, it appears that undergraduate students look at evolution acceptance as the act of accepting that evolution is true and rejecting incorrect ideas, and that within these constructs, students tend to progress from accepting evolution on short time scales, then on long time scales, and then ultimately that evolution happens in humans.
An additional motivation for constructing the I-SEA by the topics of microevolution, macroevolution, and human evolution is that students who are reluctant to accept that organisms, including humans, evolve on long time scales may nonetheless accept that organisms evolve on short, observable time scales. The Wright map for the unidimensional construction (Fig. 2) shows that the items addressing human evolution are indeed the most difficult. However, the items at the bottom of the scale are macroevolution items, and as we go up the scale, we see a mix of macroevolution and microevolution items. Much of this muddiness clears up when we model the positively and negatively worded items as separate sub-constructs (Figs. 3, 4). A two-dimensional parametrization gives the expected result that the items addressing evolution over short times scales are easiest for students to accept, those addressing evolution over long time scales are more difficult for students to accept, and those addressing human evolution are most difficult for students to accept. We see this as evidence that treating these instruments, and evolution acceptance in general, two-dimensionally gives a more realistic account of how college students think as they decide whether or not to accept evolution.
Previous work with the MATE also supports the idea that evolution acceptance should be treated two-dimensionally. Despite the fact that the MATE has mostly been used as a unidimensional instrument, an analysis by Wagler and Wagler (2013) suggests that the MATE behaves multidimensionally, and that it can be difficult to predict which items load on which dimensions as participants and contexts change. Romine et al. (2017), on the other hand, suggest that the behavior of the MATE is quite predictable in undergraduate students, and put forth a two-dimensional parametrization for the MATE that is similar to that discovered independently in this study. The only item result that was not replicated is Item 5 on the MATE (Cred5). In our previous work, this was the only positively-worded item which loaded more strongly onto the acceptance of credibility than the acceptance of facts dimension. In this study, it loaded onto the acceptance of truth dimension, which is more in line with what we would expect given that it is a positively-worded item.
This multidimensionality of evolution acceptance as measured by the MATE and I-SEA, and lack of basis upon a predefined framework for evolution acceptance, were driving factors for construction of the GAENE, which its authors claim is a generalizable unidimensional measure of evolution acceptance (Smith et al. 2016). Our analysis suggests that the main factor making the GAENE unidimensional is that all of its items are positively-worded. This means that the GAENE measures acceptance of truth but not rejection of incredible ideas. With respect to its contribution to measurement of evolution acceptance, both Figs. 2 and 3 show that the spread of the difficulty of the items is greater than those of the other instruments, meaning that these items provide information about students of a greater ability range than the items from the MATE and I-SEA. While this is attractive from a Rasch modeling perspective (Boone 2016), we also observe that some of the more difficult items from the GAENE do not fit well with the Rasch model, meaning that even students with high levels of acceptance tend to express low levels of acceptance on these items. To follow, we discuss reasons for the misfit of these and other items, and what this may tell us about how to improve our measures of evolution acceptance.
Misfitting items and implications for improving current measures
When the items from these three instruments are treated unidimensionally, we observe the pattern that it is the negatively-worded items which tend to misfit with the Rasch model, and also tend to load positively onto the residual factor (Tables 1, 3). Nine of the 19 negatively-worded items exhibit some misfit (a mean squares index over 1.3) with the Rasch model in this case. These results collectively illustrate the multidimensionality of the negatively-worded items under the assumption of a unidimensional model for evolution acceptance. It makes sense that the Rasch model would tend to model the positively-worded items more faithfully, and thus identify negatively-worded items as anomalous, since the 38 positively-worded items double the 19 items which are negatively-worded.
This apparent bias disappears when positively- and negatively-worded items are treated as separate dimensions as suggested by the EFA, making it straightforward to explore more precisely potential issues with the wording of particular items. On the acceptance of truth dimension, three items (GAENE1, GAENE6, and GAENE8) exhibited a mean squares fit index above 1.50, which is indicative that these items may be harmful to the validity of the scale as it is defined by the corpus of 38 items in this construct (Wright et al. 1994). It happens that these items are also of moderate-to-high difficulty. This suggests a tendency for higher accepting students to express lower levels of acceptance on these items despite expressing high levels of acceptance on the other items measuring this construct. GAENE8, the most difficult item (Fig. 3), states: nothing in biology makes sense without evolution. From an expert biologist’s perspective, evolution may serve as the centerpiece for understanding biology, but a college student who accepts evolution may feel that topics like anatomy and physiology, microbiology, and cell biology can be understood adequately without first understanding evolution. The context is not there since evolution is often not covered in college courses addressing these subjects.
GAENE1, everyone should understand evolution, sits at the middle of the scale (Fig. 3). Along the logic of GAENE8, the misfit of GAENE1 is likely caused by responses from students who accept evolution, but nonetheless do not view it as a necessity for engaging in other courses of study or for advancing one’s quality of life. GAENE6 sits at the top of the Rasch difficulty scale along with GAENE8 (Fig. 3). This item states: I would be willing to argue in favor of evolution in a public forum such as a school club, church group, or meeting of public school parents. Fear of public speaking is quite common (Harris et al. 2002), and it is straightforward to argue that one can display acceptance of evolution without extraversion. In addition to evolution acceptance this item measures willingness to engage in public discourse; the latter having little to do with evolution acceptance.
Two items within the rejection of incredible ideas construct misfit with the Rasch model: MATEcred17 and MATEcred19. These items were also of relatively high difficulty, indicating that students who tended to reject the other misconception statements nonetheless tended to accept these. Students may have accepted MATEcred17, much of the scientific community doubts if evolution occurs, since the term “scientific community” is quite broad. A student who has taken a variety of science classes has likely seen that evolution is not discussed in most of these classes—especially the physical sciences, but even in biology (Padian 2010). It would therefore make sense for students to accept this statement regardless of their tendency to reject other ideas which are not credible. The misfit of MATEcred19, with few exceptions, organisms came into existence about the same time, is more difficult to explain. We argue that the rejection of incredible ideas construct is comparatively aligned with one’s understanding of evolution (cognition), so the tendency for high ability students to accept this idea may be reflective of the persistent misunderstanding of deep time and the evolutionary timeline—one of the most difficult concepts for students to comprehend (Rosengren et al. 2012). Common practice may be to simply eliminate these items from the corpus in future research. However, doing this would carry the consequence of eliminating a potentially important part of the construct. We caution against making assessment decisions based on a single number. Instead the nature of the specific construct of interest should also be considered when deciding which items to use from these three instruments.
Suggestions for using current evolution acceptance instrumentation
A key question that arises from the above discussion is, how should the MATE, I-SEA, and GAENE be used in research? Much of the previous discourse around these instruments has centered on the question: which instrument works best? The I-SEA was developed to make our measures of evolution acceptance more fine-grained (Nadelson and Southerland 2012). A key motivation for developing the GAENE was to overcome the epistemological inadequacies of the MATE and I-SEA (Smith et al. 2016). Although we do not find anything inherently wrong with using a particular instrument, our data indicate that loyalty to a particular instrument is unnecessary. We instead suggest a reframing of the dialogue toward a discussion of the merits and limitations of particular items instead of the instruments themselves. For example, if an evaluation project calls for a focus on human evolution or macroevolution, then we suggest that relevant items from both the MATE and the I-SEA could be combined into a single scale. On a different project, a researcher may wish to obtain a measure that is less context dependent and more based on an individual student’s understanding of what constitutes the theory. In this case, the pool of items may be drawn primarily from the GAENE along with the more context-independent items from the I-SEA and MATE. We would, however, recommend that researchers using the GAENE consider removing the items asking students to “…argue in favor of evolution…” unless improving a student’s extraversion is an important part of the evaluation plan.