A closer look at the items within three measures of evolution acceptance: analysis of the MATE, I-SEA, and GAENE as a single corpus of items

Validated measures of evolution acceptance

Both qualitative and quantitative methods have been utilized to understand evolution acceptance. Qualitative methods include open response (Nehm and Schonfeld 2007; Robbins and Roy 2007) and interview protocols (Donnelly et al. 2009; Nehm and Reilly 2007). In this study, we focus on quantitative measurement of evolution acceptance, which has been pursued with significant interest for over two decades. Johnson and Peeples (1987) represented one of the first attempts to measure evolution acceptance using a survey with documented psychometric information. Just over a decade later, the Measure of Acceptance of the Theory of Evolution (MATE) was developed (Rutledge and Warden 1999).

The MATE contains 20 total items—10 positively-worded and 10 negatively-worded—which focus primarily on acceptance of evolution over long time scales including common ancestry and human evolution. The MATE has been both utilized and criticized extensively since its development (for a review, see Romine et al. 2017). The MATE has been used as a unidimensional measure throughout its history. By “unidimensional”, we are referring to the idea that a single measure or score is sufficient to describe the relationships between students’ responses (Kline 2014). This statistical argument is extended to the qualitative interpretation that the MATE measures acceptance of evolution as a single construct. Wagler and Wagler (2013) provided evidence that the MATE is not unidimensional, and further exhibits dynamic dimensionality across different groups, meaning that the MATE measures different constructs within different populations. Using Rasch analysis, Romine et al. (2017) corroborated the claim of multidimensionality in a sample consisting primarily of science non-majors, finding that the MATE works better as a two-dimensional instrument. This means that two scores or measures are needed to capture the relationships between undergraduate students’ responses on the items. In Romine and colleagues’ study, the MATE was found to measure the following constructs: (1) acceptance of facts and (2) acceptance of credibility of statements related to evolution. This dimensionality was delineated by whether or not the items were negatively worded.

The Inventory of Student Evolution Acceptance (I-SEA) and the Evolution Attitudes and Literacy Survey (EALS) are also constructed multidimensionally, meaning that multiple quantitative measures are used to account for students’ observed responses (Kline 2014). The I-SEA (Nadelson and Southerland 2012) was designed to improve on the MATE by disentangling microevolution, macroevolution, and human evolution contexts. This is an important contribution given that microevolutionary events may be easier for students to accept than those related to macro- and human evolution (Alters and Alters 2001; Scott 2005). Given these goals, the I-SEA consists of 24 total items, 9 of which are negatively worded, with 8 items assigned to one of three subscales for constructs defining acceptance of microevolution, macroevolution, and human evolution, respectively. The authors hypothesized that the items related to acceptance of microevolution would be easier than the items measuring acceptance of macroevolution and human evolution, and their analysis bears this out (Nadelson and Southerland 2012). Here, it may be useful to draw a distinction between Nadelson and Southerland’s (2012) decision to use 3 dimensions to explain the differences in items, including their difficulty, versus letting the items take a difficulty hierarchy along a single dimension as is often done in Rasch studies (Boone et al. 2013). Nadelson and Southerland show that the 3-dimensional model explains students’ responses adequately; however, the efficacy of the comparatively parsimonious approach of defining the items hierarchically along a single unidimensional Rasch scale remains unexplored.

The Evolution Attitudes and Literacy Survey (EALS) is a 104-item instrument designed to measure 16 lower-order and 6 higher-order constructs which have been known to influence acceptance of evolution. Some of these include creationist reasoning, political/religious conservativism, knowledge, misconceptions, and exposure to evolution (Hawley et al. 2011). Given the length of the EALS, a study to shorten it was undertaken, and a short form (EALS-SF) was thereby developed which the authors claim retains the psychometric integrity of the original (Short and Hawley 2012). The approach taken by the EALS contrasts with other existing instruments which ask students about their acceptance of evolution directly.

The Generalized Acceptance of Evolution Evaluation (GAENE) is the most recent development in the line of evolution acceptance instrumentation. It is claimed to measure evolution acceptance unidimensionally without the potential confounds with knowledge of evolution and religion (Smith et al. 2016). The GAENE consists of 13 items, all of which are positively-worded. Unlike the MATE and the I-SEA, the items on the GAENE have less focus on specific evolutionary contexts. Rather, the items tend to provide more general opinion-related statements including whether one accepts that there is evidence for evolution, whether evolution is important in the study of biology, and whether one is willing to advocate for evolution. Like the MATE, the GAENE has been validated using Rasch methods.

Lack of alignment with theoretical understanding of evolution acceptance

Although current evolution acceptance instruments generate measures which have shown to be useful within specific research studies, we find that these instruments do not address adequately current theoretical perspectives around evolution acceptance. The model put forth by Deniz et al. (2008) considers evolution thinking in terms of cognitive, affective, and contextual domains. Their model conceptualizes evolution acceptance as a product of both cognition and affect around the theory of evolution. While the model of Deniz et al. (2008) focuses on evolution thinking in general, the model of Ha et al. (2012) has a more constrained focus on evolution acceptance specifically, suggesting that this is mediated through both conscious and unconscious pathways. Foremost, people will make a conscious decision on whether or not they accept evolution through their understanding of evolution and their reasoning around what they perceive as evidence for or against evolution. Indeed, facilitation of conscious experiences is an important focus of evidence-based pedagogies focused on evolution (Romine and Todd 2017; Beard et al. 2014). Ha et al. (2012) further propose that these conscious experiences will merge with past experiences, values, and social interactions to generate unconscious feelings which manifest as an underlying feeling of certainty.

Significant work has been done towards exploring the relationship between knowledge of evolution and acceptance of evolutionary theory (Nadelson and Southerland 2010; Romine et al. 2017). However, it is currently difficult to tease out how much of this relationship is due to the fact that evolution acceptance as it is currently being measured reflects knowledge of evolution. The MATE has been criticized on the grounds that it possibly conflates knowledge of evolution with acceptance of evolutionary theory (Hogan 2000; Smith 2010; Wagler and Wagler 2013), and both the MATE and the I-SEA are criticized on the grounds that they are not tied to a theoretical foundation for evolution acceptance (Smith et al. 2016).

Purpose of the Research

Context

We administered the MATE, I-SEA, and GAENE 2.1, to 452 college students taking introductory biology classes at two doctoral universities with moderate research activity using an IRB-approved protocol. Of these students, 354 hailed from a university in the Midwestern United States, and 98 were attending a university in the Western United States. One hundred fifty-two students (34%) reported male gender, and 273 (60%) reported female. Two hundred sixty-nine students (60%) reported White ethnicity. However, other ethnicities were reported, including Black (52, 12%), Hispanic (53, 12%), and Asian (29, 7%). Eight students (2%) reported indigenous American heritage, and 25 students (6%) reported other ethnicities not included in the survey. We finally inquired into students’ religious preferences. Two hundred ninety-eight students (66%) reported Christian faith. Agnostic (28, 6%) and Atheist (23, 5%) preferences were reported by 51 students. Eighteen students (4%) reported Islamic faith, 8 (2%) were Buddhist, and 3 (1%) where Hindu. The remainder reported other faiths, or chose not to report.

Factor analysis

We used exploratory factor analysis (EFA) in SPSS 21.0 to reconcile how the items on these three instruments define latent dimensions related to evolution acceptance, and then followed Nadelson and Southerland (2012) in proceeding to use confirmatory factor analysis (CFA) to evaluate how well the hypothesized factor solution derived from EFA reproduces the relationships between the item responses. While the dimensionality of these individual instruments has been discussed and explored in previous work, we initially took the position in this study that we did not know the dimensionality of this collection of items when they are administered together, thereby warranting an a posteori approach for exploring dimensionality. This can be contrasted with the a priori, or hypothesis-driven, approach that involves study of the items through a pre-specified model. EFA is a variable-centered clustering technique where latent factors among observed variables are extracted mathematically from the data (Collins and Lanza 2013). EFA has been used extensively in science education research, particularly in the context of instrument validation (i.e. Romine et al. 2013; Corwin et al. 2015), and is a technique that is often associated with classical test validation methods and other situations where the researcher wishes to reduce a larger feature set to relatively few latent constructs or dimensions (Henson and Roberts 2006).

While methods from Rasch, Item Response Theory (IRT), and CFA traditions are confirmatory in nature, meaning that the measurement model is specified a priori, EFA is exploratory in nature, meaning that the measurement model is extracted mathematically from the data a posteori. EFA is therefore a useful tool when the researcher intends to extract a factor structure from the data mathematically without invoking prior information (Costello and Osborne 2005). Given its exploratory nature, a key challenge when conducting an EFA was deciding how many factors should be extracted from the data. Many different guidelines exist for making this decision which can result in different conclusions; we used the scree test (Cattell 1966) in combination with “reasoned reflection” (Henson and Roberts 2006, p. 399) based on the structure of the solution. Specifically, we sought a solution that both makes sense and yields dimensions which are conceptually separable, which is referred to as simple structure (Bryant and Yarnold 1995). After factor extraction, we utilized the promax rotation method (Hendrickson and White 1964) to diagonalize the factor solution. This oblique rotation method allowed the factors to be correlated, which assisted us in deriving a conceptually understandable and separable factor solution.

Although the scree criterion and the requirement of simple structure can provide evidence for the suitability of the factor solution derived from EFA, there are nonetheless multiple methods for extracting the optimal number of factors which often yield different conclusions (Fabrigar and Wegener 2011). Further, a decision regarding the extent to which a loading is high or low, important or ignorable, is subjective. Following Nadelson and Southerland (2012), we therefore used a CFA process to quantify the extent to which our factor solutions derived from EFA actually reproduced the data. Our models assumed no cross-loading between items and utilized the probit link to account for the categorical nature of the responses. Estimation was done using the diagonally weighted least squares (WLSMV) estimator in Mplus 7 (Muthén and Muthén 2012). Fit with the data was evaluated using the Root Mean Square Error of Approximation (RMSEA), Comparative Fit Index (CFI), and the Tucker-Lewis Index (TLI). An RMSEA close to 0.06 (Hu and Bentler 1999), but not above 0.07 (Steiger 2007) is indicative of acceptable fit. Acceptable fit is also indicated by a CFI and TLI above 0.9 (Bentler 1990; Bentler and Bonnet 1980).

Before proceeding to discuss the use of Rasch methods in modeling construct validity of the items, we would like to outline some limitations of EFA, and factor analysis in general. From the perspective of test validation, EFA is criticized based on its treatment of ordinal observations as linear measures and its tendency to select for items which are similar in difficulty (Wright 1996), which is contrary to the objective of more contemporary methodologies such as Rasch modeling (Wright 1996; Boone and Scantlebury 2006). Further, factor analysis is a data-fitting approach which can be dubious when validity of the assessment is in question—rather, a data-independent validity standard is needed (Boone and Scantlebury 2006). Given these criticisms, and its discord with the Rasch validation philosophy in particular, we do not use factor analysis as an end to validation, but instead as a starting point to inform us about how to proceed with Rasch modeling.

Rasch analysis

After making a reflective and informed decision on the number of factors to retain, and observing the specific items loading onto particular factors, our attention turned to construct validity of the individual items towards measurement of each dimension, which we call an acceptance sub-construct from here on. Given that a student’s appraisal of each item was rated on a Likert scale, we utilized the Rasch rating scale model (Andrich 1982) as a criterion for validity. Rasch models differ in philosophy from other models like IRT and classical test theory (CTT) in that Rasch provides a philosophical criterion for validity of the data as opposed to letting the model define the data. This practice of evaluating the quality of the data through a standard model is different from the statistical practice of fitting the model to the data, which the goal of IRT and CTT. Whether to use Rasch or IRT modeling for validation of tests and surveys has been debated contentiously for many years (Andrich 2004). This said, it is straightforward to argue that the Rasch approach better aligns with the laboratory practice of calibrating machines based on a fixed standard before their use. Further, use of a fixed standard better aligns with the scientific practice of falsification (Popper 1957; Lakatos 1976) than use of statistical data-fitting approaches. Beyond these epistemological advantages, the Rasch approach facilitates generation of item-independent estimates of acceptance and person-independent estimates of item difficulty (also true with IRT), generation of linear estimates, and the ability to map student and item measures along a common scale (Boone and Scantlebury 2006).

We used the WINSTEPS software package (Linacre 2006) to fit a unidimensional Rasch rating scale model to each dimension using the items within each acceptance sub-construct informed by the factor analyses. These provided a standard for the validity of the items and subscales. Conformity of the response patterns of each item to expected fit with the Rasch model was evaluated through mean squares fit. We used infit and outfit of the item response patterns with respect to what the model would predict based on the item’s difficulty. These measures of fit have expected values of 1, but Wright et al. (1994) suggest that values between 0.5 and 1.5 are indicative of potentially useful items. Other sources suggest a more conservative range of 0.7–1.3 for acceptable agreement with expected fit (Bond and Fox 2015). Values below 0.5 indicate that the item fits the model better than we would expect in real data, possibly indicating bias in the item that favors students with high acceptance (Masters 1988). With regards to measurement validity, misfitting items represent a significant validity concern as they indicate bias in favor of students with weak levels of acceptance which contradicts the intended scale directionality (Bond and Fox 2015). We evaluated items with respect to the mean squares fit thresholds of 1.3 and 1.5 as potentially indicative of moderate and high misfit, respectively, with Rasch model expectations.

In addition to item validity, we were also interested in the usefulness of the scale itself in explaining and quantifying students’ acceptance. Of primary concern was unidimensionality of the items in each acceptance sub-construct: the efficacy of the Rasch model in capturing the systematic variance in the item responses within each subscale. As with item fit, we evaluated unidimensionality using a falsificationist perspective. We first invoked the assumption that the items are unidimensional, and then attempted to falsify this by inspecting the residuals in the item responses with respect to the model. If the items measure a single dimension, then the residuals should be random, or homoscedastic. We implemented principal components analysis (PCA) on the residuals, and inspected the first eigenvalue. If the residuals are random, this eigenvalue should fall below 2 items of variance (Linacre and Tennant 2009). A first eigenvalue above 2 indicates some systematic departure from unidimensionality in the set of items with respect to the intended acceptance sub-construct and warrants exploration of which particular items are responsible for multidimensionality of the scale.

Factor structure of the item corpus

The first eigenvalue from the factor solution on 57 total items is 28.9 items of variance, which accounts for 50.8% of the total item variance. This large first eigenvalue provides a compelling argument that the collection of items from these three instruments may prove useful for providing a unidimensional measure for acceptance of evolution. However, the second eigenvalue accounts for 3.5 items of variance, an additional 6.2% of the total variance. After 2 dimensions, the eigenvalue of the subsequent dimension drops to 1.9 items of variance, for an additional 3.3% of variance. By Cattell’s (1966) scree criterion (scree plot in Fig. 1), this indicates that 2 dimensions may provide a more informative description of the structure of this corpus of items. CFA on the factor solution also bears this out. A 1-dimensional CFA model yields an RMSEA of 0.08, a CFI of 0.92, and a TLI of 0.91. This indicates moderate-to-good fit. However, a 2-dimensional CFA model with ignorable cross-loading between items significantly improves fit to the data (χ² = 98.2, df = 1, p ≪ 0.001), yielding an RMSEA of 0.064, a CFI of 0.95, and a TLI of 0.94. CFI and TLI values indicate that the 2-dimensional model fits well. That the RMSEA is below 0.07, also indicates acceptable fit. It is noteworthy that our RMSEA value exceeds that reported by Nadelson and Southerland (2012) for the I-SEA, and the CFI values in these respective studies match. These indices indicate collectively that a 2-dimensional factor structure is both necessary and sufficient to obtain acceptable fit with the data. If one adopts the 2-dimensional model, the correlation between these two acceptance sub-constructs is 0.73, indicating that they share 53% of their variance with each other. This demonstrates that these two sub-constructs share significant similarity, but nonetheless have important differences that need explaining.

The five items which load most highly onto the first dimension (loadings between 0.84 and 0.91) address acceptance of evolution as a mechanism for the origin of life as we know it today. Four out of these five items address human evolution specifically, meaning that a student’s score on this acceptance sub-construct may be influenced most highly by their acceptance of human evolution. For example, item GAENE11 is the highest loading item (loading = 0.91), and states: Evolution is a good explanation for how humans emerged on Earth. Item MATEfacts3 is quite similar, stating: Modern humans are the product of evolutionary processes that have occurred over millions of years. ISEAhuman4 and 7 solicit agreement with the idea that humans and apes share a common ancestor. And ISEAmacro1 asks for agreement with the idea that new species evolved from ancestral species. In sum, these items solicit appraisal of evolution as a true, useful, and valid theory. It is interesting that the highest loading items address human evolution. Conceptually, a factor loading for an item is derived from a regression equation where a student’s measure for a latent construct or dimension is a predictor of his/her response for a certain item. This addresses the question: how well do students’ measures for a certain construct explain their responses for an item defining that construct? Based on this conception, the high factor loadings for the human evolution items indicate that students’ measures for this construct do very well in predicting how they will respond on these items. This implies that students who accept evolution as true, useful, and valid will tend to report very high acceptance for these items, whereas students who reject evolution will tend to report very low acceptance with high predictability. From here on, we label the construct underlying this dimension: acceptance of the truth of evolution.

After reverse coding, the five items which load most highly onto the second dimension (loadings between 0.73 and 0.92) address rejection of incredible or false ideas underlying hesitancy against the theory of evolution. These five highest loading items get at the idea that species do not change. For example, the highest loading item (loading = 0.92), ISEAmicro4, states: Species were created to be perfectly suited to their environment, so they do not change. ISEAmicro7 and MATEcred9 are worded nearly identically, stating that organisms (or species) exist in the same form they always have. ISEAmicro5 solicits non-acceptance of the idea that organisms will evolve new traits over time. Finally, ISEAmicro1 addresses a possible root for these above conceptions: that organisms are currently perfectly suited to their environments, and so do not change. Acceptance of change (or in this case, rejection of the idea that organisms do not change over time), is required in order to find the theory of evolution credible and useful. While these represent the highest loading items, a vast majority of the negatively worded items present false ideas that must be rejected in order to find the theory of evolution credible and useful. We therefore label the construct underlying this dimension: rejection of incredible ideas about evolution.

Rasch analysis

Parametrizing evolution acceptance

The purpose of this study was to examine the measurement properties of the items contained in three quantitative evolution acceptance instruments, each of which captures evolution acceptance differently in terms of dimensionality and context. The MATE was designed as a unidimensional measure which used macroevolution and human evolution contexts in its items (Rutledge and Warden 1999). The I-SEA was developed to capture these contexts in separate constructs; it was designed to measure acceptance of microevolution, macroevolution, and human evolution as three respective dimensions (Nadelson and Southerland 2012). The GAENE was designed to provide a unidimensional measure of evolution acceptance which is independent of knowledge of evolution and religious orientation (Smith et al. 2016). Given the high similarity of wording between many of the items on the three instruments and the fact that they share a common Likert measurement scheme, we hypothesized that these instruments may share more similarities than differences and actually provide similar quantitative information about evolution acceptance. We found that this is the case. Putting the 57 items from the three instruments together to form an instrument-independent scaling results in useful unidimensional and two-dimensional parametrizations of evolution acceptance.

Given the two parametrizations, a big-picture question that comes forth immediately is: should evolution acceptance be treated unidimensionally as it has in the vast majority of quantitative work on evolution acceptance, or should it be treated as a two-dimensional construct which is apparently delineated by the positive or negative orientation of the wording of the items (also found in a recent study by Romine et al. 2017)? Our position is that a unidimensional usage of these items may provide a useful measure for evolution acceptance. Evidence for this includes: (1) the high variance of the first dimension in the EFA (28.9 items of variance out of 57 total items) in comparison to the other dimensions (Fig. 1), (2) the closeness of the first eigenvalue from PCA on Rasch residuals to 2 for a unidimensional construction (2.11 items of variance), and (3) the moderate relationship between acceptance of truth and rejection of incredible ideas (r = 0.73). In fact Metzger et al. (2018) found that much of the apparent two-dimensionality of the MATE found by Romine et al. (2017) disappears in more advanced science students. However, deciding not to recognize evolution acceptance as a two-dimensional construct may also result in a missed opportunity—CFA and Rasch analysis demonstrate that a unidimensional construction does not capture the entire story regarding how these three instruments measure evolution acceptance.

Our analyses collectively (Tables 1, 2, 3) suggest that observing a conceptual distinction between positively- and negatively-worded items is empirically useful, and we argue that recognizing the distinction between acceptance of truth and rejection of incredible ideas is supported by conceptual work around evolution acceptance (Deniz et al. 2008; Ha et al. 2012). For example, Deniz et al. (2008) propose a model in which evolution acceptance is constructed by an integration of cognitive, affective, and contextual factors. Further, they nest evolution acceptance within the theory of conceptual change (Strike and Posner 1992) in their suggestion that acceptance of evolution is necessary in order to find new ideas about evolution plausible and fruitful (Deniz et al. 2008). Our findings in this study not only support this assertion, but also allow us to expand upon it in proposing that the dimensionality by item valence aligns with specific elements of conceptual change theory. The theory of conceptual change is rooted in the idea that students construct new ideas through agreement based on their existing knowledge, experience, and social interactions (Posner et al. 1982). This implies that different individuals may interpret evolutionary phenomena in different ways based on their prior knowledge, metaphysical beliefs, and sociocultural background (Hewson 1981). It follows that one mechanism for learning involves agreement to abandon previously-held ideas and replace them with scientifically acceptable ideas (Posner et al. 1982). For a learner to agree to do this, he/she must find dissatisfaction in a previously-held idea and henceforth find the new idea more intelligible, plausible, and fruitful than the previously-held idea (Posner et al. 1982; Treagust and Duit 2008). Our first sub-construct, acceptance of the truth of evolution, aligns with the intelligibility and plausibility conditions of conceptual change, similar to that suggested by Deniz and colleagues. A student’s measure for rejection of incredible ideas about evolution aligns directly with a student’s dissatisfaction with non-scientific ideas and in turn the likelihood that he/she will find a scientifically-acceptable idea fruitful. Rejection of incredible ideas therefore more directly influences the likelihood that instruction will be successful in facilitating conceptual change. Although we did not measure knowledge of evolution in this study, previous work (Romine et al. 2017) supports this assertion in its finding that the correlation of acceptance of the credibility of evolution (aligned closely with rejection of incredible ideas in this study) with knowledge of macroevolution was significantly higher than the correlation between acceptance of facts about evolution (aligned closely with acceptance of truth in this study) with knowledge of macroevolution.

We also find that the two-dimensional parametrization aligns well with the more recent model of Ha et al. (2012), which describes evolution acceptance as manifesting through both conscious and unconscious neurological pathways. Our acceptance of truth dimension aligns most closely with the unconscious pathway which is based upon a student’s intuitive feeling of certainty. For example, evolution is a good explanation for how humans first emerged on the earth (from the GAENE) solicits an appraisal based on one’s feeling of certainty built from extracurricular experience, and not necessarily analysis of the credibility of an idea based on logic. On the other hand, the rejection of incredible ideas dimension aligns most closely with the student’s conscious, reflective thinking supported by his/her process of understanding and logical reasoning around conceptual ideas underlying evolutionary theory. For example, expressing disagreement with the statement, species were created to be perfectly suited to their environment, so they do not change (from the I-SEA) requires a student to first consider the specific idea and then decide to reject that idea. Although the decision to reject an idea is affective in nature, it is in itself a comparatively logical and reflective process.

In addition to understanding how our current measures of evolution acceptance fit with conceptual work, this study also yields insight into the empirical implications for parametrizing evolution acceptance. Various parametrizations have been explored. Nadelson and Southerland (2012) utilized a three-dimensional model for evolution acceptance in their construction of the I-SEA, suggesting that the dimensions of evolution acceptance should be delineated by the type of evolution: microevolution, microevolution, and human evolution. What this study suggests is that, from a quantitative perspective, topic is a determinant of the difficulty of an item along the same sub-construct (Figs. 3 and 4), but it does not seem to serve as the key delimiter in terms of the unique sub-constructs. In other words, while acceptance of macroevolution, human evolution, and microevolution may be distinct in their difficulty, it may not be necessary to treat them as distinct sub-constructs. Rather, the data show that differences between students’ responses on items across contexts are accounted for by the expected difficulty hierarchy imposed by the Rasch model (Boone 2016), making it unnecessary to define new sub-constructs to account for the different response patterns across contexts.

Previous literature supports the idea that acceptance of evolution varies depending on context (i.e. Scott 2005; Nadelson and Hardy 2015). Figures 3 and 4 suggest that this comprises a progression of acceptance similar in structure to what we have seen in research applying Rasch models to learning progressions (Romine et al. 2016; Todd et al. 2017; Todd and Romine 2016). From Figs. 3 and 4, it appears that undergraduate students look at evolution acceptance as the act of accepting that evolution is true and rejecting incorrect ideas, and that within these constructs, students tend to progress from accepting evolution on short time scales, then on long time scales, and then ultimately that evolution happens in humans.

An additional motivation for constructing the I-SEA by the topics of microevolution, macroevolution, and human evolution is that students who are reluctant to accept that organisms, including humans, evolve on long time scales may nonetheless accept that organisms evolve on short, observable time scales. The Wright map for the unidimensional construction (Fig. 2) shows that the items addressing human evolution are indeed the most difficult. However, the items at the bottom of the scale are macroevolution items, and as we go up the scale, we see a mix of macroevolution and microevolution items. Much of this muddiness clears up when we model the positively and negatively worded items as separate sub-constructs (Figs. 3, 4). A two-dimensional parametrization gives the expected result that the items addressing evolution over short times scales are easiest for students to accept, those addressing evolution over long time scales are more difficult for students to accept, and those addressing human evolution are most difficult for students to accept. We see this as evidence that treating these instruments, and evolution acceptance in general, two-dimensionally gives a more realistic account of how college students think as they decide whether or not to accept evolution.

Previous work with the MATE also supports the idea that evolution acceptance should be treated two-dimensionally. Despite the fact that the MATE has mostly been used as a unidimensional instrument, an analysis by Wagler and Wagler (2013) suggests that the MATE behaves multidimensionally, and that it can be difficult to predict which items load on which dimensions as participants and contexts change. Romine et al. (2017), on the other hand, suggest that the behavior of the MATE is quite predictable in undergraduate students, and put forth a two-dimensional parametrization for the MATE that is similar to that discovered independently in this study. The only item result that was not replicated is Item 5 on the MATE (Cred5). In our previous work, this was the only positively-worded item which loaded more strongly onto the acceptance of credibility than the acceptance of facts dimension. In this study, it loaded onto the acceptance of truth dimension, which is more in line with what we would expect given that it is a positively-worded item.

This multidimensionality of evolution acceptance as measured by the MATE and I-SEA, and lack of basis upon a predefined framework for evolution acceptance, were driving factors for construction of the GAENE, which its authors claim is a generalizable unidimensional measure of evolution acceptance (Smith et al. 2016). Our analysis suggests that the main factor making the GAENE unidimensional is that all of its items are positively-worded. This means that the GAENE measures acceptance of truth but not rejection of incredible ideas. With respect to its contribution to measurement of evolution acceptance, both Figs. 2 and 3 show that the spread of the difficulty of the items is greater than those of the other instruments, meaning that these items provide information about students of a greater ability range than the items from the MATE and I-SEA. While this is attractive from a Rasch modeling perspective (Boone 2016), we also observe that some of the more difficult items from the GAENE do not fit well with the Rasch model, meaning that even students with high levels of acceptance tend to express low levels of acceptance on these items. To follow, we discuss reasons for the misfit of these and other items, and what this may tell us about how to improve our measures of evolution acceptance.

Misfitting items and implications for improving current measures

When the items from these three instruments are treated unidimensionally, we observe the pattern that it is the negatively-worded items which tend to misfit with the Rasch model, and also tend to load positively onto the residual factor (Tables 1, 3). Nine of the 19 negatively-worded items exhibit some misfit (a mean squares index over 1.3) with the Rasch model in this case. These results collectively illustrate the multidimensionality of the negatively-worded items under the assumption of a unidimensional model for evolution acceptance. It makes sense that the Rasch model would tend to model the positively-worded items more faithfully, and thus identify negatively-worded items as anomalous, since the 38 positively-worded items double the 19 items which are negatively-worded.

This apparent bias disappears when positively- and negatively-worded items are treated as separate dimensions as suggested by the EFA, making it straightforward to explore more precisely potential issues with the wording of particular items. On the acceptance of truth dimension, three items (GAENE1, GAENE6, and GAENE8) exhibited a mean squares fit index above 1.50, which is indicative that these items may be harmful to the validity of the scale as it is defined by the corpus of 38 items in this construct (Wright et al. 1994). It happens that these items are also of moderate-to-high difficulty. This suggests a tendency for higher accepting students to express lower levels of acceptance on these items despite expressing high levels of acceptance on the other items measuring this construct. GAENE8, the most difficult item (Fig. 3), states: nothing in biology makes sense without evolution. From an expert biologist’s perspective, evolution may serve as the centerpiece for understanding biology, but a college student who accepts evolution may feel that topics like anatomy and physiology, microbiology, and cell biology can be understood adequately without first understanding evolution. The context is not there since evolution is often not covered in college courses addressing these subjects.

GAENE1, everyone should understand evolution, sits at the middle of the scale (Fig. 3). Along the logic of GAENE8, the misfit of GAENE1 is likely caused by responses from students who accept evolution, but nonetheless do not view it as a necessity for engaging in other courses of study or for advancing one’s quality of life. GAENE6 sits at the top of the Rasch difficulty scale along with GAENE8 (Fig. 3). This item states: I would be willing to argue in favor of evolution in a public forum such as a school club, church group, or meeting of public school parents. Fear of public speaking is quite common (Harris et al. 2002), and it is straightforward to argue that one can display acceptance of evolution without extraversion. In addition to evolution acceptance this item measures willingness to engage in public discourse; the latter having little to do with evolution acceptance.

Two items within the rejection of incredible ideas construct misfit with the Rasch model: MATEcred17 and MATEcred19. These items were also of relatively high difficulty, indicating that students who tended to reject the other misconception statements nonetheless tended to accept these. Students may have accepted MATEcred17, much of the scientific community doubts if evolution occurs, since the term “scientific community” is quite broad. A student who has taken a variety of science classes has likely seen that evolution is not discussed in most of these classes—especially the physical sciences, but even in biology (Padian 2010). It would therefore make sense for students to accept this statement regardless of their tendency to reject other ideas which are not credible. The misfit of MATEcred19, with few exceptions, organisms came into existence about the same time, is more difficult to explain. We argue that the rejection of incredible ideas construct is comparatively aligned with one’s understanding of evolution (cognition), so the tendency for high ability students to accept this idea may be reflective of the persistent misunderstanding of deep time and the evolutionary timeline—one of the most difficult concepts for students to comprehend (Rosengren et al. 2012). Common practice may be to simply eliminate these items from the corpus in future research. However, doing this would carry the consequence of eliminating a potentially important part of the construct. We caution against making assessment decisions based on a single number. Instead the nature of the specific construct of interest should also be considered when deciding which items to use from these three instruments.

Suggestions for using current evolution acceptance instrumentation

A key question that arises from the above discussion is, how should the MATE, I-SEA, and GAENE be used in research? Much of the previous discourse around these instruments has centered on the question: which instrument works best? The I-SEA was developed to make our measures of evolution acceptance more fine-grained (Nadelson and Southerland 2012). A key motivation for developing the GAENE was to overcome the epistemological inadequacies of the MATE and I-SEA (Smith et al. 2016). Although we do not find anything inherently wrong with using a particular instrument, our data indicate that loyalty to a particular instrument is unnecessary. We instead suggest a reframing of the dialogue toward a discussion of the merits and limitations of particular items instead of the instruments themselves. For example, if an evaluation project calls for a focus on human evolution or macroevolution, then we suggest that relevant items from both the MATE and the I-SEA could be combined into a single scale. On a different project, a researcher may wish to obtain a measure that is less context dependent and more based on an individual student’s understanding of what constitutes the theory. In this case, the pool of items may be drawn primarily from the GAENE along with the more context-independent items from the I-SEA and MATE. We would, however, recommend that researchers using the GAENE consider removing the items asking students to “…argue in favor of evolution…” unless improving a student’s extraversion is an important part of the evaluation plan.