Table 3 Performance measures for EvoGrader for ACORNS (n = 1100 written explanations) and ACORNS-like written responses (n = 1100 written resppnses) (n = 2200 total written explanations)
Concept frequency (%) | Kappa | Agreement (%) | Precision | Recall | F1 | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Concept type | Concept | ACORNS | ACORNS-like | ACORNS | ACORNS-like | ACORNS | ACORNS-like | ACORNS | ACORNS-like | ACORNS | ACORNS-like | ACORNS | ACORNS-like |
Normative scientific idea | Variation | 35.3 | 21.5 | 0.903 | 0.822 | 95.6 | 94.4 | 97.0 | 93.9 | 90.5 | 78.8 | 93.6 | 85.7 |
Heritability | 14.7 | 22.3 | 0.879 | 0.852 | 97.1 | 95.2 | 95.1 | 97.5 | 84.6 | 80.4 | 89.5 | 88.1 | |
Competition | 1.6 | 2.2 | 0.971 | 0.932 | 99.9 | 99.7 | 100.0 | 100.0 | 94.4 | 87.5 | 97.1 | 93.3 | |
Limited resources | 20.5 | 22.4 | 0.944 | 0.806 | 98.2 | 93.9 | 96.4 | 99.4 | 94.7 | 73.2 | 95.5 | 84.3 | |
Differential survival | 46.2 | 55.8 | 0.855 | 0.851 | 92.8 | 92.6 | 93.2 | 94.6 | 91.1 | 92.0 | 92.1 | 93.3 | |
Non-adaptive idea | 3.0 | 0.1 | 0.984 | 1.000 | 99.9 | 100.0 | 100.0 | 100.0 | 97.0 | 100.0 | 98.5 | 100.0 | |
Non-normative naïve idea | Need/Goal | 22.8 | 29.0 | 0.849 | 0.871 | 94.7 | 94.8 | 89.4 | 95.2 | 87.3 | 86.5 | 88.3 | 90.6 |
Use/Disuse | 4.1 | 4.3 | 0.848 | 0.653 | 98.8 | 97.4 | 86.4 | 72.5 | 84.4 | 61.7 | 85.4 | 66.7 | |
Adapt/Acclimation | 10.6 | 9.7 | 0.718 | 0.651 | 94.7 | 95.1 | 76.1 | 94.9 | 73.5 | 52.3 | 74.8 | 67.5 | |