Skip to main content

Table 3 The PREP survey items detect changes in instructional design with large effect sizes (Cohen’s d)

From: Using cognitive load theory to evaluate and improve preparatory materials and study time for the flipped classroom

Difference between years

Prep materials#

Median

Interquartile range

Mann-Whitney U

Cohen’s d

Δ Difficulty z-score

Changes made

No change

0.15

0.11

0.08 – 0.27

0.06 – 0.19

p < .01

1.1

Δ Familiarity z-score

Changes made

No change

0.09

0.07

0.04 – 0.14

0.07 – 0.12

  

Δ Time spent z-score

Changes made

No change

0.28

0.09

0.11 – 0.55

0.04 – 0.18

p < .0001

1.7

Δ Percent score z-score

Changes made

No change

0.12

0.08

0.07 – 0.28

0.04 – 0.14

p < .001

1.1

Δ Efficiency metric Ec* (score vs. difficulty)

Changes made

No change

0.19

0.10

0.08 – 0.34

0.04 – 0.15

p < .0001

1.4

Δ Efficiency metric Et* (score vs. time spent)

Changes made

No change

0.27

0.08

0.10 – 0.45

0.03 – 0.15

p < .0001

2.1

#N = 142 no changes

N = 51 changes made

28 sessions were changed in year 2 (out of 97) and 23 in year 3 (out of 96, one session was replaced with a new topic).

97 (Y2-Y1) + 96 (Y3-Y2) = 193 total, 51 changed, 142 no change

*E is calculated as (y – x)/√2. If E > 0 materials were considered efficient for learning, if < 0 they were considered less efficient for learning (Paas 2003).

  1. We reviewed the preparatory materials for each session across all years and marked which ones underwent revision vs. those that did not change. We subtracted the differences in scores between consecutive years (Δ). Differences were expressed as positive values and compared using non-parametric analysis. The tool was found to be capable in detecting differences in both time efficiency and performance efficiency. Familiarity with materials from prior courses did not change based on changes in the instructional design.