Post

The Evidence Doubled, the Effect Did Not: What 52 Causal Studies Say About AI in Classrooms

The Evidence Doubled, the Effect Did Not: What 52 Causal Studies Say About AI in Classrooms

Introduction

On 10 November the SCALE Initiative at Stanford published its second evidence review on generative AI in K-12 classrooms. The headline count is the easy part: the causal corpus went from 20 papers in March to 52 papers now, an increase of 32 studies published between October 2025 and June 2026.

The harder part is what the extra 32 studies did to the claims. They mostly shrank them. The review’s own summary is that AI is more likely to help when students use it to understand material than when they use it to produce work, and that neither use consistently improves learning. That is a narrower promise than the one most procurement documents are written around, and it arrives with a study table specific enough to check.

Why this is a positive story anyway A field that can say “we ran 52 causal tests and here is where the effect vanishes” is a field that can be trusted with a budget. The review also names the design choices that do seem to work, and it separates them from the ones that only look like they work. Read it as a set of instructions for building the version that survives an unassisted exam.

This is a different post from the one on this blog about teacher-facing deployment, which read a single trial’s design line by line. This one is about the corpus: which findings replicate across countries, which endpoints were actually measured, and how many schools a real effect would need before anyone could see it.

What one institution published this week

The review screens randomized controlled trials and quasi-experimental designs held in the AI Hub for Education’s research repository, then reports what the new evidence does to the old conclusions. The authors are Seth Walker, Lily Fesler, JP Martinez Claeys, Chris Agnew and Susanna Loeb, and the report carries a DOI.

ItemMarch 2026 reviewNovember 2026 review
Strong causal papers2052
New papers since last review-32
Publication window coveredthrough September 2025October 2025 to June 2026
New U.S. K-12 student-facing studies05
Fastest-growing areastudent-facing AIstudent-facing AI

Two details in that table matter more than the total. First, the growth is concentrated on tools aimed at students rather than at teachers, so the evidence on teacher-facing products is still thin. Second, five new student-facing studies finally ran in U.S. K-12 settings, against none in March. The authors are direct about what that buys: the findings “can provide guidance for education leaders, but should not be treated as broadly generalizable across U.S. K-12 settings.”

So the honest position is that the corpus more than doubled in one release cycle and still does not cover most of the classrooms where these products are being bought.

The split that survives: understanding versus producing

The review’s central finding is a use-mode split, not a product split. Every study below is causal, and the pattern holds across countries with very different school systems.

StudySetting and sampleWhat students did with the AIResult
Bastani et al. (2025)Türkiye, about 1,000 high school students, four 90-minute math sessionsPractised with GPT-4, unrestricted or with pedagogical guardrailsPractice scores up 48 percent unrestricted, 127 percent with guardrails; the unrestricted group then scored 17 percent below control on the unassisted exam
Kosmyna et al. (2025)55 students and staff, essay writingWrote an essay with a chatbot, a search engine, or nothing83 percent of chatbot users could not quote a line from their own essay, against 11 percent of the other two groups
Fischer et al. (2025)Germany, 334 university students, 25-minute sessionStudied economics with restricted access, unrestricted access, or a textbookUnrestricted beat restricted beat no AI; the ordering was about access, not about understanding
Blasco and Charisi (2025)122 high school students, Belgium and Spain, one sessionGot step-by-step reasoning, a bare solution, a Socratic prompt, or a direct oneStep-by-step reasoning beat solution-only; the Socratic version raised engagement but students rated it less helpful
LearnLM team and Eedi (2025)165 Year 9 and 10 students, five UK schools, seven weeksChatted with a pedagogy-tuned model supervised by expert tutors, versus human tutorsAI-guided tutoring matched human tutoring and transferred better to novel topics
Ferman et al. (2021)Brazil, about 19,000 students across 178 schools, one academic yearReceived AI feedback on essays, AI plus human graders, or no feedbackAI feedback improved essay scores; adding human graders did not improve them further

Take the last column across all six rows before reading any single result. When the AI produces the artifact, the artifact improves and the student does not. When the AI is constrained to explain, prompt or grade, the learning measure moves, and in two of those cases it moves as much as the human comparison arm.

The review puts the same point in one sentence: using AI to offload academic tasks can reduce student learning, and the evidence is more favourable, though still mixed, when students use it explicitly to study.

What the crutch effect actually measured

The Türkiye study is the cleanest demonstration because it has two phases and both were measured. In phase one students solved practice problems with their assigned resource. In phase two they sat an exam with no resources at all.

The gap between the two phases is the whole finding. Access to an unrestricted GPT-4 chat raised practice performance by 48 percent. On the exam that followed, the same students scored 17 percent worse than peers who never had the tool. The authors put the degradation at 0.054 on a 0 to 1 scale, and describe the result as access to generative AI improving performance while substantially inhibiting learning.

The guardrailed arm, built with teacher input and prompt engineering to give hints instead of answers, did not show the drop. It also did not show a gain on the exam. Removing the harm and producing a benefit are two different achievements, and most vendor decks present the first as the second.

That ordering explains a lot of contradictory coverage. A pilot that measures during-the-lesson work will find large positive effects almost every time. A pilot that measures a week later with the tool switched off will find much less, and sometimes a negative.

Satisfaction moves before learning does

The reviews of pilots that land on a minister’s desk are usually satisfaction surveys, and the corpus has a clean test of whether those track anything.

Kreijkes et al. (2026) ran 344 Year 10 students in England through two text passages under three conditions: traditional note-taking, note-taking with a GPT-3.5 chatbot, or the chatbot alone, all following instructions for active comprehension-focused reading. Note-taking, alone or with the chatbot, beat AI-only use on comprehension and retention. Students preferred the chatbot anyway, and rated it the most helpful of the three, despite it producing the lowest learning performance.

The pattern repeats on the teacher side. Mahinpei et al. (2026) gave 11 teaching assistants in a U.S. machine learning course optional AI-drafted feedback comments: drafts raised the likelihood of giving any feedback by about 11 percentage points and made comments longer, with similar time per character and similar rated usefulness. That is a real improvement in feedback quantity, and it is also the kind of result a survey of TAs would have reported favourably whether or not students learned more.

The review states the general version plainly: AI tools can improve students’ learning experiences even when they do not improve learning outcomes. Satisfaction is a legitimate operational metric. It is not a substitute for a learning measure, and it is the cheapest thing to collect, which is exactly why it dominates the reports.

The educator split is sharper than the student one

The teacher-side findings are where the review adds the most new detail, because the effect depends on how narrow the tool is.

StudySampleWhat the tool didResult
Wang et al. (2025), Tutor CoPilotMore than 700 tutors and 1,000 students in U.S. Title I schoolsReal-time expert-like suggestions to human tutors during math sessionsStudents 4 percentage points more likely to master topics (p<0.01), 9 points for the lowest-rated tutors, about $20 per tutor per year
Demszky et al. (2025), TeachFX369 U.S. math and science teachers, five monthsAutomated feedback on questioning techniqueTeachers asked 20 percent more focusing questions; no discernible effect on student talk time or reasoning
Roy et al. (2024)259 science teachers, 68 schools in England, ten weeksChatGPT for lesson preparation, with a guide25 minutes per week saved, a 31 percent cut, with no drop in quality
Kim et al. (2021)234 tutors and 2,220 students in South KoreaAI-generated weekly progress reports to home tutorsStudent performance improved, but technology overload cancelled the benefit for some tutors

Tutor CoPilot is the strongest single result in the corpus for a practical reason: it augments a human who is already in the room, and it pays off most for the weakest tutors. That is the shape a deployment should have if the goal is to raise the floor rather than the ceiling.

The new negative result is the closest direct test of the opposite design. Sungu et al. (2026) gave teachers in grades 5 to 12 in Türkiye a GPT-4o teaching-support chatbot for lesson planning, assessment, feedback, differentiation and administration for a full semester, and compared them with teachers continuing usual practice, across 193 teachers and 2,816 students. Average exam scores did not move. Student motivation and confidence fell. Scores declined specifically for students of lower-performing teachers, and the teachers’ own attitudes split by prior use: heavier earlier users became more pessimistic, lighter users more optimistic.

That is a harder finding to explain away than a null. A broad assistant was adopted, used for a semester, produced no average gain, and did damage concentrated in exactly the classrooms where targeted help had previously shown a benefit. The review’s explanation is the output and the expertise needed to evaluate it: a hint about the next question is usable by a novice, while a general plan requires judgement the novice does not yet have.

There is a higher-education data point worth pairing with it. A randomized trial across 2,379 undergraduates and 30 instructors at a large U.S. public university found that giving sections access to a course-integrated AI tutor reduced final grades by 0.37 standard deviations and cut learning-management-system participation by 0.90 standard deviations. Losses were larger for first-generation students. The mechanism the authors propose is displacement: the tool absorbed time that had been going into the course’s existing activities.

Why the corpus doubled and the effect did not

Three end points are being conflated in most of the numbers above, and the review separates them.

In-platform measures. Questions answered, problems completed, mastery flags. These move first and look best.

Interim quizzes. Curriculum-aligned and closer to learning, but short.

Standardized assessments. The slow, expensive one.

A ten-week randomized trial with Eedi in UK secondary schools shows the gap in a single study. It covered 1,179 students across ten schools, with expert tutors reviewing every AI message before it reached a student. AI tutoring performed similarly to human tutoring on in-platform learning and on an intermediate quiz, and both beat standard support. Neither showed a clear advantage over standard support on the Renaissance Star Maths standardized assessment during the trial. The trial also logged the mechanism: students using AI tutoring completed about 14 problems per session against about 19 with static hints. Less coverage, more depth, and no net movement on the test that counts.

That is not a failed intervention. It is a measured trade-off, and the review’s own framing is that tool features and implementation support may matter more than tool type, with interactive formats, adaptive sequencing and tool-use guidance showing promise. The corpus is now specific enough to argue about coverage versus depth instead of arguing about whether AI works.

The geography is worth naming before anyone reuses these numbers. The studies tabulated above were run in the United States, England, Türkiye, Germany, Belgium, Spain, South Korea and Brazil. The nearest sub-Saharan African data point discussed on this blog is a Sierra Leone classroom trial that randomised 48 classrooms across 12 schools, and it is not in this corpus. A ministry deciding on 50,000 seats currently has one or two trials from a neighbouring system plus a strong prior, which is the condition under which a null gets over-read and a vendor number gets repeated.

How many schools does a real effect need

The reason so many of these estimates are small and noisy is arithmetic, not failure. Classroom trials randomize clusters, and clusters cost variance. The standard expression for the variance of the treatment effect, with clusters of equal size, is

\[\mathrm{Var}(\hat{\tau}) = \frac{2\left[\rho + (1-\rho)/m\right]}{J}\]

where $\rho$ is the intraclass correlation, $m$ students per cluster and $J$ clusters per arm. The minimum detectable effect at 80 percent power and a two-sided 5 percent alpha is then $(z_{0.975} + z_{0.80})\sqrt{\mathrm{Var}}$.

That is short enough to compute directly. The script below runs the numbers for a plausible school trial, and every figure quoted after it comes from this output.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
import math

def norm_cdf(z):
    return 0.5 * (1.0 + math.erf(z / math.sqrt(2.0)))

def norm_ppf(p):
    lo, hi = -10.0, 10.0
    for _ in range(300):
        mid = (lo + hi) / 2.0
        if norm_cdf(mid) < p:
            lo = mid
        else:
            hi = mid
    return (lo + hi) / 2.0

ZA = norm_ppf(0.975)   # two-sided alpha 0.05
ZB = norm_ppf(0.80)    # 80 percent power
K = ZA + ZB

def mde(j, m, icc):
    """Minimum detectable effect, SD units, j clusters per arm of size m."""
    se = math.sqrt(2.0 * (icc + (1.0 - icc) / m) / j)
    return K * se

def clusters_needed(target, m, icc):
    """Clusters per arm to detect `target` SD at 80 percent power."""
    se = target / K
    return math.ceil(2.0 * (icc + (1.0 - icc) / m) / se ** 2)

print("z(0.975) = %.4f   z(0.80) = %.4f   sum = %.4f" % (ZA, ZB, K))
print()
print("Minimum detectable effect (SD), 30 students per school, 80% power")
header = "clusters/arm |" + "".join("  ICC %.2f" % i for i in (0.05, 0.10, 0.20, 0.35))
print(header)
for j in (10, 20, 40, 80):
    row = "         %3d |" % j
    row += "".join("    %.3f" % mde(j, 30, i) for i in (0.05, 0.10, 0.20, 0.35))
    print(row)
print()
print("Schools per arm needed to detect a given effect (30 per school)")
for icc in (0.10, 0.20, 0.35):
    for target in (0.10, 0.20, 0.30):
        j = clusters_needed(target, 30, icc)
        print("  ICC %.2f  target %.2f SD -> %4d schools/arm, %6d students/arm"
              % (icc, target, j, j * 30))
print()
for d in (0.10, 0.258, 0.37, 0.50):
    print("effect %.3f SD = median student moves to percentile %.1f"
          % (d, 100 * norm_cdf(d)))
print("mirror for -0.370 SD = percentile %.1f" % (100 * norm_cdf(-0.37)))
print()
j = clusters_needed(0.10, 30, 0.20)
print("Cheapest credible design for 0.10 SD, ICC 0.20, m=30: %d schools total" % (2 * j))
print("students in that design: %d" % (2 * j * 30))
print("cost at USD 4,000 per school for devices and training: USD %s" % format(2 * j * 4000, ","))
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
z(0.975) = 1.9600   z(0.80) = 0.8416   sum = 2.8016

Minimum detectable effect (SD), 30 students per school, 80% power
clusters/arm |  ICC 0.05  ICC 0.10  ICC 0.20  ICC 0.35
          10 |    0.358    0.452    0.597    0.764
          20 |    0.253    0.319    0.422    0.540
          40 |    0.179    0.226    0.298    0.382
          80 |    0.127    0.160    0.211    0.270

Schools per arm needed to detect a given effect (30 per school)
  ICC 0.10  target 0.10 SD ->  205 schools/arm,   6150 students/arm
  ICC 0.10  target 0.20 SD ->   52 schools/arm,   1560 students/arm
  ICC 0.10  target 0.30 SD ->   23 schools/arm,    690 students/arm
  ICC 0.20  target 0.10 SD ->  356 schools/arm,  10680 students/arm
  ICC 0.20  target 0.20 SD ->   89 schools/arm,   2670 students/arm
  ICC 0.20  target 0.30 SD ->   40 schools/arm,   1200 students/arm
  ICC 0.35  target 0.10 SD ->  584 schools/arm,  17520 students/arm
  ICC 0.35  target 0.20 SD ->  146 schools/arm,   4380 students/arm
  ICC 0.35  target 0.30 SD ->   65 schools/arm,   1950 students/arm

effect 0.100 SD = median student moves to percentile 54.0
effect 0.258 SD = median student moves to percentile 60.2
effect 0.370 SD = median student moves to percentile 64.4
effect 0.500 SD = median student moves to percentile 69.1
mirror for -0.370 SD = percentile 35.6

Cheapest credible design for 0.10 SD, ICC 0.20, m=30: 712 schools total
students in that design: 21360
cost at USD 4,000 per school for devices and training: USD 2,848,000

Three readings of that table.

Twenty schools per arm, thirty students each, is a typical funded pilot. At an intraclass correlation of 0.20, which is unremarkable for achievement outcomes, it can only resolve effects larger than 0.42 SD. Anything smaller comes back as noise, and a null result from it is uninformative rather than reassuring.

Small effects are expensive to detect. Getting to 0.10 SD at the same correlation takes 356 schools per arm, 712 in total, and 21,360 students. In device and training terms it is a multi-million dollar study, which is why almost nobody runs one in education and why most published estimates are large enough to be visible at pilot scale.

Effect sizes need units. A 0.37 SD loss, the undergraduate tutor result, lands the median student at percentile 64.4 if the effect is positive and 35.6 if it is negative. A 0.10 SD gain lands them at percentile 54.0. Both are real effects; only one is worth the procurement.

The 24-classroom-per-arm design used in the Sierra Leone trial discussed on this blog sits near the 20-cluster row. Its intent-to-treat estimate of 0.258 SD arrived with an interval running from 0.027 to 0.488, which is what a design of that size produces. Reading the point estimate without the interval is how a modest effect becomes a slogan.

What to instrument before you deploy

The review is useful even if you never run a trial, because its findings translate into design rules.

RuleWhat it prevents
Measure an unassisted endpoint, not just in-product activityThe 48 percent practice gain and the 17 percent exam loss come from the same students; only one of those numbers is usually reported
Pre-register the effect you can detect, not the one you hope forA 20-school pilot cannot see a 0.10 SD effect, so its null result proves nothing
Report the intraclass correlation you assumedWithout it, a sample size cannot be checked by anyone
Log session length, messages and problems per sessionThe 14 versus 19 problems trade-off is invisible unless you instrument it
Price per teacher-year, and compare to the coaching it replacesTutor CoPilot at about $20 per tutor per year is a price most procurement teams cannot compute, because they buy seats
Keep a human who is accountable for the last stepEvery favourable educator result cited above has a human deciding what to send

The last rule covers the widest gap in the evidence. Where a person reviewed the AI’s output before it reached a student, the results were neutral to positive. Where the tool talked directly to students with no constraint, the results split, and the split ran along whether the student was producing work or understanding it.

Key takeaways

FindingNumberWhat it means for a deployment
The causal corpus more than doubled20 to 52 papers in eight monthsThe field can now argue with evidence, but coverage still misses most classrooms
Unrestricted access can displace learning48 percent better practice, 17 percent worse unassisted examAlways measure with the tool switched off
Guardrails removed the harm, not the gapNegative effect essentially eradicated, no positive effectAssume a well-built tutor buys neutrality until proven otherwise
Narrow tools beat broad ones4 percentage points overall, 9 for the weakest tutors, at about $20 per tutor-yearTarget the human who is already in the room
Pilot designs cannot see small effects0.42 SD at 20 schools per arm, ICC 0.20State the minimum detectable effect before you cite the result
Experience is not learningPositive satisfaction alongside flat or negative outcomes in several studiesTreat survey scores as a leading indicator only

References

  • Walker, S., Fesler, L., Martinez Claeys, JP., Agnew, C., and Loeb, S. (2026). The Evidence Base on AI in K-12: A November 2026 Review. SCALE Initiative, Stanford University. doi.org/10.26300/80wn-ca81 — report page
  • Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakci, O., and Mariman, R. (2025). Generative AI Can Harm Learning. SSRN working paper — SCALE record
  • Wharton, Knowledge at Wharton. Without guardrails, generative AI can harm education. knowledge.wharton.upenn.edu
  • Wang, R. E., Ribeiro, A. T., Robinson, C. D., Loeb, S., and Demszky, D. (2025). Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise. EdWorkingPaper 24-1056
  • Liu, J., Sweet, T., Chen, M. H., Engelberg, J., Masters, M., Clark, M., Persaud, A., Lancaster, A., Hollingsworth, J., and King Rice, J. (2026). The Effects of Course-Integrated AI Tutoring on Student Performance and Engagement: A Randomized University Trial. EdWorkingPaper 26-1598
  • EdTech Innovation Hub. Google takes Guided Learning global as schools gain new AI controls. edtechinnovationhub.com
  • Kosmyna, N. et al. (2025). Your Brain on ChatGPT. arXiv:2506.08872
  • Hayes, R. J., and Moulton, L. H. Cluster Randomised Trials, second edition. Taylor and Francis
  • Cochrane Handbook, chapter 23: including variants on randomized trials, design effect and intracluster correlation. cochrane.org

Sample sizes and point estimates for Sungu et al. (2026), Kreijkes et al. (2026), Mahinpei et al. (2026), Ferman et al. (2021), Roy et al. (2024), Demszky et al. (2025), Kim et al. (2021), LearnLM and Eedi (2025), Fischer et al. (2025) and Blasco and Charisi (2025) are quoted from the study tables in the appendix of the review above, where each is classified by design.

This post is licensed under CC BY 4.0 by the author.