Machine Learning in Production course revamps grading after AI aces all homework

The instructor of an upper-level university course called Machine Learning in Production, which usually enrolls 100 to 170 students, writes that after AI coding agents became able to complete every homework assignment, most of the course's assessments had to be redesigned even though the underlying learning goals, engineering tradeoffs, risk mitigation and teamwork, stayed the same. The turning point traces back to around 2021, when a colleague, Vincent Hellendoorn, suggested testing GPT-3 on the course's reading quizzes; it passed the grading rubric without having seen the assigned paper, but the instructor changed nothing at the time. By fall 2025, Claude Code could solve the course's original screening assignment, extending a small Instagram clone called albumy (12,000 lines of starter code, with a correct solution of roughly 20 lines) entirely unaided, including the writeup and reflection. That assignment was replaced with a similar task built on Zulip, a production team chat app with more than 500,000 lines of code, which current agents can only handle with active human interaction.

Written reflections, once part of nearly every assignment, have been dropped because they can be fully delegated to an LLM; in their place, each student now holds a 15-minute check-in with a teaching assistant, currently worth 20% of the assignment's points and graded pass or fail, with penalty-free retries if a student fails. The course runs at a 20:1 student-to-TA ratio, with TAs working about 10 hours a week and check-ins taking roughly 300 minutes per TA every two weeks. Resubmissions of assignments, previously unlimited, now carry a 10% grading penalty after the instructor found students submitting AI-generated solutions first and only engaging with feedback on resubmission. More of the grade has also shifted toward the classroom: exams now count for 25% of the grade instead of 15%, and team debriefing sessions after project milestones make up 10 to 20% of the homework and group-work grade. Weekly coding labs and technical debriefs are similarly checked in person by a TA rather than graded on written output, and video demos of working features have also proven effective for some assignments.

To offset the added grading burden, the course built an LLM-as-a-judge autograding system, prompted with rubric items against student code and reports, which flags each answer as "pass" or "needs review"; TAs spend 50 to 80% less time grading as a result, reviewing about 80% less content overall, though the instructor stresses that no points are ever deducted from an autograded answer without a human reviewing it first. The same approach now generates automated feedback on students' in-class Slack discussion answers. The instructor also describes two AI failure modes that have inadvertently become teaching moments: ChatGPT was confidently wrong about where hardware fits in Jackson's World vs the Machine framework, and the share of students making that mistake rose from about 20% before ChatGPT existed to 80% after, dropping only to 50% even once students were warned about it in advance; separately, Claude Code proposed an insecure "two-phase confirmation protocol" for an agent security problem involving financial transactions, and 80% of students, including most of the TAs, initially accepted it as sound. The instructor notes that using AI coding tools well is deliberately not a learning goal of the course, and defends an expected paid AI subscription by comparing its roughly $20 a month against a textbook's roughly $100 cost, while acknowledging this raises equity concerns that the instructor believes institutions, not students, should address.

Key facts

The course Machine Learning in Production (100 to 170 students) replaced graded written reflections with 15-minute TA check-ins worth 20% of assignment points, graded pass/fail with unlimited penalty-free retries.
Resubmissions now carry a 10% penalty, and exams now count for 25% of the grade instead of 15%, after AI undermined the value of low-stakes homework and reduced practice.
LLM-as-a-judge autograding cut TA grading time by 50 to 80% (about 80% less content reviewed manually), though a human still checks every case before points are deducted.
The screening assignment built on a 12k-line codebase (albumy, solvable in about 20 lines) was fully solved unaided by Claude Code in fall 2025, so it was replaced with a task on Zulip, a codebase of more than 500,000 lines.
AI errors became accidental lessons: student mistakes on Jackson's World vs the Machine question rose from about 20% to 80% after ChatGPT, and 80% of students (and most TAs) initially accepted Claude Code's insecure "two-phase confirmation protocol."

Why it matters

The account is a concrete, first-hand case of AI coding agents outpacing an entire course's homework design within a few years, forcing a specific instructor to trade evidence-based low-stakes assessment practices for exams and live check-ins. It is a granular data point on how AI capability growth is reshaping graded work in higher education, not a general trend piece.

Who it affects

University instructors and teaching assistants designing assessments for technical courses, and students in AI-adjacent programs who now face oral check-ins, higher-stakes exams and an expected paid AI subscription instead of traditional take-home work.

How to use it

The instructor's specific moves, such as 15-minute pass/fail TA check-ins worth 20% of assignment points, a 10% resubmission penalty, LLM-as-a-judge autograding that still requires human sign-off before any deduction, and scaling assignments to codebases (like Zulip's 500k+ lines) that agents cannot solve unaided, are described as concrete templates other educators could adapt.

How solid is it

The account comes directly from the course instructor writing about their own course, with specific figures for enrollment, grading weights, TA workload and error rates; it is a personal essay rather than an institutional report or independent study, and the instructor's own institution is not explicitly named in the piece.

Risks and caveats

The instructor states outright that some changes violate evidence-based teaching practices, since research favors frequent low-stakes assessment over few high-stakes exams, and admits to lacking evidence on whether students are actually learning more or less under the new format. Equity concerns around students needing a paid AI subscription are acknowledged but not resolved, and several anti-cheating measures, such as tasks that reliably trip up coding agents, are noted to decay as newer models improve.

“AI is actively undermining several evidence-based teaching practices”

— the course instructor