THE ALGORITHMIC PANOPTICON: EVALUATING TECHNICAL UNRELIABILITY, ETHICAL FAILURES AND THE REGULATORY FUTURE OF EMOTION AI IN EDUCATION

Abstract

Emotion recognition AI is now present in education, with mixed and unintended consequences. This paper is focused on understanding what that introduction looks like specifically for neurodivergent pupils and those with diverse cultural backgrounds, for whom these technologies fare worst visibly. We will consider its particular limitations, significant ethical counter-criticism, and the need for strong non-supervisory arrangements that lag, focusing on the EU AI Act.

We also discuss the Act's Composition 5(1)(f) prohibits emotion recognition in educational settings as a general rule. Yet Annex III contemporaneously classifies certain functionalities, such as AI-driven gestures in exams as "high risk" instead of banned, which in turn impacts its transparency and accountability scores. In practice, neither provision has stopped deployment. Obtaining the student's consent is procedural at best, and not substantive. We have drafted an ethical integration framework and have pointed to empirically-based alternatives to biometric surveillance. The article concludes with recommendations of how to move away from the surveilled pedagogy to one of trust and ethics.