AI Therapy under the EU AI Act

AI systems used for therapy or emotional support, including general-purpose AI (GPAI, like ChatGPT or Claude that can do many tasks) systems, can be convenient but may cause harm, especially to vulnerable users like children or people in distress. Under the EU AI Act, providers of these systems must comply with various obligations depending on whether the system is banned, classified as high-risk, or subject to transparency rules (requiring the AI to be honest about how it works when talking directly to users). Providers of GPAI models must also identify and reduce systemic risks to mental health and fundamental rights, and report serious incidents of harm.