Why fears of AI self-improvement are causing ‘existential’ concerns at Anthropic and OpenAI

Researchers at Anthropic and OpenAI are concerned about recursive self-improvement (RSI, where AI systems help train better versions of themselves), which they say is accelerating faster than expected and could eventually lead to AI systems improving themselves without human control. The worry is that if AI takes over its own development process, humans might lose the ability to manage these increasingly powerful systems, and there is currently no clear scientific plan to prevent risks from this scenario.