Speeding up agentic workflows with WebSockets in the Responses API
Summary
Codex (an AI coding assistant) agent loops involved many back-and-forth API requests that added significant delays, especially as model inference speeds improved to nearly 1,000 tokens per second (words generated per second). To reduce this overhead, the team implemented WebSockets (a protocol that maintains a persistent connection between client and server, rather than opening a new connection for each request), along with caching and eliminating unnecessary network calls, achieving a 40% overall speedup in end-to-end performance.
Solution / Mitigation
The team implemented WebSockets as a persistent connection protocol for the Responses API instead of using multiple synchronous HTTP requests. Additionally, they applied caching to store rendered tokens and model configuration in memory to skip expensive tokenization and network calls, reduced network hop latency by eliminating intermediate service calls and directly contacting the inference service, and improved the safety stack to run classifiers faster.
Classification
Affected Vendors
Related Issues
Original source: https://openai.com/index/speeding-up-agentic-workflows-with-websockets
First tracked: April 22, 2026 at 02:00 PM
Classified by LLM (prompt v3) · confidence: 85%