These startups are chasing the next big thing in LLMs

Transformers, the neural network architecture (a type of AI model structure) that powers modern large language models, are becoming a bottleneck because they require massive amounts of computation to process text, especially when handling large amounts of input data simultaneously. Researchers and startups are exploring new approaches to replace or improve transformers, with one promising direction being sparse attention, which reduces computational load by only comparing some word pairs instead of all pairs.