Taalas serves Llama 3.1 8B at 17,000 tokens/second

Taalas, a Canadian hardware startup, has created custom silicon (specialized computer chips) that runs Llama 3.1 8B (a type of AI language model that processes text) at 17,000 tokens per second (units of text the AI can process). The hardware uses aggressive quantization (a technique that compresses the model by reducing precision of its numerical values) with 3-bit and 6-bit parameters (different levels of data compression), and their next version will use 4-bit compression.