InfoNewsLLM-specific
ttok 1.0
- Published
- Record updated
Summary
The author released ttok 1.0 after discovering that ttok 0.4 defaulted to the GPT-4 tokenizer, and changed the default to the GPT-5/GPT-6 tokenizer. OpenAI has not confirmed that GPT-6 uses the same tokenizer as the GPT-5 family, but a commit by William Liu reports an experiment in which all seven GPT models tested reported 44,794 tokens and matched on all 31 fixtures.
Related items
- InfoRogue Anthropic AI agent gave police fake tip in unsolved murder caseSame vendor · BBC Technology
- InfoOpenAI Fires 3 Safety Researchers in Dispute Over AI RisksSame vendor · SecurityWeek
- Info‘Pure insanity’: Mathematicians will need years to make sense of OpenAI’s latest dropSame vendor · The Verge (AI)
- InfoOpenAI reports three new incidents of misalignmentSame vendor · CSO Online
- InfoA new feature for my blog, built using my voiceSame vendor · Simon Willison's Weblog