tokenizers
Declares an LLM dependency since 2023-09-07 (version 0.14.0).
Advisories
Advisories that name tokenizers as affected. Advisory records do not state an ecosystem, so packages with the same name in other ecosystems also match. For a version-exact check of your own dependencies, use Stack Check.
No advisory in the database names this package.
Dependencies of the latest release
As declared in PyPI metadata for version 0.23.3. Optional extras are listed with their extra name.
- datasets[testing]FAISS, Hugging Face Hub / Transformers
- huggingface-hubHugging Face Hub / Transformers
- numpy[testing]
- pytest[testing]
- pytest-asyncio[testing]
- requests[testing]
- ruff[testing]
- setuptools-rust[docs]
- sphinx[docs]
- sphinx-rtd-theme[docs]
- ty[testing]
Tracked packages that depend on it
Among the packages in the registry; not every dependent on PyPI.
- agno
- aider-chat
- axolotl
- chonkie
- chromadb
- cohere
- crawl4ai
- crewai
- fastembed
- fastembed-gpu
- faster-whisper
- gpt-researcher
- guidance
- hf-doc-builder
- langchain-huggingface
- langchain-mistralai
- litellm
- manifest-ml
- model2vec
- modelscope
- neural-compressor
- r2r
- semantic-router
- sentence-transformers
- sglang
- snowflake-ml-python
- sudachipy
- transformers
- txtai
- unclecode-litellm
- verdict
- vllm
- voyageai