Aussie AI
Vocabulary Expansion
-
Last Updated 7 December, 2024
-
by David Spuler, Ph.D.
Vocabulary expansion, or vocabulary extension, is increasing the size of the LLM vocabulary. This means that the overall model has more distinct tokens, which can increase the ability of individual tokens to encode particular states or outputs.
LLM vocabulary expansion can be performed for increased accuracy, such as in foreign languages which have a much greater range of words and symbols (e.g., Unicode and DBCS languages). An increased LLM vocabulary can also be used for improved efficiency, as input sequences may be able to be encoded in fewer tokens, which makes it similar to token merging.
Most of the research on vocabulary expansion is related to foreign language translation via the research area of Neural Machine Translation (NMT). This research has existed for some time prior to much of the LLM research, and often uses non-LLM types of AI models. Hence, there is a need for more research on vocabulary extension with LLMs.
Related areas of LLM inference optimization include:
- Embeddings
- Tokenization
- Vocabulary trimming
- Token pruning
- Embeddings pruning
- Shortlisting
- Funnel transformer
Research on Vocabulary Expansion
Research papers on increasing the size of the LLM vocabulary in tokenization:
- J Hong, G Lee, J Cho, Accelerating Multilingual Language Model for Excessively Tokenized Languages, Findings of the Association for Computational Linguistics: ACL 2024, pages 11095–11111 August 11-16, 2024, https://arxiv.org/abs/2401.10660 https://aclanthology.org/2024.findings-acl.660/ https://aclanthology.org/2024.findings-acl.660.pdf
- Atsuki Yamaguchi, Aline Villavicencio, Nikolaos Aletras, 16 Sep 2024 (v2), How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text? https://arxiv.org/abs/2406.11477
- HyoJung Han, Akiko Eriguchi, Haoran Xu, Hieu Hoang, Marine Carpuat, Huda Khayrallah, 12 Oct 2024, Adapters for Altering LLM Vocabularies: What Languages Benefit the Most? https://arxiv.org/abs/2410.09644
- Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, Ngai Wong, 1 Nov 2024 (v3), Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies, https://arxiv.org/abs/2407.13623 https://github.com/sail-sg/scaling-with-vocab https://hf.co/spaces/sail/scaling-with-vocab-demo
More AI Research
Read more about: