Embedding Size Optimization with NAS

Embedding Size Optimization with NAS

A conceptually simple way to reduce embedding size is to choose a smaller embedding size as a model hyper-parameter. The size of the embedding is a model “hyper-parameter” that is chosen before training. Optimizing this number is a sub-problem of “neural architecture search” (NAS), also called “hyper-parameter optimization” (HPO). The embedding-specific NAS problem has some research papers:

1. Haochen Liu, Xiangyu Zhao, Chong Wang, Xiaobing Liu, and Jiliang Tang. 2020. Automated embedding size search in deep recommender systems, In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2307–2316, https://dl.acm.org/doi/abs/10.1145/3397271.3401436
2. L Qu, Y Ye, N Tang, L Zhang, Y Shi, H Yin, 2022, Single-shot embedding dimension search in recommender system, 2022, https://dl.acm.org/doi/abs/10.1145/3477495.3532060, https://arxiv.org/abs/2204.03281
3. Xiangyu Zhao, Chong Wang, Ming Chen, Xudong Zheng, Xiaobing Liu, and Jiliang Tang, 2020, AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations, CoRR abs/2002.11252 (2020). arXiv:2002.11252, https://arxiv.org/abs/2002.11252
4. Zi Yin and Yuanyuan Shen. 2018. On the dimensionality of word embedding, In Advances in Neural Information Processing Systems. 887 898, https://arxiv.org/abs/1812.04224
5. Maxim Naumov. 2019. On the Dimensionality of Embeddings for Sparse Features and Data, arXiv preprint arXiv:1901.02103 (2019), https://arxiv.org/abs/1901.02103

For more research papers on embeddings NAS, see https://www.aussieai.com/research/embeddings#nas.

• Next: • Up: Table of Contents • Buy: Generative AI in C++: Coding Transformers and LLMs

The new AI programming book by Aussie AI co-founders: AI coding in C++ Transformer engine speedups LLM models Phone and desktop AI Code examples Research citations Get your copy from Amazon: Generative AI in C++