Memory-Efficient Embeddings

Creating smaller models with a new kind of embedding layer

Photo by Kostiantyn Vierkieiev on Unsplash

Whenever dealing with categorical data, beginners resort to one-hot encoding. This is often okay, but if you are dealing with thousands or even millions of categories, this approach becomes infeasible. This has the following reasons:

1. Increased dimensionality: For each category, you get an…