On the contribution of neural networks and word embeddings in Natural Language Processing

Cluster of the Word2Vec vector space reduced to two dimensions using t-SNE [6]
Standard neural network architecture (source: http://cs231n.github.io/convolutional-networks/)
  • First, it is not obvious that the word “leopards” in its plural form (which would be one of the main clues to correctly classify the text) would occur in our training data. If this word does not occur in the training data our model would not be able to infer anything from its occurrence. This is often alleviated by lemmatizing the text, i.e., associating all variants of the same word with the same lemma: “leopards” -> “leopard”. All these preprocessing techniques, while useful for linear models [8,9], are not really necessary in neural architectures [10].
  • Second, even with this preprocessing of the input text, it is still likely that the lemma “leopard” is not present in our training corpus (or occurs only a few times). Here is where word embeddings come into play. If you look again at the image from the beginning of the post, you will see that “leopard” is very close in the vector space to words like “tiger” or “panther”. This means that properties across these similar words are shared, and therefore we can infer decisions when “leopard” occurs, even if this word has not been explicitly seen during training.
  • Finally, relying on simple word-based features may work fine for detecting the topic of a text, but may fail in other complex tasks. For example, understanding the syntactic structure of a sentence is generally essential to succeed in a task like sentiment analysis. Let’s consider the polarity detection subtask in which we should predict whether a movie review is positive or negative:



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Jose Camacho Collados

Jose Camacho Collados

Mathematician, AI/NLP researcher and chess International Master. http://www.josecamachocollados.com