The Spectral Underpinning of word2vec

Ariel Jaffe, Yuval Kluger, Ofir Lindenbaum, Jonathan Patsenker, Erez Peterfreund, Stefan Steinerberger

Research output: Contribution to journalArticlepeer-review

Abstract

Word2vec introduced by Mikolov et al. is a word embedding method that is widely used in natural language processing. Despite its success and frequent use, a strong theoretical justification is still lacking. The main contribution of our paper is to propose a rigorous analysis of the highly nonlinear functional of word2vec. Our results suggest that word2vec may be primarily driven by an underlying spectral method. This insight may open the door to obtaining provable guarantees for word2vec. We support these findings by numerical simulations. One fascinating open question is whether the nonlinear properties of word2vec that are not captured by the spectral method are beneficial and, if so, by what mechanism.

Original languageEnglish
Article number593406
JournalFrontiers in Applied Mathematics and Statistics
Volume6
DOIs
StatePublished - 3 Dec 2020
Externally publishedYes

Keywords

  • dimensionality reduction
  • nonlinear functional
  • skip-gram model
  • spectral method
  • word embedding
  • word2vec

All Science Journal Classification (ASJC) codes

  • Statistics and Probability
  • Applied Mathematics

Fingerprint

Dive into the research topics of 'The Spectral Underpinning of word2vec'. Together they form a unique fingerprint.

Cite this