An LP-based hyperparameter optimization model for language modeling

Amir Hossein Akhavan Rahnama, Mehdi Toloo, Nezer Jacob Zaidenberg

Research output: Contribution to journalArticlepeer-review

Abstract

In order to find hyperparameters for a machine learning model, algorithms such as grid search or random search are used over the space of possible values of the models’ hyperparameters. These search algorithms opt the solution that minimizes a specific cost function. In language models, perplexity is one of the most popular cost functions. In this study, we propose a fractional nonlinear programming model that finds the optimal perplexity value. The special structure of the model allows us to approximate it by a linear programming model that can be solved using the well-known simplex algorithm. To the best of our knowledge, this is the first attempt to use optimization techniques to find perplexity values in the language modeling literature. We apply our model to find hyperparameters of a language model and compare it to the grid search algorithm. Furthermore, we illustrate that it results in lower perplexity values. We perform this experiment on a real-world dataset from SwiftKey to validate our proposed approach.

Original languageEnglish
Pages (from-to)2151-2160
Number of pages10
JournalJournal of Supercomputing
Volume74
Issue number5
DOIs
StatePublished - 1 May 2018
Externally publishedYes

Keywords

  • Hyperparameter optimization
  • Language model
  • Linear programming
  • Machine learning
  • Optimization
  • n-Grams

All Science Journal Classification (ASJC) codes

  • Software
  • Theoretical Computer Science
  • Information Systems
  • Hardware and Architecture

Fingerprint

Dive into the research topics of 'An LP-based hyperparameter optimization model for language modeling'. Together they form a unique fingerprint.

Cite this