TY - GEN
T1 - PaLM
T2 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019
AU - Peng, Hao
AU - Schwartz, Roy
AU - Smith, Noah A.
N1 - Funding Information: We thank members of the ARK at the University of Washington, and researchers at the Allen Institute for Artificial Intelligence for their helpful comments on an earlier version of this work, and the anonymous reviewers for their insightful feedback. This work was supported in part by NSF grant 1562364. Publisher Copyright: © 2019 Association for Computational Linguistics
PY - 2019
Y1 - 2019
N2 - We present PaLM, a hybrid parser and neural language model. Building on an RNN language model, PaLM adds an attention layer over text spans in the left context. An unsupervised constituency parser can be derived from its attention weights, using a greedy decoding algorithm. We evaluate PaLM on language modeling, and empirically show that it outperforms strong baselines. If syntactic annotations are available, the attention component can be trained in a supervised manner, providing syntactically-informed representations of the context, and further improving language modeling performance.
AB - We present PaLM, a hybrid parser and neural language model. Building on an RNN language model, PaLM adds an attention layer over text spans in the left context. An unsupervised constituency parser can be derived from its attention weights, using a greedy decoding algorithm. We evaluate PaLM on language modeling, and empirically show that it outperforms strong baselines. If syntactic annotations are available, the attention component can be trained in a supervised manner, providing syntactically-informed representations of the context, and further improving language modeling performance.
UR - http://www.scopus.com/inward/record.url?scp=85084317215&partnerID=8YFLogxK
M3 - Conference contribution
T3 - EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference
SP - 3644
EP - 3651
BT - EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference
Y2 - 3 November 2019 through 7 November 2019
ER -