SenseBERT: Driving some sense into BERT

Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, Yoav Shoham

نتاج البحث: فصل من :كتاب / تقرير / مؤتمرمنشور من مؤتمرمراجعة النظراء

ملخص

The ability to learn from large unlabeled corpora has allowed neural language models to advance the frontier in natural language understanding. However, existing self-supervision techniques operate at the word form level, which serves as a surrogate for the underlying semantic content. This paper proposes a method to employ weak-supervision directly at the word sense level. Our model, named SenseBERT, is pre-trained to predict not only the masked words but also their WordNet supersenses. Accordingly, we attain a lexical-semantic level language model, without the use of human annotation. SenseBERT achieves significantly improved lexical understanding, as we demonstrate by experimenting on SemEval Word Sense Disambiguation, and by attaining a state of the art result on the 'Word in Context' task.

اللغة الأصليةالإنجليزيّة
عنوان منشور المضيفACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference
ناشرAssociation for Computational Linguistics (ACL)
الصفحات4656-4667
عدد الصفحات12
رقم المعيار الدولي للكتب (الإلكتروني)9781952148255
حالة النشرنُشِر - 2020
منشور خارجيًانعم
الحدث58th Annual Meeting of the Association for Computational Linguistics, ACL 2020 - Virtual, Online, الولايات المتّحدة
المدة: ٥ يوليو ٢٠٢٠١٠ يوليو ٢٠٢٠

سلسلة المنشورات

الاسمProceedings of the Annual Meeting of the Association for Computational Linguistics

!!Conference

!!Conference58th Annual Meeting of the Association for Computational Linguistics, ACL 2020
الدولة/الإقليمالولايات المتّحدة
المدينةVirtual, Online
المدة٥/٠٧/٢٠١٠/٠٧/٢٠

All Science Journal Classification (ASJC) codes

  • !!Language and Linguistics
  • !!Computer Science Applications
  • !!Linguistics and Language

بصمة

أدرس بدقة موضوعات البحث “SenseBERT: Driving some sense into BERT'. فهما يشكلان معًا بصمة فريدة.

قم بذكر هذا