MASS-EDITING MEMORY IN A TRANSFORMER

Kevin Meng; Arnab Sen Sharma; Alex Andonian; Yonatan Belinkov; David Bau

MASS-EDITING MEMORY IN A TRANSFORMER

Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, David Bau

Computer Science

Technion - Israel Institute of Technology

Research output: Contribution to conference › Paper › peer-review

Abstract

Recent work has shown exciting promise in updating large language models with new memories, so as to replace obsolete information or add specialized knowledge. However, this line of work is predominantly limited to updating single associations. We develop MEMIT, a method for directly updating a language model with many memories, demonstrating experimentally that it can scale up to thousands of associations for GPT-J (6B) and GPT-NeoX (20B), exceeding prior work by orders of magnitude. Our code and data are at memit.baulab.info.

Original language	English
State	Published - 2023
Event	11th International Conference on Learning Representations, ICLR 2023 - Kigali, Rwanda Duration: 1 May 2023 → 5 May 2023 https://iclr.cc/Conferences/2023

Conference

Conference	11th International Conference on Learning Representations, ICLR 2023
Country/Territory	Rwanda
City	Kigali
Period	1/05/23 → 5/05/23
Internet address	https://iclr.cc/Conferences/2023

All Science Journal Classification (ASJC) codes

Language and Linguistics
Computer Science Applications
Education
Linguistics and Language

Cite this

@conference{73a42be91cfb45879802a9ecaa616b9f,

title = "MASS-EDITING MEMORY IN A TRANSFORMER",

abstract = "Recent work has shown exciting promise in updating large language models with new memories, so as to replace obsolete information or add specialized knowledge. However, this line of work is predominantly limited to updating single associations. We develop MEMIT, a method for directly updating a language model with many memories, demonstrating experimentally that it can scale up to thousands of associations for GPT-J (6B) and GPT-NeoX (20B), exceeding prior work by orders of magnitude. Our code and data are at memit.baulab.info.",

author = "Kevin Meng and Sharma, \{Arnab Sen\} and Alex Andonian and Yonatan Belinkov and David Bau",

note = "Publisher Copyright: {\textcopyright} 2023 11th International Conference on Learning Representations, ICLR 2023. All rights reserved.; 11th International Conference on Learning Representations, ICLR 2023 ; Conference date: 01-05-2023 Through 05-05-2023",

year = "2023",

language = "الإنجليزيّة",

url = "https://iclr.cc/Conferences/2023",

}

TY - CONF

T1 - MASS-EDITING MEMORY IN A TRANSFORMER

AU - Meng, Kevin

AU - Sharma, Arnab Sen

AU - Andonian, Alex

AU - Belinkov, Yonatan

AU - Bau, David

PY - 2023

Y1 - 2023

N2 - Recent work has shown exciting promise in updating large language models with new memories, so as to replace obsolete information or add specialized knowledge. However, this line of work is predominantly limited to updating single associations. We develop MEMIT, a method for directly updating a language model with many memories, demonstrating experimentally that it can scale up to thousands of associations for GPT-J (6B) and GPT-NeoX (20B), exceeding prior work by orders of magnitude. Our code and data are at memit.baulab.info.

AB - Recent work has shown exciting promise in updating large language models with new memories, so as to replace obsolete information or add specialized knowledge. However, this line of work is predominantly limited to updating single associations. We develop MEMIT, a method for directly updating a language model with many memories, demonstrating experimentally that it can scale up to thousands of associations for GPT-J (6B) and GPT-NeoX (20B), exceeding prior work by orders of magnitude. Our code and data are at memit.baulab.info.

UR - http://www.scopus.com/inward/record.url?scp=85182964015&partnerID=8YFLogxK

M3 - محاضرة

T2 - 11th International Conference on Learning Representations, ICLR 2023

Y2 - 1 May 2023 through 5 May 2023

ER -

MASS-EDITING MEMORY IN A TRANSFORMER

Abstract

Conference

All Science Journal Classification (ASJC) codes

Other files and links

Fingerprint

Cite this