Instructable intelligent personal agent

Amos Azaria, Jayant Krishnamurthy, Tom M. Mitchell

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review


Unlike traditional machine learning methods, humans often learn from natural language instruction. As users become increasingly accustomed to interacting with mobile devices using speech, their interest in instructing these devices in natural language is likely to grow. We introduce our Learning by Instruction Agent (LIA), an intelligent personal agent that users can teach to perform new action sequences to achieve new commands, using solely natural language interaction. LIA uses a CCG semantic parser to ground the semantics of each command in terms of primitive executable procedures defining sensors and effectors of the agent. Given a natural language command that LIA does not understand, it prompts the user to explain how to achieve the command through a sequence of steps, also specified in natural language. A novel lexicon induction algorithm enables LIA to generalize across taught commands, e.g., having been taught how to "forward an email to Alice," LIA can correctly interpret the command "forward this email to Bob." A user study involving email tasks demonstrates that users voluntarily teach LIA new commands, and that these taught commands significantly reduce task completion time. These results demonstrate the potential of natural language instruction as a significant, under-explored paradigm for machine learning.

Original languageEnglish
Title of host publication30th AAAI Conference on Artificial Intelligence, AAAI 2016
Number of pages9
ISBN (Electronic)9781577357605
StatePublished - 2016
Externally publishedYes
Event30th AAAI Conference on Artificial Intelligence, AAAI 2016 - Phoenix, United States
Duration: 12 Feb 201617 Feb 2016

Publication series

Name30th AAAI Conference on Artificial Intelligence, AAAI 2016


Conference30th AAAI Conference on Artificial Intelligence, AAAI 2016
Country/TerritoryUnited States

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence


Dive into the research topics of 'Instructable intelligent personal agent'. Together they form a unique fingerprint.

Cite this