Software Agents
How software can act as an assistant to the user rather than a tool, by learning from interaction and by proactively anticipating the user's needs.
The Software Agents group investigates a new paradigm for software that acts like an assistant to a user of an interactive interface rather than simply as a tool. While not necessarily as intelligent as a human agent, agent software can learn from interaction with the user, and proactively anticipate the user's needs. We build prototype agent systems in a wide variety of domains, including text and graphic editing, Web browsing, e-commerce, information visualization, and more.

Research Projects

  • AIGRE: A natural language interface that accommodates vague and ambiguous input

    Henry Lieberman and Dustin Arthur Smith

    A major problem for natural language interfaces is their inability to handle text whose meaning depends in part on context. If a user asks his car radio to play "a fast song", or his calendar to schedule "a short meeting," the interpreter would have to accommodate vagueness and ambiguity to figure out what he meant based on what he said. For it to understand what songs or events the speaker intended, it must make decisions that depend on assumed common knowledge about the world and language. Our research presents two approaches for reducing uncertainty in natural language interfaces, by modeling interpretation as a plan recognition problem.

    view site

  • Common-Sense Reasoning for Interactive Applications

    Henry Lieberman and Xinyu H. Liu
    A long-standing dream of artificial intelligence has been to put common-sense knowledge into computers–enabling machines to reason about everyday life. Some projects, such as Cyc, have begun to amass large collections of such knowledge. However, it is widely assumed that the use of common sense in interactive applications will remain impractical for years, until these collections can be considered sufficiently complete, and common-sense reasoning sufficiently robust. Recently we have had some success in applying common-sense knowledge in a number of intelligent interface agents, despite the admittedly spotty coverage and unreliable inference of today's common-sense knowledge systems.
    view site

  • CommonConsensus: A Game for Collecting Commonsense Goals

    Henry Lieberman and Dustin Smith
    We have developed, Common Consensus: a fun, self-sustaining web-based game, that both collects and validates Commonsense knowledge about everyday goals. Goals are a key element of commonsense knowledge; in many of our inferface agents, we need to recognize goals from user actions (plan recognition), and generate sequences of actions that implement goals (planning). We also often need to answer more general questions about the situations in which goals occur, such as when and where a particular goal might be likely, or how long it is likely to take to achieve.
    view site

  • E-Commerce When Things Go Wrong

    Henry Lieberman
    One of the biggest challenges for the digital economy is what to do when things go wrong. Orders get misplaced, numbers mistyped, requests misunderstood: then what? Consumers are frustrated by long waits on hold, misplaced receipts, and delays to problem resolution; companies are frustrated by the cost of high-quality customer service. Online companies want customers’ trust, and how a company handles problems directly impacts that. We explore how software agents and other technologies can help with this issue. Borrowing ideas from software debugging, we can have agents help to automate record-keeping and retrieval, track dependencies, and provide visualization of processes. Diagnostic problem-solving can generate hypotheses about causes of errors, and seek information that allows hypotheses to be tested. Agents act on behalf of both the consumer and the vendor to resolve problems more quickly and at lower cost.
    view site

  • Goal-Oriented Interfaces for Consumer Electronics

    Henry Lieberman and Pei-Yu Chi
    Consumer electronics devices are becoming more complicated, intimidating users. These devices do not know anything about everyday life or human goals, and they show irrelevant menus and options. Using common-sense reasoning, we are building a system, Roadie, with knowledge about the user's intentions; this knowledge will help the device to display relevant information to reach the user's goal. For example, an amplifier should suggest a play option when a new instrument is connected, or a DVD player suggest a sound configuration based on the movie it is playing. This will lead to more human-like interactions with these devices. We have constructed a Roadie interface to real consumer electronics devices: a television, set top box, and smart phone. The devices communicate over Wi-Fi, and use the UPnP protocols.
    view site

  • Goal-Oriented Interfaces for Mobile Phones

    Christopher Fry, karthik, Dustin Arthur Smith, Hal Abelson and Venky Raju

    Contemporary mobile phones provide a vast array of capabilities in so-called "apps," but currently each app lives in its own little world, with its own interface. Apps are usually unable to communicate with each other and unable to cooperate to meet users' needs. This project intends to enable end-users to "program" their phones using natural language and speech recognition to perform complex tasks. A user, for example, could say: "Send the song I play most often to Bill." The phone should realize that an MP3 player holds songs, and that the MP3 app has a function to order songs by play frequency. It should know how to send a file to another user, and how to look up the user's contact information. We use state-of-the art natural language understanding, commonsense reasoning, and a partial-order planner.

  • Graphical Interfaces for Software Visualization and Debugging

    Henry Lieberman
    This project explores how modern graphical interface techniques and explicit support for the user's problem-solving activities can make more productive interfaces for debugging, which accounts for half the cost of software development. Animated representations of code, a reversible control structure, and instant connections between code and graphical output are some of the techniques used.

  • Human Goal Network

    Henry Lieberman and Dustin Smith
    What motivates people? What changes do people want in the world? We approach questions of this kind by mining goals and plans from text-based websites: wikiHow, eHow, 43things, to-do lists, and commonsense knowledge bases. 43things tells us about people's long term ambitions. How-to instructions and to-do lists tell us about everyday activities. We've analyzed the corpus to find out which goals are most popular, controversial, and concealed. The resulting goal network can be used for plan recognition, natural language understanding, and building intelligent interfaces that understand why they are being used. Come by and learn about how you can use this knowledge about actions/goals, their properties (cost, duration, location) and their relations in your own applications.
    view site

  • Learning Common Sense in a Second Language

    Henry Lieberman, Ned Burns and Li Bian
    It's well known that living in a foreign country dramatically improves the effectiveness of learning a second language over classroom study alone. This is likely because people make associations with the foreign language as they see and participate in everyday life activities. We are designing language-teaching sequences for a sensor-equipped residence that can detect user interaction with household objects. We use our common-sense knowledge base and reasoning tools to construct teaching sequences, wholly in the target language, of sentences and question-answering interactions that gradually improve the learner's language competence. For example, the first time the user sits in a chair, the system responds with the foreign-language word for "chair," and later with statements and questions such as, "You sit in the chair" (complete sentence), "You sat in the chair" (tenses), "What is the chair made of?" (question, materials), or "Why are you sitting in the chair?" (goals, plans).

  • Multi-Lingual ConceptNet

    Hyemin Chung, Jaewoo Chung, Wonsik Kim, Sung Hyon Myaeng and Walter Bender
    A ConceptNet in English is already established and working well. We are now attempting to expand it to other languages and cultures. This project is an extended ConceptNet with Korean common sense, which is fundamentally different from English. Through this project, we can learn how to expand the ConceptNet into other languages and how to connect them. By connecting English and Korean ConceptNets, we are hoping not only to see cultural or linguistic differences, but also to solve problems such as the ambiguity of multivocal words, which were difficult to solve with only one ConceptNet.
    view site

  • Multilingual Common Sense

    Aparecido Fabiano Pinatti de Carvalho, Jesus Savage Carmona, Marie Tsutsumi, Junia Anacleto, Henry Lieberman, Jason Alonso, Kenneth Arnold, Robert Speer, Vania Paula de Almeida and Veronica Arreola Rios
    This project aims to collect and reason over common-sense knowledge in languages other than English. We have collected large bodies of common-sense knowledge in Portuguese and Korean, and we are expanding to other languages such as Spanish, Dutch, and Italian. We can use techniques based on AnalogySpace to discover correlations between languages, enabling our knowledge bases in different languages to learn from each other.
    view site

  • Navigating in Very Large Display Spaces

    Henry Lieberman
    How would you browse a VERY large display space, such as a street map of the entire world? The traditional solution is zoom and pan, but these operations have drawbacks that have gone unchallenged for decades. Shifting attention loses the wider context, leading to that "lost in hyperspace" feeling. We are exploring alternative solutions, such as a new technique that allows zooming and panning in multiple translucent layers.
    view site

  • Open Interpreter

    Henry Lieberman and Dustin Arthur Smith

    Language interpretation requires going beyond the words to derive what the speaker meant–cooperatively making 'leaps of faith' and putting forth assumptions that can later be revised or redacted. Current natural language interfaces are opaque; when interpretation goes wrong–which it inevitably does–the human is left without recourse. The Open Interpreter project brings the assumptions involved with interpreting English event descriptions into the user interface, so people can participate in teaching the computer to derive the same common-sense assumptions that they expected. We show the immediate applications for an intelligent calendaring application.

  • ProcedureSpace: Managing Informality by Example

    Henry Lieberman and Kenneth C. Arnold
    Computers usually require us to be precise about what we want them to do and how, but humans find it hard to be so formal. If we gave computers formal examples of our informal instructions, maybe they could learn to relate ordinary users' natural instructions with the specifications, code, and tests with which they are comfortable. Zones and ProcedureSpace are examples of this. Zones is a code search interface that connects code with comments about its purpose. Completed searches become annotations, so the system learns by example. The backend, ProcedureSpace, finds code for a purpose comment (or vice versa) by relating words and phrases to code characteristics and natural language background knowledge. Users of the system were able describe what they wanted in their own words, and often found that the system gave them helpful code.

  • Programming in Natural Language

    Henry Lieberman and Moin Ahmad
    We want to build programming systems that can converse with their users to build computer programs. Such systems will enable users without programming expertise to write programs using natural language. The text-based, virtual-world environments called the MOO (multi-user, object-oriented Dungeons and Dragons) allow their users to build objects and give them simple, interactive, text-based behaviors. These behaviors allow other participants in the environment to interact with those objects by invoking actions and receiving text messages. Through our natural-language dialogue system, the beginning programmer will be able to describe objects and the messages in MOO environments.
    view site

  • Raconteur: From Chat to Stories

    Henry Lieberman and Pei-Yu Chi
    Raconteur is a story-editing system for conversational storytelling that provides intelligent assistance in illustrating a story with photos and videos from an annotated media library. It performs natural language processing on a text chat between two or more participants, and recommends appropriate items from a personal media library to illustrate a story. A large common-sense knowledge base and a novel common-sense inference technique are used to find relevant media materials to match the story intent in a way that goes beyond keyword matching or word co-occurrence based techniques. Common-sense inference can identify larger-scale story patterns such as expectation violation or conflict and resolution, and helps a storyteller to chat and brainstorm his personal stories with a friend.
    view site

  • Relational Analogies in Semantic Networks

    Henry Lieberman and Jayant Krishnamurthy
    Analogy is a powerful comparison mechanism, commonly thought to be central to human problem solving. Analogies like "an atom is like the solar system" enable people to effectively transfer knowledge to new domains. Can we enable computers to do similar comparisons? Prior work on analogy (structure mapping) provides guidance about the nature of analogies, but implementations of these theories are inefficient and brittle. We are working on a new analogy mechanism that uses instance learning to make robust, efficient comparisons.

  • Ruminati: Tackling Cyberbullying with Computational Empathy

    Karthik Dinakar, Henry Lieberman, and Birago Jones

    The scourge of cyberbullying has assumed worrisome proportions with an ever-increasing number of adolescents admitting to having dealt with it either as a victim or bystander. Anonymity and the lack of meaningful supervision in the electronic medium are two factors that have exacerbated this social menace. This project explores computational methods from natural language processing and reflective user interfaces to alleviate this problem.

  • Storied Navigation

    Henry Lieberman
    Today, people can tell stories by composing, manipulating, and sequencing individual media artifacts using digital technologies. However, these tools offer little help in developing a story's plot. Specifically, when a user tries to construct her story based on a collection of individual media elements (videos, audio samples), current technological tools do not provide helpful information about the possible narratives that these pieces can form. Storied Navigation is a novel approach to this problem; media sequences are tagged with free-text annotations and stored as a collection. To tell a story, the user inputs a free-text sentence and the system suggests possible segments for a storied succession. This process iterates progressively, helping the user to explore the domain of possible stories. The system achieves the association between the input and the segments' annotations using reasoning techniques that exploit the WordNet semantic network and common-sense reasoning technology.
    view site

  • Time Out: Reflective User Interface for Social Networks

    Birago Jones, Henry Lieberman and Karthik Dinakar

    Time Out is a experimental user interface system for addressing cyberbullying on social networks. A Reflective User Interface (RUI) is a novel concept to help users consider the possible consequences of their online behavior, and assist in intervention or mitigation of potentially negative/harmful actions.