Software Agents
How software can act as an assistant to the user rather than a tool, by learning from interaction and by proactively anticipating the user's needs.
The Software Agents group investigates a new paradigm for software that acts like an assistant to a user of an interactive interface rather than simply as a tool. While not necessarily as intelligent as a human agent, agent software can learn from interaction with the user, and proactively anticipate the user's needs. We build prototype agent systems in a wide variety of domains, including text and graphic editing, Web browsing, e-commerce, information visualization, and more.

Research Projects

  • AIGRE: Natural Language Interface that Accommodates Vague and Ambiguous Input

    Henry Lieberman and Dustin Arthur Smith

    A major problem for natural language interfaces is their inability to handle text whose meaning depends in part on context. If a user asks his car radio to play "a fast song," or his calendar to schedule "a short meeting," the interpreter would have to accommodate vagueness and ambiguity to figure out what he meant based on what he said. For it to understand what songs or events the speaker intended, it must make decisions that depend on assumed common knowledge about the world and language. Our research presents two approaches for reducing uncertainty in natural language interfaces, by modeling interpretation as a plan recognition problem.

  • Common-Sense Reasoning for Interactive Applications

    Henry Lieberman and Xinyu H. Liu
    A long-standing dream of artificial intelligence has been to put common-sense knowledge into computers–enabling machines to reason about everyday life. Some projects, such as Cyc, have begun to amass large collections of such knowledge. However, it is widely assumed that the use of common sense in interactive applications will remain impractical for years, until these collections can be considered sufficiently complete, and common-sense reasoning sufficiently robust. Recently we have had some success in applying common-sense knowledge in a number of intelligent interface agents, despite the admittedly spotty coverage and unreliable inference of today's common-sense knowledge systems.
  • CommonConsensus: A Game for Collecting Commonsense Goals

    Henry Lieberman and Dustin Smith
    Common Consensus is a fun, self-sustaining web-based game that both collects and validates commonsense knowledge about everyday goals. Goals are a key element of commonsense knowledge; in many of our inferface agents, we need to recognize goals from user actions (plan recognition), and generate sequences of actions that implement goals (planning). We also often need to answer more general questions about the situations in which goals occur, such as when and where a particular goal might be likely, or how long it is likely to take to achieve.
  • Divisi For Dummies (D4D)

    Henry A. Lieberman and Christopher Fry

    A library of Python 2.7 code for facilitating commonsense reasoning with ConceptNet4. This makes high-level features of the ConceptNet4 code base more accessible to application developers. It does not require that users understand matrix math or many underlying concepts. The code is extensively documented, including example calls and lots of default arguments.

  • E-Commerce When Things Go Wrong

    Henry Lieberman
    One of the biggest challenges for the digital economy is what to do when things go wrong. Orders get misplaced, numbers mistyped, requests misunderstood: then what? Consumers are frustrated by long waits on hold, misplaced receipts, and delays to problem resolution; companies are frustrated by the cost of high-quality customer service. Online companies want customers’ trust, and how a company handles problems directly impacts that. We are exploring how software agents and other technologies can help with this issue. Borrowing ideas from software debugging, we can have agents help to automate record-keeping and retrieval, track dependencies, and provide visualization of processes. Diagnostic problem-solving can generate hypotheses about causes of errors, and seek information that allows hypotheses to be tested. Agents act on behalf of both the consumer and the vendor to resolve problems more quickly and at lower cost.
  • Fathom: Probabilistic Graphical Models to Help Mental Health Counselors

    Karthik Dinakar, Jackie Chen, Henry A. Lieberman, and Rosalind W. Picard

    We explore advanced machine learning and reflective user interfaces to scale the national Crisis Text Line. We are using state-of-the-art probabilistic graphical topic models and visualizations to help a mental health counselor, extract patterns of mental health issues experienced by participants, and bring large scale data science to understanding the distribution of mental health issues in the United States.

  • Goal-Oriented Interfaces for Consumer Electronics

    Henry Lieberman and Pei-Yu Chi
    Consumer electronics devices are becoming more complicated, intimidating users. These devices do not know anything about everyday life or human goals, and they show irrelevant menus and options. We are building Roadie, a system with knowledge about the user's intentions to help the device to display relevant information to reach the user's goal. For example, a DVD player might suggest a sound configuration based on the movie it is playing. This will lead to more human-like interactions with these devices. We have constructed a Roadie interface to real consumer electronics devices: a television, set top box, and smart phone. The devices communicate over Wi-Fi, and use the UPnP protocols.
  • Goal-Oriented Interfaces for Mobile Phones

    Henry Lieberman, Karthik Dinakar, Christopher Fry, Dustin Arthur Smith, Hal Abelson and Venky Raju

    Currently each app lives in its own little world, with its own interface. Apps are usually unable to communicate with each other and unable to cooperate to meet users' needs. This project intends to enable end-users to "program" their phones using natural language and speech recognition to perform complex tasks. A user, for example, could say: "Send the song I play most often to Bill." The phone should realize that an MP3 player holds songs; that the MP3 app has a function to order songs by play frequency; how to send a file to another user; and how to look up the user's contact information. We use state-of-the art natural language understanding, common-sense reasoning, and a partial-order planner.

  • Graphical Interfaces for Software Visualization and Debugging

    Henry Lieberman
    This project explores how modern graphical interface techniques and explicit support for the user's problem-solving activities can make more productive interfaces for debugging, which accounts for half the cost of software development. Animated representations of code, a reversible control structure, and instant connections between code and graphical output are some of the techniques used.
  • Human Goal Network

    Henry Lieberman and Dustin Smith
    What motivates people? What changes do people want in the world? We approach questions of this kind by mining goals and plans from text-based websites: wikiHow, eHow, 43things, to-do lists, and commonsense knowledge bases. 43things tells us about people's long term ambitions. How-to instructions and to-do lists tell us about everyday activities. We've analyzed the corpus to find out which goals are most popular, controversial, and concealed. The resulting goal network can be used for plan recognition, natural language understanding, and building intelligent interfaces that understand why they are being used. Come by and learn about how you can use this knowledge about actions/goals, their properties (cost, duration, location), and their relations in your own applications.
  • Justify

    Henry Lieberman and Christopher Fry

    Making optimal decisions can improve a wide array of situations. Humans often perform well on small, focused choices, but performance degrades as complexity increases. Justify leverages human fine-grained reasoning capabilities into a hierarchy that automatically aggregates and summarizes at each level. This flexible organization makes understanding complex arguments more manageable. A Justify discussion is comprised of points; each point has a type that conveys its domain-independent meaning and determines its "summarization strategy." There are points for questions, answers, arithmetic, pro and con rationale, voting, and grouping that help to crystalize an issue. These point types represent a language to facilitate reasoning both for humans and the Justify program itself.

  • Learning Common Sense in a Second Language

    Henry Lieberman, Ned Burns and Li Bian
    It's well known that living in a foreign country dramatically improves the effectiveness of learning a second language over classroom study alone. We are designing language-teaching sequences for a sensor-equipped residence that can detect user interaction with household objects. We use our common-sense knowledge base and reasoning tools to construct teaching sequences, wholly in the target language, of sentences and question-answering interactions that gradually improve the learner's language competence. For example, the first time the user sits in a chair, the system responds with the foreign-language word for "chair," and later with statements and questions that use complete sentences, tenses, questions, materials, goals, and plans.
  • Multi-Lingual ConceptNet

    Hyemin Chung, Jaewoo Chung, Wonsik Kim, Sung Hyon Myaeng and Walter Bender
    A ConceptNet in English is already established and working well. We are now attempting to expand it to other languages and cultures. This project is an extended ConceptNet with Korean common-sense language, which is fundamentally different from English. Through this project, we can learn how to expand the ConceptNet into other languages and how to connect them. By connecting English and Korean ConceptNets, we are hoping not only to see cultural or linguistic differences, but also to solve problems such as the ambiguity of multivocal words, which were difficult to solve with only one ConceptNet.
  • Multilingual Common Sense

    Aparecido Fabiano Pinatti de Carvalho, Jesus Savage Carmona, Marie Tsutsumi, Junia Anacleto, Henry Lieberman, Jason Alonso, Kenneth Arnold, Robert Speer, Vania Paula de Almeida and Veronica Arreola Rios
    This project aims to collect and reason over common-sense knowledge in languages other than English. We have collected large bodies of common-sense knowledge in Portuguese and Korean, and we are expanding to other languages such as Spanish, Dutch, and Italian. We can use techniques based on AnalogySpace to discover correlations between languages, enabling our knowledge bases in different languages to learn from each other.
  • Navigating in Very Large Display Spaces

    Henry Lieberman
    How would you browse a VERY large display space, such as a street map of the entire world? The traditional solution is zoom and pan, but these operations have drawbacks that have gone unchallenged for decades. Shifting attention loses the wider context, leading to that "lost in hyperspace" feeling. We are exploring alternative solutions, such as a new technique that allows zooming and panning in multiple translucent layers.
  • ProcedureSpace: Managing Informality by Example

    Henry Lieberman and Kenneth C. Arnold
    Computers usually require us to be precise about what we want them to do and how we want them to do it, but humans find it hard to be so formal. If we were to give computers formal examples of our informal instructions, maybe they could learn to relate the natural instructions of ordinary users with the specifications, code, and tests with which they are comfortable. Zones and ProcedureSpace are examples of this. Zones is a code search interface that connects code with comments about its purpose. Completed searches become annotations, so the system learns by example. The backend, ProcedureSpace, finds code for a purpose comment (or vice versa) by relating words and phrases to code characteristics and natural language background knowledge. Users of the system are able to describe what they want in their own words, and often find that the system gives them helpful code.
  • Programming in Natural Language

    Henry Lieberman and Moin Ahmad
    We want to build programming systems that can converse with their users to build computer programs. Such systems will enable users without programming expertise to write programs using natural language. The text-based, virtual-world environments called the MOO (multi-user, object-oriented Dungeons and Dragons) allow their users to build objects and give them simple, interactive, text-based behaviors. These behaviors allow other participants in the environment to interact with those objects by invoking actions and receiving text messages. Through our natural-language dialogue system, the beginning programmer will be able to describe objects and the messages in MOO environments.
  • Raconteur: From Chat to Stories

    Henry Lieberman and Pei-Yu Chi
    Raconteur is a story-editing system for conversational storytelling that provides intelligent assistance in illustrating a story with photos and videos from an annotated media library. It performs natural language processing on a text chat between two or more participants, and recommends appropriate items from a personal media library to illustrate a story. A large common-sense knowledge base and a novel common-sense inference technique are used to find relevant media materials to match the story intent in a way that goes beyond keyword matching or word co-occurrence based techniques. Commonsense inference can identify larger-scale story patterns such as expectation violation or conflict and resolution, and helps a storyteller to chat and brainstorm personal stories with a friend.
  • Relational Analogies in Semantic Networks

    Henry Lieberman and Jayant Krishnamurthy
    Analogy is a powerful comparison mechanism, commonly thought to be central to human problem solving. Analogies like "an atom is like the solar system" enable people to effectively transfer knowledge to new domains. Can we enable computers to do similar comparisons? Prior work on analogy (structure mapping) provides guidance about the nature of analogies, but implementations of these theories are inefficient and brittle. We are working on a new analogy mechanism that uses instance learning to make robust, efficient comparisons.
  • Ruminati: Tackling Cyberbullying with Computational Empathy

    Karthik Dinakar, Henry Lieberman, and Birago Jones

    The scourge of cyberbullying has assumed worrisome proportions, with an ever-increasing number of adolescents admitting to having dealt with it either as a victim or a bystander. Anonymity and the lack of meaningful supervision in the electronic medium are two factors that have exacerbated this social menace. This project explores computational methods from natural language processing and reflective user interfaces to alleviate this problem.

  • Steptorials: A New Interaction Technique for Learning Complex Topics

    Henry A. Lieberman, Christopher Fry, Bentley University and Elizabeth Rosenzweig

    A steptorial ("step tutorial") is a new interaction strategy for learning complex topics. Conventional tutorials—such as Khan Academy-style videos, or interactive guided tours—require the student to cede control to the tutorial. Many students prefer to dive right in to try to learn how to do something by exploration, but they are quite likely to get lost or confused if the topic is complex. Steptorials are unique because they allow varying the autonomy of the user at every step. A steptorial has a control structure of a reversible programming language stepper. The user may choose, at any time, to be shown how to do a step, be guided through it, to try to attempt the task by themselves, or to return to a previous step. Steptorials introduce a new paradigm of mixed-initiative learning.

  • Storied Navigation

    Henry Lieberman
    Today, people can tell stories by composing, manipulating, and sequencing individual media artifacts using digital technologies. However, these tools offer little help in developing a story's plot. Specifically, when a user tries to construct her story based on a collection of individual media elements (videos, audio samples), current technological tools do not provide helpful information about the possible narratives that these pieces can form. Storied Navigation is a novel approach to this problem; media sequences are tagged with free-text annotations and stored as a collection. To tell a story, the user inputs a free-text sentence and the system suggests possible segments for a storied succession. This process iterates progressively, helping the user to explore the domain of possible stories. The system achieves the association between the input and the segments' annotations using reasoning techniques that exploit the WordNet semantic network and common-sense reasoning technology.
  • Time Out: Reflective User Interface for Social Networks

    Birago Jones, Henry Lieberman, and Karthik Dinakar

    Time Out is an experimental user interface system for addressing cyberbullying on social networks. A Reflective User Interface (RUI) is a novel concept to help users consider the possible consequences of their online behavior, and assist in intervention or mitigation of potentially negative/harmful actions.

  • You Too!

    Henry Lieberman and Karthik Dinakar

    We are working on ways to detect sensitive themes in the online discussions between teenagers on social networks, and to match them to similar experiences shared publicly by other teenagers. For example, imagine a thread between two teenagers on a social networking website; if the theme of their discussion involves "pressure and regret conditioned on sex," we can show them stories by other teenagers who have publicly talked about pressure and regret based on sex. We're trying to help teenagers in difficult situations, helping them to feel that they are not alone, and that there have been other teenagers in similar plights.