Research Group Projects and Descriptions

The Software Agents group investigates a new paradigm for software that acts like an assistant to a user of an interactive interface rather than simply as a tool. While not necessarily as intelligent as a human agent, agent software can learn from interaction with the user, and proactively anticipate the user's needs. We build prototype agent systems in a wide variety of domains, including text and graphic editing, Web browsing, e-commerce, information visualization, and more.

Current experiments in agent software rely mostly on domain-specific applications that either have been programmed from scratch, or explicitly modified with an agent in mind. Is it possible to make a tool kit or protocol that would allow an agent to communicate and to control applications that have been constructed more conventionally? Can the agent "take the place" of the user in the interface? Can the agent have access to the application's data and behavior? Will commercial "inter-application communication" mechanisms suffice? What is the division of labor between the agent and the application? This work will explore these questions.

Effective image annotation and retrieval is bound up with image use. In this project, annotation, retrieval, and use are integrated, facilitating the finding and using images. A proactive user-interface agent seeks chances for image annotation and retrieval in the context of the user's everyday work, using an agent that sit in the user's text editor or other application and continuously monitors typing. Searches are automatically performed from an image library, and images relevant to the current text can be inserted in a single operation. Descriptions of images for storytelling can be seamlessly employed as raw material for annotation. Common-sense knowledge about situations in which pictures are taken, described, or used can help provide semi-automatic annotation and indirect inference for retrieval. Our approach does not completely automate the annotation/retrieval process, but it does reduce user-interface overhead, leading to better-annotated image libraries and fewer missed opportunities for image use.

Alumni Contributor(s): Kim Waters

We are building a new kind of agent that acts as a user's assistant in browsing the World Wide Web. Many current Web tools perform searches for the user, but our approach is to consider the search for information as a cooperative venture between the human user and an intelligent software agent. Rather than search a pre-indexed portion of the Web according to user-stated keywords, the agent, Letizia, infers interest implicitly from observing user actions and tries to stay just a few steps ahead of the user, searching the user's immediately accessible links dynamically.

AnalogySpace enables common-sense reasoning through principal component analysis. It projects the information in ConceptNet into a reduced-dimensional space that describes common-sense concepts and their properties in terms of automatically discovered correlations called "eigenconcepts." AnalogySpace can be used to infer new information, reason about ad hoc categories, detect topics in text, and compare concepts on scales that can be generated on the fly.

This project aims to develop an intelligent personal-finance advisory agent that bridges the gap between the novice user and the expert model of the finance domain. The agent uses common-sense reasoning and inference for associating the user's personal life, financial situation, and goals with the attributes of the expert domain model and vice versa. The agent interface provides a natural-language interface for elicitation and explanations of design and process rationale. The architecture of the system is domain-independent and consequently can be used for any novice-expert domain model.

Alumni Contributor(s): Ashwani Kumar

A long-standing dream of artificial intelligence has been to put common-sense knowledge into computers–enabling machines to reason about everyday life. Some projects, such as Cyc, have begun to amass large collections of such knowledge. However, it is widely assumed that the use of common sense in interactive applications will remain impractical for years, until these collections can be considered sufficiently complete, and common-sense reasoning sufficiently robust. Recently we have had some success in applying common-sense knowledge in a number of intelligent interface agents, despite the admittedly spotty coverage and unreliable inference of today's common-sense knowledge systems.

Alumni Contributor(s): Push Singh

We have developed, Common Consensus: a fun, self-sustaining web-based game, that both collects and validates Commonsense knowledge about everyday goals. Goals are a key element of commonsense knowledge; in many of our inferface agents, we need to recognize goals from user actions (plan recognition), and generate sequences of actions that implement goals (planning). We also often need to answer more general questions about the situations in which goals occur, such as when and where a particular goal might be likely, or how long it is likely to take to achieve.

Alumni Contributor(s): Push Singh

We are developing next-generation architectures for artificial intelligence based on Professor Minsky's "Society of Mind" theory of human thinking. The main idea is that the key to human flexibility and resourcefulness is mental diversity: we have many ways to solve every kind of problem; when we get stuck trying one method of solution, we can switch to another. We are exploring how this idea can be applied at different places and levels in a cognitive architecture, in order to build systems capable of robust common-sense reasoning.

Alumni Contributor(s): Push Singh

Imparting common-sense knowledge to computers enables a new class of intelligent applications better equipped to make sense of the everyday world and assist people with everyday tasks. While previous attempts have been made to acquire and structure common-sense knowledge, they have either been inadequate in capturing the breadth of knowledge needed for the enterprise, or their complicated representation schemes have made them difficult to incorporate into applications. Our approach to this problem is ConceptNet, a freely available common-sense knowledge base that possesses a great breadth of knowledge that can be easily incorporated into applications. Built from the Open Mind Common Sense corpus, which acquires common-sense knowledge from a Web-based community of instructors, ConceptNet is a semantic network of 1.6 million items of common-sense knowledge, and a set of tools for making inferences using this knowledge.

Alumni Contributor(s): Xinyu H. Liu and Push Singh

We have developed technology that enables easy analysis of semantic data, blended in various ways with common-sense world knowledge. The results support reasoning by analogy and association. A packaged library of code is being made available to all sponsors.

One of the biggest challenges for the digital economy is what to do when things go wrong. Orders get misplaced, numbers mistyped, requests misunderstood: then what? Consumers are frustrated by long waits on hold, misplaced receipts, and delays to problem resolution; companies are frustrated by the cost of high-quality customer service. Online companies want customers’ trust, and how a company handles problems directly impacts that. We explore how software agents and other technologies can help with this issue. Borrowing ideas from software debugging, we can have agents help to automate record-keeping and retrieval, track dependencies, and provide visualization of processes. Diagnostic problem-solving can generate hypotheses about causes of errors, and seek information that allows hypotheses to be tested. Agents act on behalf of both the consumer and the vendor to resolve problems more quickly and at lower cost.

Alumni Contributor(s): Earl Wagner, Ethan Mollick and Tom Stocky

Story understanding is a notoriously difficult problem in AI. Broad-spectrum, common-sense knowledge about the world is a good resource, but current common-sense knowledge bases are far from human-level story understanding. We examine affective story understanding in order to perceive the broad emotional overtones of a story at the sentence level, using both a common-sense perspective and the observation that much of the way we emote in response to everyday situations is part of a shared human experience and therefore a part of common sense. With a corpus of common-sense knowledge, we create a semantic network of everyday situations and the emotions associated with them, which, when combined with our linguistic processing, lets our system classify story sentences into six primitive emotions. We then explore how this technology enables innovations in emotional UIs such as EmpathyBuddy, or in prosody, emotional TTS, gaming, story evaluation, and emotional indexing of documents.

Consumer electronics devices are becoming more complicated, intimidating users. These devices do not know anything about everyday life or human goals, and they show irrelevant menus and options. Using common-sense reasoning, we aim to build interfaces with knowledge about the user’s intentions; this knowledge will help the device to display relevant information to reach the user’s goal. For example, an amplifier should suggest a play option when a new instrument is connected, or a DVD player suggest a sound configuration based on the movie it is playing. This will lead to more human-like interactions with these devices.

Alumni Contributor(s): Jose H. Espinosa

This project explores how modern graphical interface techniques and explicit support for the user's problem-solving activities can make more productive interfaces for debugging, which accounts for half the cost of software development. Animated representations of code, a reversible control structure, and instant connections between code and graphical output are some of the techniques used.

Technical documentation for hardware and software is expensive to produce, and often inaccurate and inadequate. We are exploring a new approach to producing technical documentation in which an expert interacts with a simulation of a device, and the system automatically produces both written English descriptions and visual illustrations.

Recognition-based technology has made substantial advances in the past few years because of enhanced algorithms and faster processing speeds. However, current recognition systems are still not reliable enough to be integrated into user interface designs. A possible solution to this problem is to combine results from existing recognition systems and mutually disambiguate the unreliable sections. Piecing together partial results obtained from each mode of recognition can derive more reliable results. In addition, the results of one recognition system can be used to prepare the other recognition system. We are experimenting with an approach that uses a software agent to integrate off-the-shelf recognition applications via scripting languages. We use a software agent called MARCO (Multimodal Agent for Route Construction) that utilizes multiple recognition systems to assist users in giving directions for urban navigation.

This project aims to collect common-sense knowledge in languages other than English. We have launched sites in Portuguese and French, and we aim to launch Arabic and Dutch sites in the near future.

Alumni Contributor(s): Hyemin Chung, Jose H. Espinosa, Wonsik Kim and Yu-Te Shen

How would you browse a VERY large display space, such as a street map of the entire world? The traditional solution is zoom and pan, but these operations have drawbacks that have gone unchallenged for decades. Shifting attention loses the wider context, leading to that "lost in hyperspace" feeling. We are exploring alternative solutions, such as a new technique that allows zooming and panning in multiple translucent layers.

The biggest problem facing artificial intelligence today is how to teach computers enough about the everyday world so that they can reason about it like we do—so that they can develop "common sense." We think this problem may be solved by harnessing the knowledge of people on the Internet, and we have built a Web site to make it easy and fun for people to work together to give computers the millions of little pieces of ordinary knowledge that constitute "common sense." Teaching computers how to describe and reason about the world will give us exactly the technology we need to take the Internet to the next level, from a giant repository of Web pages to a new state where it can think about all the knowledge it contains; in essence, to make it a living entity.

Alumni Contributor(s): Push Singh

The Open Mind Common Sense project has collected hundreds of thousands of statements of common-sense knowledge from volunteers on the Internet, using a variety of online activities in several different languages. Open Mind Commons aims to use analogical reasoning to make connections between similar ideas while highlighting the relevant differences as well. These analogies can give a computer a better understanding of the relationships between objects, situations, and cultures. It is often difficult to search through and coordinate lexical information across data sources, each of which has its own separate interface and viewing software. We have approached this problem by creating a unified, flexible interface for various natural-language processing resources.

Alumni Contributor(s): Hyemin Chung

Words mean different things to different people, and capturing these differences is often a subtle art. This project is for the development of a system for discovering distinct communities of people with distinct jargon usage or belief structures from simple rating data on common sense knowledge. PerspectiveSpace is an approach whereby elementary linear operations are used to perform calculations on user models and microtheories.

We want to build programming systems that can converse with their users to build computer programs. Such systems will enable users without programming expertise to write programs using natural language. The text-based, virtual-world environments called the MOO (multi-user, object-oriented Dungeons and Dragons) allow their users to build objects and give them simple, interactive, text-based behaviors. These behaviors allow other participants in the environment to interact with those objects by invoking actions and receiving text messages. Through our natural-language dialog system, the beginning programmer will be able to describe objects and the messages in MOO environments.

Today, people can tell stories by composing, manipulating, and sequencing individual media artifacts using digital technologies. However, these tools offer little help in developing a story's plot. Specifically, when a user tries to construct her story based on a collection of individual media elements (videos, audio samples), current technological tools do not provide helpful information about the possible narratives that these pieces can form. Storied Navigation is a novel approach to this problem; media sequences are tagged with free-text annotations and stored as a collection. To tell a story, the user inputs a free-text sentence and the system suggests possible segments for a storied succession. This process iterates progressively, helping the user to explore the domain of possible stories. The system achieves the association between the input and the segments' annotations using reasoning techniques that exploit the WordNet semantic network and common-sense reasoning technology.

Alumni Contributor(s): Barbara Barry and Push Singh

Our project, ToDoGo, learns by observing patterns of the user's location to answer the questions "What should I do?", "Where am I going", and "How do I get there?". For example, if you need to send a letter, you'll typically do this at a post office or drop box. ToDoGo uses common-sense knowledge and data about local points of interests to associate everyday activities with locations in the Boston area. From this, ToDoGo provides a wide range of applications, including a just-in-time reminder, activity/destination/path recommendations, path optimization, and distributed household management.

Because common-sense knowledge differs culturally, misunderstandings frequently occur. Because differences can be subtle, there has been little work in trying to detect places in text where cultural differences might arise. We explicitly represent the common-sense knowledge of each culture in separate knowledge bases. By analyzing a text, we can find differences between each culture's knowledge concerning its subject. For example, given an invitation to a party, the system is able to infer that in an American cultural context, hip-hop dancing might be expected, but in a Mexican context, salsa dancing might be the norm. We are building an email client that suggests knowledge from multiple cultures that might be relevant, while watching the user's typing.

Alumni Contributor(s): Jose H. Espinosa