Software Agents

Current experiments in agent software rely mostly on domain-specific applications that either have been programmed from scratch, or explicitly modified with an agent in mind. Is it possible to make a tool kit or protocol that would allow an agent to communicate and to control applications that have been constructed more conventionally? Can the agent "take the place" of the user in the interface? Can the agent have access to the application's data and behavior? Will commercial "inter-application communication" mechanisms suffice? What is the division of labor between the agent and the application? This work will explore these questions.

Effective image annotation and retrieval is bound up with image use. In this project, annotation, retrieval, and use are integrated, facilitating the finding and using images. A proactive user-interface agent seeks chances for image annotation and retrieval in the context of the user's everyday work, using an agent that sit in the user's text editor or other application and continuously monitors typing. Searches are automatically performed from an image library, and images relevant to the current text can be inserted in a single operation. Descriptions of images for storytelling can be seamlessly employed as raw material for annotation. Common-sense knowledge about situations in which pictures are taken, described, or used can help provide semi-automatic annotation and indirect inference for retrieval. Our approach does not completely automate the annotation/retrieval process, but it does reduce user-interface overhead, leading to better-annotated image libraries and fewer missed opportunities for image use.

AnalogySpace enables common-sense reasoning through principal component analysis. It projects the information in ConceptNet into a reduced-dimensional space that describes common-sense concepts and their properties in terms of automatically discovered correlations called axes. AnalogySpace can be used to infer new information, reason about ad hoc categories, detect topics in text, and compare concepts on scales that can be generated on the fly. We can then use a technique called Blending to create such a space, integrating any other type of free text or structured information with the common sense. A blend can help add common-sense intuition into a wide variety of tasks.

The Open Mind Common Sense project collects its knowledge base from ordinary people. Acquiring useful knowledge from untrained volunteers requires asking them the right questions and keeping them interested, and we use a variety of online interfaces and games to do so. We present some of these interfaces that enable people to teach computers what they know.

This project aims to develop an intelligent personal-finance advisory agent that bridges the gap between the novice user and the expert model of the finance domain. The agent uses common-sense reasoning and inference for associating the user's personal life, financial situation, and goals with the attributes of the expert domain model and vice versa. The agent interface provides a natural-language interface for elicitation and explanations of design and process rationale. The architecture of the system is domain-independent and consequently can be used for any novice-expert domain model.

A long-standing dream of artificial intelligence has been to put common-sense knowledge into computers–enabling machines to reason about everyday life. Some projects, such as Cyc, have begun to amass large collections of such knowledge. However, it is widely assumed that the use of common sense in interactive applications will remain impractical for years, until these collections can be considered sufficiently complete, and common-sense reasoning sufficiently robust. Recently we have had some success in applying common-sense knowledge in a number of intelligent interface agents, despite the admittedly spotty coverage and unreliable inference of today's common-sense knowledge systems.

Common sense enables us to build innovative recommendation systems that are more interactive and user-friendly than traditional collaborative filtering systems. By applying ideas from blending and PerspectiveSpace to recommendations, we discover the characteristics of products that drive user ratings. We can use these characteristics to build intelligent recommendation agents and effective product exploration tools.

We have developed, Common Consensus: a fun, self-sustaining web-based game, that both collects and validates Commonsense knowledge about everyday goals. Goals are a key element of commonsense knowledge; in many of our inferface agents, we need to recognize goals from user actions (plan recognition), and generate sequences of actions that implement goals (planning). We also often need to answer more general questions about the situations in which goals occur, such as when and where a particular goal might be likely, or how long it is likely to take to achieve.

Imparting common-sense knowledge to computers enables a new class of intelligent applications better equipped to make sense of the everyday world and assist people with everyday tasks. While previous attempts have been made to acquire and structure common-sense knowledge, they have either been inadequate in capturing the breadth of knowledge needed for the enterprise, or their complicated representation schemes have made them difficult to incorporate into applications. Our approach to this problem is ConceptNet, a freely available common-sense knowledge base that possesses a great breadth of knowledge that can be easily incorporated into applications. Built from the Open Mind Common Sense corpus, which acquires common-sense knowledge from a Web-based community of instructors, ConceptNet is a semantic network of 1.6 million items of common-sense knowledge, and a set of tools for making inferences using this knowledge.

When programmers face unfamiliar or challenging tasks, code written by others could give them inspiration or reusable pieces. But how can they find appropriate code? We believe the key to meaningful code search is to relate code (written in a programming language) to the purpose for which the code was written (expressed in natural language). Our interface enables searching for, reusing, and sharing code fragments within the development environment by connecting code with a brief English description of what it is for. To understand users' descriptions of the purpose of the code they have written or want to find, we reason jointly over code structure features, code linguistic features (e.g., variable and event names), and both domain-specific and common-sense knowledge. We show how the system suggests possible purposes for unannotated code and finds potentially applicable code given brief English purpose statements.

One of the biggest challenges for the digital economy is what to do when things go wrong. Orders get misplaced, numbers mistyped, requests misunderstood: then what? Consumers are frustrated by long waits on hold, misplaced receipts, and delays to problem resolution; companies are frustrated by the cost of high-quality customer service. Online companies want customers’ trust, and how a company handles problems directly impacts that. We explore how software agents and other technologies can help with this issue. Borrowing ideas from software debugging, we can have agents help to automate record-keeping and retrieval, track dependencies, and provide visualization of processes. Diagnostic problem-solving can generate hypotheses about causes of errors, and seek information that allows hypotheses to be tested. Agents act on behalf of both the consumer and the vendor to resolve problems more quickly and at lower cost.

Story understanding is a notoriously difficult problem in AI. Broad-spectrum, common-sense knowledge about the world is a good resource, but current common-sense knowledge bases are far from human-level story understanding. We examine affective story understanding in order to perceive the broad emotional overtones of a story at the sentence level, using both a common-sense perspective and the observation that much of the way we emote in response to everyday situations is part of a shared human experience and therefore a part of common sense. With a corpus of common-sense knowledge, we create a semantic network of everyday situations and the emotions associated with them, which, when combined with our linguistic processing, lets our system classify story sentences into six primitive emotions. We then explore how this technology enables innovations in emotional UIs such as EmpathyBuddy, or in prosody, emotional TTS, gaming, story evaluation, and emotional indexing of documents.

Consumer electronics devices are becoming more complicated, intimidating users. These devices do not know anything about everyday life or human goals, and they show irrelevant menus and options. Using common-sense reasoning, we are building a system, Roadie, with knowledge about the user's intentions; this knowledge will help the device to display relevant information to reach the user's goal. For example, an amplifier should suggest a play option when a new instrument is connected, or a DVD player suggest a sound configuration based on the movie it is playing. This will lead to more human-like interactions with these devices. We have constructed a Roadie interface to real consumer electronics devices: a television, set top box, and smart phone. The devices communicate over Wi-Fi, and use the UPnP protocols.

This project explores how modern graphical interface techniques and explicit support for the user's problem-solving activities can make more productive interfaces for debugging, which accounts for half the cost of software development. Animated representations of code, a reversible control structure, and instant connections between code and graphical output are some of the techniques used.

We examined a large corpus of people's goals and their descriptions of accomplishing them. We asked the questions: which goals are most popular, most controversial, easiest to accomplish? From this corpus we inferred a goal network that can be used for plan recognition, story understanding, and building software agents that are aware of their users' goals.

Technical documentation for hardware and software is expensive to produce, and often inaccurate and inadequate. We are exploring a new approach to producing technical documentation in which an expert interacts with a simulation of a device, and the system automatically produces both written English descriptions and visual illustrations.

It's well known that living in a foreign country dramatically improves the effectiveness of learning a second language over classroom study alone. This is likely because people make associations with the foreign language as they see and participate in everyday life activities. We are designing language-teaching sequences for a sensor-equipped residence that can detect user interaction with household objects. We use our common-sense knowledge base and reasoning tools to construct teaching sequences, wholly in the target language, of sentences and question-answering interactions that gradually improve the learner's language competence. For example, the first time the user sits in a chair, the system responds with the foreign-language word for "chair," and later with statements and questions such as, "You sit in the chair" (complete sentence), "You sat in the chair" (tenses), "What is the chair made of?" (question, materials), or "Why are you sitting in the chair?" (goals, plans).

Recognition-based technology has made substantial advances in the past few years because of enhanced algorithms and faster processing speeds. However, current recognition systems are still not reliable enough to be integrated into user interface designs. A possible solution to this problem is to combine results from existing recognition systems and mutually disambiguate the unreliable sections. Piecing together partial results obtained from each mode of recognition can derive more reliable results. In addition, the results of one recognition system can be used to prepare the other recognition system. We are experimenting with an approach that uses a software agent to integrate off-the-shelf recognition applications via scripting languages. We use a software agent called MARCO (Multimodal Agent for Route Construction) that utilizes multiple recognition systems to assist users in giving directions for urban navigation.

Moral Compass is a model of how children learn in a problem-solving environment where the child is learning to accomplish goals in the context of parents, strangers, and cultural knowledge. The child learns in multiple ways: playing alone, being told stories, and being rewarded or punished. Our model aims to provide an explanation for relatively complex reflective states of mind, such as desire, avoidance, focus, Ignorance, and personality traits. Our model also emphasizes different types of failure in its reflective approach to learning, including: surprise, disappointment, and guilt. Possible applications include better understanding of the mental health of cognition in social domains.

A ConceptNet in English is already established and working well. We are now attempting to expand it to other languages and cultures. This project is an extended ConceptNet with Korean common sense, which is fundamentally different from English. Through this project, we can learn how to expand the ConceptNet into other languages and how to connect them. By connecting English and Korean ConceptNets, we are hoping not only to see cultural or linguistic differences, but also to solve problems such as the ambiguity of multivocal words, which were difficult to solve with only one ConceptNet.

This project aims to collect and reason over common-sense knowledge in languages other than English. We have collected large bodies of common-sense knowledge in Portuguese and Korean, and we are expanding to other languages such as Spanish, Dutch, and Italian. We can use techniques based on AnalogySpace to discover correlations between languages, enabling our knowledge bases in different languages to learn from each other.

How would you browse a VERY large display space, such as a street map of the entire world? The traditional solution is zoom and pan, but these operations have drawbacks that have gone unchallenged for decades. Shifting attention loses the wider context, leading to that "lost in hyperspace" feeling. We are exploring alternative solutions, such as a new technique that allows zooming and panning in multiple translucent layers.

The Open Mind Common Sense project has collected hundreds of thousands of statements of common-sense knowledge from volunteers on the Internet, using a variety of online activities in several different languages. Open Mind Commons aims to use analogical reasoning to make connections between similar ideas while highlighting the relevant differences as well. These analogies can give a computer a better understanding of the relationships between objects, situations, and cultures. It is often difficult to search through and coordinate lexical information across data sources, each of which has its own separate interface and viewing software. We have approached this problem by creating a unified, flexible interface for various natural-language processing resources.

We want to build programming systems that can converse with their users to build computer programs. Such systems will enable users without programming expertise to write programs using natural language. The text-based, virtual-world environments called the MOO (multi-user, object-oriented Dungeons and Dragons) allow their users to build objects and give them simple, interactive, text-based behaviors. These behaviors allow other participants in the environment to interact with those objects by invoking actions and receiving text messages. Through our natural-language dialogue system, the beginning programmer will be able to describe objects and the messages in MOO environments.

When presenting or editing a large set of material, such as photos or video captured in daily life, it is not easy to understand how particular scenes fit into the overall intent of a story. For example, the goal of the story in the captured content is usually missing, and there is a lack of coherent connections between materials; issues such as these make it difficult for viewers to follow the story. We present Raconteur, a story-editing system that helps users think about material in a story by showing related scenes, or other stories with similar goals. We suggest that presenting analogous stories may provide a guideline for users to tell their story in a similar pattern.

Analogy is a powerful comparison mechanism, commonly thought to be central to human problem solving. Analogies like "an atom is like the solar system" enable people to effectively transfer knowledge to new domains. Can we enable computers to do similar comparisons? Prior work on analogy (structure mapping) provides guidance about the nature of analogies, but implementations of these theories are inefficient and brittle. We are working on a new analogy mechanism that uses instance learning to make robust, efficient comparisons.

Today, people can tell stories by composing, manipulating, and sequencing individual media artifacts using digital technologies. However, these tools offer little help in developing a story's plot. Specifically, when a user tries to construct her story based on a collection of individual media elements (videos, audio samples), current technological tools do not provide helpful information about the possible narratives that these pieces can form. Storied Navigation is a novel approach to this problem; media sequences are tagged with free-text annotations and stored as a collection. To tell a story, the user inputs a free-text sentence and the system suggests possible segments for a storied succession. This process iterates progressively, helping the user to explore the domain of possible stories. The system achieves the association between the input and the segments' annotations using reasoning techniques that exploit the WordNet semantic network and common-sense reasoning technology.

The Internet has become a stream of information, and understanding that stream is increasingly important. How do we understand what is being talked about and what emotions are being expressed? How do world events change people's views, emotions, and goals? TwitterMap is a visualizer which can help depict how the world's mental model shifts with different world events.