The Memory Glasses Project

MIThril, a borglab production. Richard W. DeVaul, Jonathan Gips, Michael Sung, Sandy Pentland
MIThril home


MIThril wiki

Anduin window manager

MIThril Code

hardware design

project overview

The MIThril vision

MIThril diagrams

MIThril people

MIThril photos

Memory Glasses

MIThril Enchantment Software

Real-Time Contxt Engine


Wearables Home Page

The Memory Glasses is a wearable, proactive, context-aware memory aid based on the MIThril platform and wearable sensors. The primary goal of this project is to produce an effective memory aid and reminder system that requires a minimum of the wearer's attention. Our Memory Glasses paper that was published in ISWC 2003 describes recent research on the use of subliminal visual cues for memory support.

The function of our system is to deliver reminders to the wearer in a timely, situation-appropriate way, without requiring intervention on the part of the wearer beyond the initial request to be reminded. In other words, the system behaves like a reliable human assistant that remembers reminder requests and delivers them under appropriate circumstances. Such a system is qualitatively different from a passive reminder system (such as a paper organizer) or a context blind reminder system (a modern PDA) which records and structures reminder requests but which can not know the user's context.

The Problem to be Addressed, or the Limitations of Context Blind Systems

The Problem

People's lives are increasingly complicated, and the need for effective logistical and memory aids is on the rise. For years, digital technology and computers have promised effective solutions for the problems of organization and memory, but present technology offers little more than an incremental improvement over the paper scheduler and alarm clock of the 19th century. Even the most sophisticated PDA is deaf and blind, and knows no more about the wearer's context than the time of day.

The Memory Glasses Solution, a Proactive Context-Aware Memory Aid

The Solution

The solution we propose is a proactive, context-aware memory aid called the Memory Glasses By creating a reminder system that is context aware, the role of the system is transformed from a blind, passive organizer to a proactive, perceptive entity akin to a human assistant. The delivery of information is situation conditioned; reminders are associated with context and are delivered when and where appropriate.

Open Questions in the Design and Implementation of the Memory Glasses

Open Questions

There are a number of open questions to be addressed in the construction of a proactive, context aware reminder system. These questions may be broken down as follows:
  • Defining Useful Context What is useful context for proactive reminders? Assuming that we cannot know everything about the wearer's state and actions, what are the most important things to know? Of these, which are technologically feasible to sense and classify?

    Time and location are obviously useful. In addition, it may be useful to know socially or logistically important features of the wearer's activity state, such as "in a conversation with person X" or "driving to work."

  • Context Sensing and Classification How do we sense and classify useful context? What types of sensors, signal processing, and inference techniques are necessary? What are the bandwidth and computational requirements for this classification?

    Time and location are relatively easy to sense and classify. Aspects of the wearer's activity state which are independent of time and location are harder. High-bandwidth, computationally expensive computer perception techniques may be used to provide infrastructure-free sensing and classification, and low-bandwidth low-computing power tag readers and tags may be used in cases where tagging infrastructure, people, and objects is feasible. Increasingly, we are seeing the value of medium-bandwidth sensing and real-time classification of signals such as accelerometer data, as described in the MIThril Real-Time Context Engine page

    A white paper discussing a preliminary implementation of a high-bandwidth context sensing Memory Glasses implementation that predates MIThril is available in HTML and gzip'd PostScript. A brief description of our current low-bandwidth MIThril implementation is described below.

  • Wearer Interaction Humans have limited bandwidth, and a proactive information delivery system must always assume that it is a secondary task. One of the hardest questions is how to present information to the wearer in a way that supports their interaction with and involvement in the world around them. Arguably this is the most important question, since no amount of sophistication and effectiveness in context definition, classification and sensing can be effective if the wearer refuses to use it.

    Bad interactions on the desktop are annoying, bad interactions on a wearable can be life-threatening. If distracted at the wrong time (crossing the street, driving a car, etc.) the wearer's life may be placed in real physical danger by an ill-timed, distracting interaction. In addition, different wearers will have different preferences, primary tasks, perception limitations and levels of cognitive function. (See Medical Applications, below.)

    Our recent research suggests that it is possible to support memory recall without the user's conscious attention. The research behind this exciting possibility is described in the Memory Glasses paper that was published in ISWC 2003.

Medical Application Classes

Classes of Medical Application

One strong area of interest in our research is medical applications for the Memory Glasses proactive context-aware reminder system. We believe that almost anyone could benefit from an appropriate, well implemented Memory Glasses system, but those who stand to gain the most are people with substantial memory (amnesia) or recognition (agnosia) problems. In the following section we outline five classes of possible wearers, ranging from the substantively unimpaired to the significantly cognitively or perceptually impaired wearer. Each class presents different challenges and opportunities.

Memory Glasses Implementation for Substantively Unimpaired Wearers

Substantively Unimpaired Wearers

The substantively unimpaired wearer (a healthy young person, for instance) who leads a busy life may significantly benefit from artificial memory and logistical aids, as the author can confirm from personal experience. The challenge this group of wearers presents is to build a system which provides a significant benefit beyond what is provided by innate memory alone and innate memory in combination with context-blind memory aids.

We believe that the key features of a successful Memory Glasses implementation is sufficient range in context classification and well-designed interaction. To be useful to unimpaired users, the Memory Glasses must function over a broad range of circumstances which complicates the definition and sensing of relevant context. At the very least, the Memory Glasses must do no worse than a conventional context-blind reminder system in situations where the context is ambiguous or unknown (which is to say, it should not get in the way) and work reliably in at least some socially or occupationally important circumstances.

Offsetting the demanding scope of this application class, the unimpaired cognitive and perceptual capabilities of the wearer will provide the maximum adaptation and accommodation to the additional "channel" of input, imposing the fewest constraints on the interaction design.

Memory Glasses Implementation in the Context of Normal Aging

Normal Older People

In normal aging, the learning of new material may become more difficult and the time to recall learned material may increase, but memory loss per se (amnesia) is not normal. (This is the result of other diseases, most often Alzheimer's; see the discussion of amnesia below.) A normal older person might rely on memory aids somewhat more than an unimpaired younger person, but the high-level cognitive and perceptual capabilities of normal older and younger people are comparable.

Decreased acuity in vision and hearing resulting from the normal aging process may impose somewhat tighter technical constraints on the interaction design, and increased learning time may require a longer training and adaptation period. Although a normal older person (suffering from increased learning and recall time) might have more to gain from a Memory Glasses system than a normal younger one, persuading an older person to wear such a system may be difficult, especially if they have developed effective coping strategies using conventional memory aids.

Memory Glasses Implementation in the Context of Amnesia, including early onset Alzheimer's

Amnesia Patients

Amnesia is loss of memory resulting from brain damage (physical trauma or disease) or severe emotional trauma. Amnesia may occur suddenly or slowly, may effect short-term or long term memory, and may be permanent or temporary depending on the cause. Effective treatment of amnesia depends almost entirely on the cause; amnesia resulting from some types of physical trauma or disease, such as amnesia resulting from the pressure of tumors or swelling brain tissue, may be effectively treated surgically or with drugs. Amnesia resulting from emotional trauma, usually related to a specific event (anterograde amnesia), may be effectively treated by addressing the underlying emotional cause of the trauma. Other types of amnesia, such as that resulting from Alzheimer's disease (also known as primary degenerative dementia) are progressive and incurable.

We believe that patients with chronic short-term memory problems, such as those resulting from certain types of head injuries and the early stages of Alzheimer's, may significantly benefit from a proactive reminder system. However, the high-level cognitive and perceptual problems that often accompany amnesia symptoms may seriously restrict the class of patient likely to benefit.

An Active-Tag Based Implementation of the Memory Glasses for Amnesia Patients

An active-tag based implementation of the Memory Glasses is feasible in the case where context is circumscribed and the more-or-less comprehensive tagging of important locations, objects and people is feasible. Such a setting is often present in the institutionalized care of amnesia patients, and an active-tag based implementation greatly simplifies the context sensing and classification problem. If an effective Memory Glasses implementation could be constructed for this patient group and setting, it might provide significant advantages to both patients and care providers by increasing patient independence and reducing required supervision.

Memory Glasses Implementation in the Context of Agnosia and Prosopagnosia

Agnosia (Prosopagnosia) Patients

Agnosia is the inability to recognize objects through the senses, "a normal perception stripped of its meaning." Agnosia may be subdivided into two types: associative agnosia is a failure of recognition (association with memory) given a properly integrated stimulus, and apperceptive agnosia is a disturbance of the integration of otherwise normally perceived components of a stimulus (disturbances in perceived orientation, inability to make comparisons and matches, etc.). Agnosia typically effects only a single sensory modality, vision being the most common. A particularly famous (and clinically important) form of visual Agnosia is prosopagnosia which is the inability to recognize faces. To be properly called agnosia, the recognition problems must be associated with normal or near-normal functioning in other aspects of high-level cognition and perception.

Agnosia patients in general and prosopagnosia patients in particular present an interesting application class because the impairment is so specific. A reliable face recognizer could almost literally replace the damaged brain functionality of the prosopagnosia patient, perhaps allowing the patient to lead a normal or near-normal life.

The modality and type of agnosia has interesting implications for the Memory Glasses interaction design, since the role of the application becomes the recognition of and translation from an unrecognizable perception into a recognizable one. For instance, an implementation for associative prosopagnosia might present the wearer with both a picture of the recognized face and the name, so that the wearer might compare the picture with the person and confirm recognition through visual comparison. However, an implementation for apperceptive prosopagnosia might present the wearer with name and a recorded sample of the recognized person's voice, so that the wearer could confirm the identification aurally.

Memory Glasses Implementation and Anomia, etc.

Patients with Other Disorders

It is possible that other disorders of memory and recognition could be ameliorated through a context-aware reminder system. For instance, a tag-reader based application which supplied the names of tagged objects to a sufferer of anomia, an impairment in naming and word choice that is associated with all forms of aphasia and present to some degree in the production of normal speech (the "tip of the tongue" feeling). However, anomia effects all word choice and such an aid would only be useful for objects immediately at hand.

An Active-Tag based Prototype Amnesia/Agnosia Memory Glasses Implementation

Prototype Memory Glasses Implementation

We have constructed an active-tag based amnesia/agnosia Memory Glasses implementation using a vest-based MIThril system and the "squirt" active IR tags and tag reader depicted on the MIThril photos page. This is a very simple application which allows images and text to be associated with tagged locations and people (or objects). For instance, a picture of Sandy Pentland and his name might be mapped to tag ID# "seen" by the forward-looking IR detector mounted on the MicroOptical clip-on display. A location, the dev-tty1 workstation in the borglab, might be mapped to tag ID# seen by the upward looking detector.

Overt Cuing

Overt cues may be associated with people, locations, times, or combinations of these contexts. If overt cues (in the form of pictures) were associated with both Sandy Pentland and the dev-tty1 workstation, a user at dev-tty1 talking to Sandy might see the following on the head mounted display:

Image and text output (overt cues) for the amnesia/agnosia application, actual resolution Sandy Pentland
dev-tty1, borglab Sandy Pentland
  superimposed on one small part of the visual field of one eye. Specific reminder text (beyond place and person names), other images, or auditory content could be associated as well. Even if the cues are nothing more than pictures and names, this application could be a significant aid to a visual agnosia patient, providing them the means to recognize the locations and people around them.

Subliminal Cuing Memory Glasses

Subliminal cues can also be associated with contexts such as locations and people. These cues, in the form of images or text, are triggered just like the overt cues but presented in such a way as to fall below the threshold of conscious perception. None the less, our research suggests that these cues can substantially and significantly improve performance on a memory recall task (an improvment of about 1.5 compared to the uncued control, p=0.02 one-tailed student's T). Perhaps more important than the positive effect of correct cuing, our research suggests that incorrect or misleading subliminal cues do not interfere with memory recall. This is in stark contrast to the effect of overt miscues, which appear to have a large misleading effect. These results are discussed further in our ISWC 2003 paper.

Conclusions and Future Work

Our work on the use subliminal visual cues and real-time context classification continues. We believe that if our findings are supported by future research, the use of subliminal cues for proactive memory support could find wide application in a range of task support applications.

Richard W. DeVaul,     Tue Oct 28 16:05:43 EST 2003