Research on Sensing Human Affect

Sensors are an important part of an Affective Computing System because they provide information about the wearer's physical state or behavior. They can gather data in a continuous way without having to interrupt the user. The emphasis on this page is on describing physiological sensors; however, we are interested in many kinds of new sensors that might be useful in recognizing affective cues. Several projects in sensing affect are listed below.

A Prototype Physiological Sensing System had been developed for use on a UNIX based operating system. The system shown below includes a Galvanic Skin Response (GSR) Sensor, a Blood Volume Pulse (BVP) sensor, a Respiration sensor and an Electromyogram (EMG). This system was adopted for the following reasons: The sensors are available through a commercial manufacturer who has obtained FDA approval for use of these sensors on human subjects; the system is lightweight and portable and relatively robust to changes in user position; the system is entirely on the user which helps maintain the privacy of the wearer's data. Each of the sensors is explained below and a sample of the data taken from each sensor is shown for clarity.

The prototypical sensing system shown here is a combination of four sensors: The Blood Volume Pulse (BVP) sensor, the Galvanic Skin Response (GSR) sensor, the Electromyogram (EMG) sensor, and the respiration sensor. Each of these sensors is described in detail below. The sensors all attach to the Thought Technology ProComp Encoder Unit, a device that receives the signals and translates them into digital form. The output from the ProComp unit then ports to a computer for processing and recognition.

---

The Galvanic Skin Response (GSR) Sensor

Galvanic Skin Response is a measure of the skin's conductance between two electrodes. Electrodes are small metal plates that apply a safe, imperceptibly tiny voltage across the skin. The electrodes are typically attached to the subject's fingers or toes using electrode cuffs (as shown on the left electrode in the diagram) or to any part of the body using a silver-Chloride electrode patch such as that shown on the EMG. To measure the resistance, a small voltage is applied to the skin and the skin's current conduction is measured.

Skin conductance is considered to be a function of the sweat gland activity and the skin's pore size. An individual's baseline skin conductance will vary for many reasons, including gender, diet, skin type and situation. Sweat gland activity is controlled in part by the sympathetic nervous system. When a subject is startled or experiences anxiety, there will be a fast increase in the skin's conductance (a period of seconds) due to increased activity in the sweat glands (unless the glands are saturated with sweat.)

After a startle, the skin's conductance will decrease naturally due to reabsorption. There is a saturation to the effect: when the duct of the sweat gland fills there is no longer a possibility of further increasing skin conductance. Excess sweat pours out of the duct. Sweat gland activity increases the skin's capacity to conduct the current passing through it and changes in the skin conductance reflect changes in the level of arousal in the sympathetic nervous system.

A graph of Galvanic Skin Response (GSR) skin conductance over a 27-minute period during an experiment. Increased GSR indicates a heightened sympathetic nervous system arousal.

---

The Blood Volume Pulse Sensor

The Blood Volume pulse sensor uses photoplethysmography to detect the blood pressure in the extremities. Photoplethysmography is a process of applying a light source and measuring the light reflected by the skin. At each contraction of the heart, blood is forced through the peripheral vessels, producing engorgement of the vessels under the light source--thereby modifying the amount of light to the photosensor. The resulting pressure waveform is recorded.

A graph of Blood Volume Pulse (BVP) sensor output showing a waveform of human heart beats, measured over a 27-minute period during an experiment. Note in this signal how the envelope (the overall shape of the waveform) of the BVP "pinches" as the subject is startled in this diagram, at around the 16-18 minute period.



Three typical heart beats, as measured in the fingertips by the Blood Volume Pulse sensor.

Since vasomotor activity (activity which controls the size of the blood vessels) is controlled by the sympathetic nervous system, the BVP measurements can display changes in sympathetic arousal. An increase in the BVP amplitude indicates decreased sympathetic arousal and greater blood flow to the fingertips.

---

The Respiration Sensor

The respiration sensor can be placed either over the sternum for thoracic monitoring or over the diaphram for diaphragmatic monitoring. In all experiments so far we have used diaphragmatic monitoring. The sensor consists mainly of a large velcro belt which extends around the chest cavity and a small elastic which stretches as the subject's chest cavity expands. The amount of stretch in the elastic is measured as a voltage change and recorded. From the waveform, the depth the subject's breath and the subject's rate of respiration can be learned.

A typical respiration waveform reading over approximately 27 minutes.

---

The Electromyogram (EMG) Sensor

The electromyographic sensors measure the electromyographic activity of the muscle (the electrical activity produced by a muscle when it is being contracted), amplify the signal and send it to the encoder. In the encoder, a band pass filter is applied to the signal. For all our experiments, the sensor has used the 0-400 microvolt range and the 20-500 Hz filter, which is the most commonly used position.

An electromyogram of jaw clenching during an experiment.

Projects in Sensing Affective Signals:

• AffQuake
• Affective Carpet
• Affective Tangibles
• Detection and Analysis of Driver Stress
• The Galvactivator
• Robotic Computer