Methods and apparatus for controlling a user interface based on the emotional state of a user

• US 7,665,024 B1
• Filed: 07/22/2002
• Issued: 02/16/2010
• Est. Priority Date: 07/22/2002
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Verizon Services Corporation (Verizon Communications Inc.)
• Inventors: Kondziela, James Mark
• Primary Examiner(s): Pitaro; Ryan F
• Application Number:
  US10/200,898
• Time in Patent Office: 2,766 Days
• Field of Search: 715/707, 715/728
• US Class Current: 715/745
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices

Abstract

Methods and apparatus for modifying a user interface as a function of the detected emotional state of a system user are described. In one embodiment, stress analysis is performed on received speech to generate an emotional state indicator value, e.g., a stress level indicator value. The stress level indicator value is compared to one or more thresholds. If a first threshold is exceeded the user interface is modified, e.g., the presentation rate of speech is slowed. If a second threshold is not exceeded, another modification to the user interface is made, e.g., the speech presentation rate is accelerated. If the stress level indicator value is between first and second thresholds, user interface operation continues unchanged. The user interface modification techniques of the present invention may be used in combination with known knowledge or expertise based user interface adaptation features.

Claims

1. A method controlling an apparatus, the method comprising the steps of:

modifying a user interface in response to detecting that an emotional state indicator value exceeds a first threshold, including slowing the rate at which prompts are provided to a user of said apparatus, the method further comprising, prior to performing said modifying step;

generating said emotional state indicator value from user input; and

comparing said emotional state indicator value to said first threshold to detect if said emotional state indicator value is above said first threshold.

2. A method controlling an apparatus, the method comprising the steps of:

modifying a user interface in response to detecting that an emotional state indicator value exceeds a first threshold, including increasing the length of a pause between sentences included in speech generated by said user interface, the method further comprising, prior to performing said modifying step;

generating said emotional state indicator value from user input; and

comparing said emotional state indicator value to said first threshold to detect if said emotional state indicator value is above said first threshold, wherein said emotional state indicator is a stress level value, the step of generating an emotional state indicator value including;

performing stress analysis on received speech to generate said stress level value.

3. A method controlling an apparatus, the method comprising the steps of:

modifying a user interface in response to detecting that an emotional state indicator value exceeds a first threshold, including reducing the number of options presented to a user as part of a menu from which the user can make a selection, the method further comprising, prior to performing said modifying step;

generating said emotional state indicator value from user input; and

comparing said emotional state indicator value to said first threshold to detect if said emotional state indicator value is above said first threshold, wherein said emotional state indicator is a stress level value, the step of generating an emotional state indicator value including;

performing stress analysis on received speech to generate said stress level value.

4. A method controlling an apparatus, the method comprising the steps of:

modifying a user interface in response to detecting that an emotional state indicator value exceeds a first threshold, including slowing the rate at which prompts are provided to a user of said apparatus;

increasing the length of a pause between sentences included in speech generated by said user interface; and

reducing the number of options presented to a user as part of a menu from which the user can make a selection;

the method further comprising, prior to performing said modifying step;

generating said emotional state indicator value from user input; and

comparing said emotional state indicator value to said first threshold to detect if said emotional state indicator value is above said first threshold, wherein said emotional state indicator is a stress level value, the step of generating an emotional state indicator value including;

performing stress analysis on received speech to generate said stress level value.

5. A method controlling an apparatus, the method comprising the steps of:

modifying a user interface in response to detecting that an emotional state indicator value exceeds a first threshold, the method further comprising, prior to performing said modifying step;

receiving speech from a user;

generating said emotional state indicator value from said received speech; and

comparing said emotional state indicator value to said first threshold to detect if said emotional state indicator value is above said first threshold;

the method further comprising the steps of;

periodically performing said generating step using speech input received during different periods of time to generate a set of emotional state indicator values, each emotional state indicator value corresponding to a different period of time; and

monitoring said emotional state indicator values to determine if the generated emotional state indicator values stay above a second threshold for more than one period of time.

7. The method of claim 5, wherein said apparatus is a telephone call servicing apparatus, the method further comprising the step of:

connecting a caller from which said speech input is received to a human service representative when it is determined that the generated emotional state indicator values stay above said second threshold for said more than one period of time.

8. The method of claim 7, wherein said telephone call servicing apparatus is a telephone operator terminal.

6. A method controlling an apparatus, the method comprising the steps of:

modifying a user interface in response to detecting that an emotional state indicator value exceeds a first threshold, the method further comprising, prior to performing said modifying step;

receiving speech from a user;

generating said emotional state indicator value from said received speech; and

comparing said emotional state indicator value to said first threshold to detect if said emotional state indicator value is above said first threshold;

the method further comprising the steps of;

periodically performing said generating step using speech input received during different periods of time to generate a set of emotional state indicator values, each emotional state indicator value corresponding to a different period of time; and

connecting a caller from which said speech input is received to a human service representative when it is determined that one of the generated emotional state indicator values exceed a second threshold for more than one period of time.

9. An apparatus used to interact with a user of said apparatus, the apparatus including:

a user interface including;

means for detecting the emotional state of said user, including a module for generating a stress level value; and

an output circuit for adjusting an output presented to said user as a function of the detected emotional state of said user, wherein said output to said user is a voice prompt; and

where said output circuit for adjusting an output presented to said user includes;

control logic for adjusting the rate of speech included in said voice prompt as a function of a generated stress level value,

control logic for altering prompts presented to said user as a function of said generated stress level value, and

control logic for adjusting prompts as a function of the experience level of the user.