Interface with gaze detection and voice input

• US 9,250,703 B2
• Filed: 05/18/2011
• Issued: 02/02/2016
• Est. Priority Date: 03/06/2006
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Hernandez-Abrego, Gustavo A., Osman, Steven, Mikhailov, Anton, Chen, Ruxin
• Primary Examiner(s): To, Jennifer
• Assistant Examiner(s): Xia, Xuyang
• Application Number:
  US13/110,817
• Publication Number:
  US 20120295708A1
• Time in Patent Office: 1,721 Days
• Field of Search: 715/863
• US Class Current: 1/1
• CPC Class Codes: A63F 13/10 Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/424 involving acoustic input signals, e.g. by using the results of pitch or rhythm extraction or voice recognition
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 2300/1081 Input via voice recognition
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/308 Details of the user interface
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 2360/148 Instrument input by voice
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: H04M 3/567 Multimedia conference systems

Abstract

Methods, computer programs, and systems for interfacing a user with a computer program, utilizing gaze detection and voice recognition, are provided. One method includes an operation for determining if a gaze of a user is directed towards a target associated with the computer program. The computer program is set to operate in a first state when the gaze is determined to be on the target, and set to operate in a second state when the gaze is determined to be away from the target. When operating in the first state, the computer program processes voice commands from the user, and, when operating in the second state, the computer program omits processing of voice commands.

Claims

1. A method for interfacing with a computer program executing on a computing device, the method comprising:

presenting a character on a display;

determining if a point of gaze (POG) of a user is on the display;

setting the character to present a first behavior when the POG is determined to be on the display, wherein the first behavior provides an impression of awareness of the user by the character, wherein a gaze of the character is maintained toward the user and tracked while the POG of the user is on the character, wherein the tracking causes the character to show motion that enables the maintaining of the gaze of the character toward the user;

setting the character to present a second behavior when the POG is determined to be away from the display, wherein the second behavior provides an impression that the character is not interacting with the user, wherein the gaze of the character is not tracked while the POG of the user is away from the character, wherein the computer program processes voice commands from the user when operating in the first behavior, and wherein the computer program omits processing of voice commands when operating in the second behavior; and

setting the character to operate in a third behavior when multiple users are detected, wherein the character, when operating in the third behavior, performs a validation operation after receiving a voice command, the validation operation including asking a question to the user to validate the voice command, wherein the validation operation is not performed when the voice command is received while the character is presenting the first behavior.

2. The method as recited in claim 1, wherein the computer program is a game executing on a game console, and wherein the voice command is an input for the game to perform an operation related to the character.

3. The method as recited in claim 1, wherein determining if the POG of the user is directed towards the display further includes:

performing image analysis for a plurality of images of a video stream; and

detecting facial features of the user based on the image analysis.

4. The method as recited in claim 1, wherein determining if the POG of the user is directed towards the display further includes:

analyzing corneal reflection to determine a direction of the gaze of the user.

5. The method as recited in claim 1, wherein the method operations are performed by the computer program when executed by one or more processors, the computer program being embedded in a non-transitory computer-readable storage medium.

6. The method as recited in claim 1, further including:

identifying a plurality of gaze targets on the display, each gaze target corresponding to a menu selection;

detecting the POG on a first gaze target from the plurality of gaze targets;

highlighting on the display a first menu selection corresponding to the first gaze target to differentiate the first menu selection with respect to other menu selections, wherein the user utilizes gaze to select from the menu selections; and

processing voice commands to initiate an action for the first menu selection.

7. The method as recited in claim 1, wherein operating in the first behavior further includes:

processing controller inputs to perform corresponding actions on the character.

8. The method as recited in claim 1, wherein the first behavior further includes one or more of the character providing eye contact, or showing excitement, or smiling, or jumping, or providing verbal cues to the user.

9. The method as recited in claim 1, wherein the second behavior further includes the character acting in a passive way showing lack of enthusiasm and awareness of the user.

10. The method as recited in claim 1, further including:

tracking a location of a controller using captured images with a camera;

determining which user from a plurality of users in front of the display is holding the controller based on the captured images; and

determining the gaze of the user holding the controller.

11. A method for controlling interfacing reactions with game objects produced by a computer program and rendered on a display, comprising:

detecting presence of a user in a zone defined in front of an image capture device;

presenting a character on a display;

determining if a point of gaze (POG) of the user is on the character on the display;

setting the character to present a first behavior when the POG is on the character on the display, wherein the first behavior provides an impression of awareness of the user by the character, wherein a gaze of the character is maintained toward the user and tracked while the POG of the user is on the character, wherein the tracking causes the character to show motion that enables the maintaining of the gaze of the character toward the user, the first behavior including processing of user voice commands by the computer program to perform actions on the character;

setting the character to present a second behavior when the POG is away from the character, wherein the second behavior provides an impression that the character is not interacting with the user, wherein the gaze of the character is not tracked while the POG of the user is away from the character, the second behavior including an omission of processing of user voice commands by the computer program; and

setting the character to operate in a third behavior when multiple users are detected, wherein the character, when operating in the third behavior, performs a validation operation after receiving a voice command, the validation operation including asking a question to the user to validate the voice command, wherein the validation operation is not performed when the voice command is received while the character is presenting the first behavior.

12. The method as recited in claim 11, wherein determining if the POG is on the character further includes:

analyzing reflection from pupils of the user from an infrared light.

13. The method as recited in claim 11, wherein the first behavior further includes taking turns speaking to the user.

14. The method as recited in claim 11, further including:

tracking a location of a controller using captured images with a camera;

determining which user from a plurality of users in front of the display is holding the controller based on the captured images; and

determining the gaze of the user holding the controller.

15. The method as recited in claim 11, further including:

identifying a plurality of gaze targets on the display, each gaze target corresponding to a menu selection;

detecting the POG on a first gaze target from the plurality of gaze targets;

highlighting on the display a first menu selection corresponding to the first gaze target, wherein the user utilizes gaze to select from the menu selections; and

processing voice commands to initiate an action for the first menu selection.

16. An apparatus with a visual interface for capturing user input to a computer program executing on a computing device, the apparatus comprising:

a camera that captures images of a user situated in front of a display, the computer program presenting a character on the display;

a gaze detection module that determines if a point of gaze (POG) of a user is directed towards the character on the display based on the captured images, wherein the computer program sets the character to present a first behavior when the POG is on the character on the display, and wherein the computer program sets the character to present a second behavior when the gaze is determined to be away from the character on the display, wherein the first behavior provides an impression of awareness of the user by the character, wherein a gaze of the character is maintained toward the user and tracked while the POG of the user is on the character on the display, wherein the tracking causes the character to show motion that enables the maintaining of the gaze of the character toward the user, wherein the gaze of the character is not tracked while the POG of the user is away from the character on the display, wherein the second behavior provides an impression that the character is not interacting with the user, and wherein the computer program sets the character to operate in a third behavior when multiple users are detected; and

a voice command processing module that receives voice commands from the user, wherein the computer program processes the voice commands from the user when operating in the first behavior to perform actions on the character, wherein the computer program omits processing of the voice commands when operating in the second behavior, and wherein the computer program performs a validation operation after receiving a voice command from the user when the character is operating in the third behavior, the validation operation including asking a question to the user to validate the voice command, wherein the validation operation is not performed when the voice command is received while the character is presenting the first behavior.

17. The apparatus as recited in claim 16, wherein the computer program tracks a location of a controller using the captured images, wherein the computer program determines which user from a plurality of users in front of the display is holding the controller based on the captured images, and wherein the gaze detection module determines the gaze of the user holding the controller.

18. The apparatus as recited in claim 16, wherein the voice command processing module is coupled to a microphone array and determines a direction of the voice commands, and wherein the gaze detection module focuses on the gaze of the user based on the direction of the voice commands.

19. The apparatus as recited in claim 16, wherein the gaze detection module performs a calibration operation to determine a location of the display.