INTERACTION OF MULTIPLE PERCEPTUAL SENSING INPUTS

• US 20140258942A1
• Filed: 03/05/2013
• Published: 09/11/2014
• Est. Priority Date: 03/05/2013
• Status: Abandoned Application

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation, Omek Interactive Ltd. (Intel Corporation)
• Inventors: Kutliroff, Gershom, Yanai, Yaron
• Application Number:
  US13/785,669
• Publication Number:
  US 20140258942A1
• Time in Patent Office: Days
• US Class Current: 715/863
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures

Abstract

A system and method for using multiple perceptual sensing technologies to capture information about a user'"'"'s actions and for synergistically processing the information is described. Non-limiting examples of perceptual sensing technologies include gesture recognition using depth sensors, two-dimensional cameras, gaze detection, and/or speech recognition. The information captured about a user'"'"'s gestures using one type of sensing technology is often not able to be captured with another type of technology. Thus, using multiple perceptual sensing technologies allows more information to be captured about the user'"'"'s gestures. Further, by synergistically leveraging the information acquired using multiple perceptual sensing technologies, a more natural user interface can be created for a user to interact with an electronic device.

Claims

1. A method comprising:

acquiring data about a user'"'"'s actions using a plurality of perceptual sensing technologies;

analyzing the acquired data to identify a gesture from the user'"'"'s actions, wherein the gesture is defined based upon information able to be detected by the plurality of perceptual sensing technologies.

2. The method of claim 1, wherein the gesture is performed by the user to interact with a user interface to control an electronic device.

3. The method of claim 2, wherein the plurality of perceptual sensing technologies includes a gaze detection system and a depth camera, wherein the user interface includes a cursor, and further wherein the gesture comprises gazing at the cursor on a screen and moving the user'"'"'s gaze from the cursor to a virtual object on the screen to map the cursor to the virtual object, and performing a hand gesture to select the virtual object on the screen.

4. The method of claim 3, wherein the hand gesture is a pinch of two fingers.

5. The method of claim 3, wherein the hand gesture is a grabbing motion of a hand.

6. The method of claim 2, wherein the plurality of perceptual sensing technologies includes a depth camera and a microphone array, and wherein the user interface includes a cursor, and further wherein the gesture comprises a hand movement for controlling the cursor and spoken words for selecting or manipulating the cursor.

7. The method of claim 2, wherein the plurality of perceptual sensing technologies includes a gaze detection system and a microphone array, and wherein the user interface includes a cursor, and further wherein the gesture comprises gazing at the cursor and moving the user'"'"'s gaze to control the cursor and spoken words for selecting or manipulating the cursor.

8. The method of claim 1, wherein the plurality of perceptual sensing technologies includes a depth camera and a gaze detection system, wherein the data acquired from the depth camera is a select gesture made by the user'"'"'s hand for selecting a virtual object on a screen, and wherein the data acquired from the gaze detection system is a gaze at the selected virtual object, wherein the gaze detection reduces a false positive in identifying the virtual object selected by the user.

9. The method of claim 1, wherein the plurality of perceptual sensing technologies includes a touch screen and a depth camera.

10. The method of claim 9, wherein the data acquired from the touch screen is a location of a touch on the touch screen, and further wherein the data acquired from the depth camera identifies which one of a user'"'"'s fingers touched the touch screen.

11. The method of claim 9, wherein the data acquired from the touch screen is multiple locations of multiple touches on the touch screen, and further wherein the data acquired from the depth camera identifies whether the multiple touches are from only the user or from the user and one or more other users.

12. The method of claim 9, wherein the data acquired from the touch screen is a location of a touch on the touch screen, and further wherein the data acquired from the depth camera is an angle with which the user'"'"'s finger touched the touch screen.

13. The method of claim 9, wherein the data acquired from the touch screen is a location of a touch on the touch screen, and further wherein the data acquired from the depth camera identifies which one of the user'"'"'s hands touched the touch screen.

14. The method of claim 1, wherein the plurality of perceptual sensing technologies includes a touch screen and a depth camera, and further wherein the gesture comprises a touch on the touch screen and subsequent movement away from the touch screen.

15. The method of claim 1, wherein the plurality of perceptual sensing technologies includes a depth camera and a touch screen, and further wherein the gesture comprises a hand and finger movement a distance from the touch screen and a subsequent touch on the touch screen.

16. A system comprising:

a plurality of perceptual sensors configured to acquire data about a user'"'"'s actions;

a processing module configured to analyze the acquired data to identify a gesture from the user'"'"'s actions,

wherein the gesture is defined based upon data able to be detected by the plurality of perceptual sensors.

17. The system of claim 16, further comprising a user interface application module configured to allow the user to control an electronic device based on the identified gesture.

18. The system of claim 16, wherein the plurality of perceptual sensors includes a touch screen and a depth camera, and further wherein the data acquired by the depth camera augments the data acquired by the touch screen.

19. The system of claim 16, wherein the plurality of perceptual sensing technologies includes a gaze detection system and a depth camera, wherein the user interface includes a cursor, and further wherein the gesture comprises gazing at the cursor on a screen and moving the user'"'"'s gaze from the cursor to a virtual object on the screen to map the cursor to the virtual object, and performing a hand gesture to select the virtual object on the screen.

20. The system of claim 16, wherein the plurality of perceptual sensing technologies includes a depth camera and a gaze detection system, wherein the data acquired from the depth camera is a select gesture made by the user'"'"'s hand for selecting a virtual object on a screen, and wherein the data acquired from the gaze detection system is a gaze at the selected virtual object, wherein the gaze detection reduces a false positive in identifying the virtual object selected by the user.

21. A system comprising:

a first means for acquiring data about a user'"'"'s action;

a second means for acquiring data about the user'"'"'s action;

one or more processing modules configured to analyze the acquired data to identify a gesture from the user'"'"'s actions,

wherein the gesture is defined based upon data able to be detected by the first means for acquiring data and the second means for acquiring data.

22. The system of claim 21, further comprising a user interface application module configured to allow the user to control an electronic device based on the identified gesture.