Interaction using touch and non-touch gestures

• US 8,433,138 B2
• Filed: 10/29/2008
• Issued: 04/30/2013
• Est. Priority Date: 10/29/2008
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Corporation
• Inventors: Wang, Hao, Chai, Xiujuan, Yu, Kun
• Primary Examiner(s): Desire, Gregory M
• Application Number:
  US12/260,295
• Publication Number:
  US 20100104134A1
• Time in Patent Office: 1,644 Days
• Field of Search: 345/156, 345/173, 382/103, 382/181
• US Class Current: 382/181
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A computer interface may use touch- and non-touch-based gesture detection systems to detect touch and non-touch gestures on a computing device. The systems may each capture an image, and interpret the image as corresponding to a predetermined gesture. The systems may also generate similarity values to indicate the strength of a match between a captured image and corresponding gesture, and the system may combine gesture identifications from both touch- and non-touch-based gesture identification systems to ultimately determine the gesture. A threshold comparison algorithm may be used to apply different thresholds for different gesture detection systems and gesture types.

Claims

1. A method comprising:

receiving image data corresponding to a user'"'"'s gesture;

processing, by a computing device, the image data to generate a touch gesture similarity value that defines similarity of the user'"'"'s gesture to one or more touch gestures and a non-touch gesture similarity value that defines similarity of the user'"'"'s gesture to one or more non-touch gestures;

processing the touch gesture similarity value and the non-touch gesture similarity value to identify the user'"'"'s gesture, resulting in an identified gesture; and

generating an output in response to the identified gesture.

2. The method of claim 1, wherein processing the touch gesture similarity value and the non-touch gesture similarity value includes:

comparing the touch gesture similarity value with a first threshold value; and

comparing the non-touch similarity value with a second threshold value.

3. The method of claim 2, further comprising setting the first and second threshold values to different values based on gesture type.

4. The method of claim 1, wherein a first portion of the image data is received from a touch-based gesture detection system, and a second portion of the image data is received from a non-touch based gesture detection system.

5. The method of claim 4, wherein processing the image data includes using the first portion of the image data from the touch-based gesture detection system to detect a non-touch gesture.

6. The method of claim 1, further comprising determining a differential similarity value, and comparing the differential similarity value with a difference between the touch gesture similarity value and the non-touch gesture similarity value.

7. The method of claim 1, further comprising:

using contextual information to select between a touch and non-touch gesture when processing the touch gesture similarity value and the non-touch gesture similarity value.

8. The method of claim 7, wherein the contextual information includes information identifying a previously-identified gesture.

9. The method of claim 7, wherein the contextual information includes information identifying a command set of an application.

10. One or more computer-readable media, storing computer-executable instructions that, when executed, cause a computer to at least:

receive image data corresponding to a user'"'"'s gesture;

process the image data to generate a touch gesture similarity value that defines similarity of the user'"'"'s gesture to one or more touch gestures and a non-touch gesture similarity value that defines similarity of the user'"'"'s gesture to one or more non-touch gestures;

process the touch gesture similarity value and the non-touch gesture similarity value to identify the user'"'"'s gesture, resulting in an identified gesture; and

generate an output in response of the identified gesture.

11. The one or more computer-readable media of claim 10, wherein processing the touch gesture similarity value and the non-touch gesture similarity value includes:

comparing the touch gesture similarity value with a first threshold value; and

comparing the non-touch gesture similarity value with a second threshold value.

12. The one or more computer-readable media of claim 11, wherein the first and second threshold values differ based on different gesture types.

13. The one or more computer-readable media of claim 10, wherein a first portion of the image data is received from a touch-based gesture detection system, and a second portion of the image data is received from a non-touch based gesture detection system.

14. The one or more computer-readable media of claim 13, further storing computer-executable instructions that, when executed, cause the computer to:

use the first portion of the image data from the touch-based gesture detection system to detect a non-touch gesture.

15. The one or more computer-readable media of claim 10, further storing computer-executable instructions that, when executed, cause the computer to compare a differential similarity value with a difference between the touch gesture similarity value and the non-touch gesture similarity value.

16. The one or more computer-readable media of claim 10, further storing computer-executable instructions that, when executed, cause the computer to:

use contextual information to select between a touch gesture and a non-touch gesture.

17. The one or more computer-readable media of claim 16, wherein the contextual information includes information identifying a previously-identified gesture.

18. The one or more computer-readable media of claim 16, wherein the contextual information includes information identifying a command set of an application.

19. An apparatus, comprising:

one or more processors; and

memory storing computer-executable instructions that, with the processor, cause the apparatus to at least;

receive image data corresponding to a user'"'"'s gesture;

process the image data to generate a touch gesture similarity value that defines similarity of the user'"'"'s gesture to one or more touch gestures and a non-touch gesture similarity value that defines similarity of the user'"'"'s gesture to one or more non-touch gestures;

process the touch gesture similarity value and the non-touch gesture similarity value to identify the user'"'"'s gesture, resulting in an identified gesture; and

generate an output in response to the identified gesture.

20. The apparatus of claim 19, wherein processing the touch gesture similarity value and the non-touch gesture similarity value includes:

comparing the touch gesture similarity value with a first threshold value; and

comparing the non-touch similarity value with a second threshold value.

21. The apparatus of claim 20, wherein the first and second threshold values differ based on different gesture types.

22. The apparatus of claim 19, wherein a first portion of the image data is received from a touch-based gesture detection system, and a second portion of the image data is received from a non-touch based gesture detection system.

23. The apparatus of claim 19, wherein the memory further stores computer-executable instructions that, with the processor, cause the apparatus to use the first portion of the image data from a touch-based gesture detection system to detect a non-touch gesture.

24. The apparatus of claim 19, wherein the memory further stores computer-executable instructions that, with the processor, cause the apparatus to compare a differential similarity value with a difference between the touch gesture and non-touch gesture similarity values.

25. The apparatus of claim 19, wherein the memory further stores computer-executable instructions that, with the processor, cause the apparatus to use contextual information to select between a touch gesture and a non-touch gesture.

26. The apparatus of claim 25, wherein the contextual information includes information identifying a previously-identified gesture.

27. The apparatus of claim 25, wherein the contextual information includes information identifying a command set of an application.

28. The apparatus of claim 19, further comprising an infrared camera and a plurality of infrared light sources positioned below a screen, wherein the infrared camera is configured to supply at least a portion of the image data.

29. The apparatus of claim 19, further comprising a visible light camera positioned above a screen, wherein the visible light camera is configured to supply at least a portion of the image data.