Gesture Detection Based on Information from Multiple Types of Sensors

• US 20130229508A1
• Filed: 02/14/2013
• Published: 09/05/2013
• Est. Priority Date: 03/01/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Li, Ren, Gupta, Samir K., Clarkson, Ian, Krulce, Darrell L.
• Granted Patent: US 9,389,690 B2
• Time in Patent Office: Days
• US Class Current: 348/77
• CPC Class Codes: G06F 1/3287 by switching off individual functional units in the computer system
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management
• CPC Class Codes: Y02D 30/50 in wire-line communication networks, e.g. low power modes or reduced link rate

Abstract

A method includes receiving a first output from a first sensor of an electronic device and receiving a second output from a second sensor of the electronic device. The first sensor has a first sensor type and the second sensor has a second sensor type that is different from the first sensor type. The method also includes detecting a gesture based on the first output and the second output according to a complementary voting scheme that is at least partially based on gesture complexity.

Claims

1. An apparatus comprising:

a first sensor configured to generate a first output;

a camera configured to generate a second output;

a processor; and

a gesture detection module executable by the processor to detect a gesture based on at least one of the first output and the second output,

wherein at least a portion of the camera is selectively deactivated based on a lighting level and a gesture range.

2. The apparatus of claim 1, wherein the first sensor comprises an ultrasound sensor.

3. The apparatus of claim 2, further comprising:

an ultrasound processing path configured to process the first output; and

an image processing path configured to process the second output,

wherein the ultrasound processing path is configured to send gesture range information to the image processing path.

4. The apparatus of claim 1, further comprising a light detector internal to the camera and configured to determine the lighting level.

5. The apparatus of claim 1, further comprising a light detector external to the camera and configured to determine the lighting level.

6. The apparatus of claim 1, wherein detecting the gesture comprises:

when the lighting level is less than a lighting threshold, deactivating the camera and performing gesture recognition based on the first output from the first sensor; and

when the lighting level is greater than or equal to the lighting threshold;

when the gesture range is less than a nearness threshold, performing gesture detection based on the first output from the first sensor;

when the gesture range is greater than a farness threshold, activating the camera and performing gesture recognition based on the second output from the camera; and

when the gesture range is greater than the nearness threshold and less than the farness threshold, activating the camera and performing gesture recognition by applying a complementary voting scheme to the first output and to the second output.

7. The apparatus of claim 6, wherein the complementary voting scheme is at least partially based on gesture complexity.

8. A method comprising:

receiving a first output from a first sensor of an electronic device, wherein the first sensor has a first sensor type;

receiving a second output from a second sensor of the electronic device, wherein the second sensor has a second sensor type that is different from the first sensor type; and

detecting a gesture based on the first output and the second output according to a complementary voting scheme that is at least partially based on gesture complexity.

9. The method of claim 8, wherein the first sensor type and the second sensor type each comprise at least one of a camera sensor type, an ultrasound sensor type, an infrared sensor type, and a magnetic sensor type.

10. The method of claim 8, wherein the first output identifies a first gesture with a first confidence score and wherein the second output identifies a second gesture with a second confidence score.

11. The method of claim 10, wherein applying the complementary voting scheme comprises:

when the first sensor and the second sensor identify the same gesture, selecting the same gesture as an output gesture; and

when the first sensor and the second sensor identify different gestures, selecting a gesture having a higher confidence score as the output gesture.

12. The method of claim 11, wherein applying the complementary voting scheme further comprises, when the first sensor and the second sensor identify gestures of varying complexity, selecting a more complex gesture as the output gesture.

13. The method of claim 11, wherein the first sensor and the second sensor are configured to output data in accordance with a common data model.

14. The method of claim 13, wherein a first processing path associated with the first sensor receives the data output by the second sensor, and wherein a second processing path associated with the second sensor receives the data output by the first sensor.

15. The method of claim 8, further comprising receiving third output from a third sensor, wherein the gesture is detected further based on the third output from the third sensor.

16. An apparatus, comprising:

first sensor means for generating a first output and having a first sensor type;

second sensor means for generating a second output and having a second sensor type that is different from the first sensor type; and

means for detecting a gesture based on the first output and the second output according to a complementary voting scheme that is at least partially based on gesture complexity.

17. The apparatus of claim 16, further comprising:

means for determining a lighting level; and

means for determining a gesture range.

18. The apparatus of claim 17, further comprising means for selectively deactivating the second sensor means based on the lighting level and the gesture range.

19. An apparatus, comprising:

an ultrasound sensor configured to generate a first output in accordance with a common data model and to provide the first output to an ultrasound processing path;

a camera configured to generate a second output in accordance with the common data model and to provide the second output to an image processing path;

a processor; and

a gesture detection module executable by the processor to detect a gesture based on at least one of the first output and the second output,

wherein the ultrasound processing path and the image processing path are configured to exchange data in accordance with the common data model.

20. The apparatus of claim 19, wherein the data exchanged in accordance with the common data model includes the first output from the ultrasound sensor, the second output from the camera, data related to a range of an object to the ultrasound sensor or the camera, data related to a position of the object relative to the ultrasound sensor or the camera, data associated with an active configuration of the ultrasound sensor or the camera, or any combination thereof.

21. The apparatus of claim 19, wherein the camera is configured to identify an area of interest based on the data received from the ultrasound processing path.

22. The apparatus of claim 19, wherein the ultrasound processing path is configured to determine whether a particular detected gesture is a false positive based on the data received from the image processing path.

23. The apparatus of claim 19, further comprising a light detector configured to detect a lighting level, wherein at least a portion of the camera is selectively deactivated based on the lighting level.

24. An apparatus comprising:

an ultrasound sensor configured to provide a first output to an ultrasound processing path;

a camera configured to provide a second output to an image processing path;

a processor; and

a gesture detection module executable by the processor to detect a gesture based on at least one of the first output and the second output,

wherein the ultrasound sensor and the camera are each configured to self-adjust, independent of the processor, based on data exchanged between the ultrasound processing path and the image processing path.

25. The apparatus of claim 24, wherein the ultrasound sensor self-adjusts an active configuration, a signal emission frequency, a signal emission direction, or any combination thereof based on the second output from the camera.

26. The apparatus of claim 24, wherein the camera self-adjusts an active configuration, a focal length, an area of focus, or any combination thereof based on the first output from the ultrasound sensor.

27. The apparatus of claim 24, wherein the gesture detection module detects the gesture by applying a complementary voting scheme that is at least partially based on gesture complexity.

28. A non-transitory processor-readable medium comprising instructions that, when executed by a processor, cause the processor to:

receive a first output from a first sensor of an electronic device, wherein the first sensor has a first sensor type;

receive a second output from a second sensor of the electronic device, wherein the second sensor has a second sensor type that is different from the first sensor type; and

detect a gesture based on the first output and the second output according to a complementary voting scheme that is at least partially based on gesture complexity.

29. The non-transitory processor-readable medium of claim 28, wherein the first output identifies a first gesture with a first confidence score and wherein the second output identifies a second gesture with a second confidence score.

30. The non-transitory processor-readable medium of claim 28, further comprising instructions that, when executed by the processor, cause the processor to receive third output from a third sensor, wherein the gesture is detected further based on the third output from the third sensor.