Gesture detection based on information from multiple types of sensors

• US 9,389,690 B2
• Filed: 02/14/2013
• Issued: 07/12/2016
• Est. Priority Date: 03/01/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Li, Ren, Clarkson, Ian, Gupta, Samir K., Krulce, Darrell L.
• Primary Examiner(s): Eisen, Alexander
• Assistant Examiner(s): Sarma, Abhishek
• Application Number:
  US13/767,698
• Publication Number:
  US 20130229508A1
• Time in Patent Office: 1,244 Days
• Field of Search: 345/156, 348/77, 348/164, 348/207.1, 348/E5.024, 348/E5.09
• US Class Current: 1/1
• CPC Class Codes: G06F 1/3287 by switching off individual functional units in the computer system
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management
• CPC Class Codes: Y02D 30/50 in wire-line communication networks, e.g. low power modes or reduced link rate

Abstract

A method includes receiving a first output from a first sensor of an electronic device and receiving a second output from a second sensor of the electronic device. The first sensor has a first sensor type and the second sensor has a second sensor type that is different from the first sensor type. The method also includes detecting a gesture based on the first output and the second output according to a complementary voting scheme that is at least partially based on gesture complexity.

Claims

1. An apparatus comprising:

a first sensor configured to generate a first output;

a camera configured to generate a second output; and

a processor configured to;

detect a gesture based on at least one of the first output and the second output when a lighting level is greater than or equal to a threshold; and

deactivate at least a portion of the camera and perform gesture recognition based on the first output from the first sensor when the lighting level is less than the threshold.

2. The apparatus of claim 1, wherein the first sensor comprises an ultrasound sensor.

3. The apparatus of claim 2, further comprising:

an ultrasound processing path configured to process the first output; and

an image processing path configured to process the second output,

wherein the ultrasound processing path is configured to send gesture range information to the image processing path,

wherein the gesture range information includes data related to a position of an object relative to the ultrasound sensor,

wherein the camera is configured to identify an area of interest based on the position of the object, and

wherein the camera is configured to activate a portion of a sensor array of the camera that corresponds to the area of interest and to deactivate other portions of the sensor array of the camera.

4. The apparatus of claim 1, further comprising a light detector internal to the camera and configured to determine the lighting level.

5. The apparatus of claim 1, further comprising a light detector external to the camera and configured to determine the lighting level.

6. The apparatus of claim 1, wherein

the processor is configured to perform gesture recognition by using a voting scheme to select between the first output and the second output conditioned on a gesture range exceeding a nearness threshold and being less than a farness threshold.

7. The apparatus of claim 6, wherein the voting scheme is at least partially based on gesture complexity.

8. A method comprising:

receiving a first output from a first sensor of an electronic device, wherein the first sensor has a first sensor type;

receiving a second output from a second sensor of the electronic device, wherein the second sensor has a second sensor type that is different from the first sensor type, and wherein the second sensor type comprises a camera sensor type;

when a lighting level is greater than or equal to a threshold, detecting a gesture based on the first output and the second output according to a voting scheme that is at least partially based on gesture complexity; and

when the lighting level is less than the threshold, deactivating at least a portion of the second sensor and performing gesture recognition based on the first output from the first sensor.

9. The method of claim 8, wherein the first sensor type comprises at least one of an ultrasound sensor type, an infrared sensor type, and a magnetic sensor type.

10. The method of claim 8, wherein the first output identifies a first gesture with a first confidence score and wherein the second output identifies a second gesture with a second confidence score.

11. The method of claim 10, wherein detecting the gesture according to the voting scheme comprises:

when the first sensor and the second sensor identify the same gesture, selecting the same gesture as an output gesture; and

when the first sensor and the second sensor identify different gestures, selecting a gesture having a higher confidence score as the output gesture.

12. The method of claim 11, wherein detecting the gesture according to the voting scheme further comprises, when the first sensor and the second sensor identify gestures of varying complexity, selecting a more complex gesture as the output gesture.

13. The method of claim 11, wherein the first sensor and the second sensor output data in accordance with a common data model.

14. The method of claim 13, wherein a first processing path associated with the first sensor receives the data output by the second sensor, and wherein a second processing path associated with the second sensor receives the data output by the first sensor.

15. The method of claim 8, further comprising receiving third output from a third sensor, wherein the gesture is detected further based on the third output from the third sensor.

16. An apparatus, comprising:

first sensor means for generating a first output and having a first sensor type;

second sensor means for generating a second output and having a second sensor type that is different from the first sensor type;

means for detecting a lighting level;

means for selectively deactivating the second sensor means in response to determining that the lighting level is less than a threshold; and

means for detecting a gesture, the means for detecting the gesture configured to, when the lighting level is greater than or equal to the threshold, perform gesture recognition based on the first output and the second output according to a voting scheme that is at least partially based on gesture complexity and configured to perform gesture recognition based on the first output from the first sensor means when the lighting level is less than the threshold.

17. The apparatus of claim 16, further comprising

means for determining a lighting level;

and means for determining a gesture range.

18. The apparatus of claim 17, wherein the means for selectively deactivating is configured to selectively deactivate the at least a portion of the second sensor means further based on and the gesture range.

19. An apparatus, comprising:

an ultrasound sensor configured to generate a first output in accordance with a common data model and to provide the first output to an ultrasound processing path;

a camera configured to generate a second output in accordance with the common data model and to provide the second output to an image processing path;

a processor; and

a gesture detection module executable by the processor to detect a gesture based on at least one of the first output and the second output,

wherein the ultrasound processing path and the image processing path are configured to exchange data in accordance with the common data model, wherein the data includes a position of an object relative to the ultrasound sensor or the camera, and wherein the camera is configured to identify an area of interest based on the position of the object.

20. The apparatus of claim 19, wherein the data exchanged in accordance with the common data model further includes the first output from the ultrasound sensor, the second output from the camera, data related to a range of the object to the ultrasound sensor or the camera, data associated with an active configuration of the ultrasound sensor or the camera, or any combination thereof.

21. The apparatus of claim 19, wherein the camera is configured to activate a portion of a sensor array in the camera that corresponds to the area of interest and to deactivate other portions of the sensor array in the camera.

22. The apparatus of claim 19, wherein the ultrasound processing path is configured to determine whether a particular detected gesture is a false positive based on the data received from the image processing path.

23. The apparatus of claim 19, further comprising a light detector configured to detect a lighting level.

24. An apparatus comprising:

an ultrasound sensor configured to provide a first output to an ultrasound processing path;

a camera configured to provide a second output to an image processing path;

a processor; and

a gesture detection module executable by the processor to detect a gesture based on at least one of the first output and the second output,

wherein the ultrasound sensor and the camera are each configured to self-adjust, independent of the processor, based on data exchanged between the ultrasound processing path and the image processing path.

25. The apparatus of claim 24, wherein the ultrasound sensor is configured to self-adjust an active configuration, a signal emission frequency, a signal emission direction, or any combination thereof, based on the second output from the camera.

26. The apparatus of claim 24, wherein the camera is configured to self-adjust an active configuration, a focal length, an area of focus, or any combination thereof, based on the first output from the ultrasound sensor.

27. The apparatus of claim 24, wherein the gesture detection module is configured to detect the gesture by using a voting scheme that is at least partially based on gesture complexity.

28. A non-transitory processor-readable medium comprising instructions that, when executed by a processor, cause the processor to:

receive a first output from a first sensor of an electronic device, wherein the first sensor has a first sensor type;

receive a second output from a second sensor of the electronic device, wherein the second sensor has a second sensor type that is different from the first sensor type, and wherein the second sensor type comprises a camera sensor type;

when a lighting level is greater than or equal to a threshold, detect a gesture based on the first output and the second output according to a voting scheme that is at least partially based on gesture complexity; and

when the lighting level is less than the threshold, deactivate at least a portion of the second sensor and perform gesture recognition based on the first output from the first sensor.

29. The non-transitory processor-readable medium of claim 28, wherein the first output identifies a first gesture with a first confidence score and wherein the second output identifies a second gesture with a second confidence score.

30. The non-transitory processor-readable medium of claim 28, further comprising instructions that, when executed by the processor, cause the processor to receive a third output from a third sensor, wherein the gesture is detected further based on the third output from the third sensor.