Camera based sensor for motion detection

• US 9,176,608 B1
• Filed: 06/27/2011
• Issued: 11/03/2015
• Est. Priority Date: 06/27/2011
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Baldwin, Leo B., Noble, Isaac S.
• Primary Examiner(s): Abdulselam, Abbas
• Assistant Examiner(s): OLIVER, GERALD L
• Application Number:
  US13/170,164
• Time in Patent Office: 1,590 Days
• Field of Search: 345156-184, 178 1801- 2004
• US Class Current: 1/1
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

The amount of power and processing needed to process gesture input for a computing device can be reduced by utilizing a separate gesture sensor. The gesture sensor can have a form factor similar to that of conventional camera elements, in order to reduce costs by being able to utilize readily available low cost parts, but can have a lower resolution and adjustable virtual shutter such that fast motions can be captured and/or recognized by the device. In some devices, a subset of the pixels of the gesture sensor can be used as a motion detector, enabling the gesture sensor to run in a low power state unless there is likely gesture input to process. Further, at least some of the processing and circuitry can be included with the gesture sensor such that various functionality can be performed without accessing a central processor or system bus, thus further reducing power consumption.

Claims

1. A method of enabling gesture input for a computing device, comprising:

capturing first image information at a first frame rate using a subset of a set of pixels of a gesture sensor, wherein the set of pixels includes a plurality of pairs of pixels, each pair of pixels including a first pixel to be exposed to incident light and a second pixel to be shielded from the incident light and for storing charge received from the first pixel exposed to the incident light;

analyzing, using a processor of the gesture sensor, the first image information to detect motion near the computing device by determining that a difference between (a) a first ratio of (i) a first group of pixels, of a first image of the first image information, located substantially in a first region of the gesture sensor, and (ii) a second group of pixels, of the first image, located substantially in a second region of the gesture sensor, and (b) a second ratio of (i) a third group of pixels, of a second image of the first image information, located substantially in the first region, and (ii) a fourth group of pixels, of the second image, located substantially in the second region, exceeds a predetermined threshold, wherein the first region and the second region are opposing edges of the gesture sensor, the processor of the gesture sensor being separate from a processor of the computing device;

activating a gesture input mode to capture second image information at a full resolution of the gesture sensor at a second frame rate, wherein the second frame rate is greater than the first frame rate;

detecting one or more gestures in the second image information;

causing a signal to be sent to an image capture element for capturing third image information of the one or more gestures, the image capture element having a similar form factor but higher resolution than the gesture sensor; and

providing information corresponding to the one or more gestures as input to the processor of the computing device.

2. The method of claim 1, wherein the second image information is captured at least in part by timing at least one of a capture time of each first pixel and a transfer time to each second pixel in a pair of pixels.

3. The method of claim 2, further comprising:

determining an amount of ambient light; and

adjusting at least one of the capture time and the transfer time based at least in part upon the amount of ambient light.

4. The method of claim 2, further comprising:

determining whether an illumination element will be active during a time of image capture; and

adjusting at least one of the capture time and the transfer time based at least in part upon whether the illumination element will be active.

5. The method of claim 1, further comprising:

deactivating the gesture input mode if no gesture is detected within a specified period of inactivity.

6. The method of claim 1, wherein the third image information is to be analyzed using at least one facial recognition algorithm, and wherein providing information corresponding to the one or more gestures as input is performed in response to the facial recognition algorithm identifying a user performing the one or more gestures.

7. The method of claim 6, wherein the gesture sensor has a wider field of view than the image capture element, and wherein the gesture sensor is able to track a user outside a field of view of the image capture element but within a field of view of the gesture sensor without requiring a subsequent image recognition process when the user returns to the field of view of the image capture element.

8. A method of enabling gesture input for a computing device, comprising:

capturing image information at a first frame rate using a gesture sensor of the computing device, the gesture sensor having a first form factor at most similar to a second form factor of an image capture element of the computing device but having a lower resolution, wherein the gesture sensor includes a plurality of pairs of pixels, each pair of pixels including a first pixel to be exposed to incident light and a second pixel to be shielded from the incident light and for storing charge received from the first pixel exposed to the incident light, wherein the gesture sensor is capable of capturing image information at a second frame rate, the second frame rate being greater than the first frame rate;

analyzing, using a processor of the gesture sensor, the image information to detect a gesture performed within a capture range of the gesture sensor by determining that a difference between (a) a first ratio of (i) a first group of pixels, of a first image of the image information, located substantially in a first region of the gesture sensor, and (ii) a second group of pixels, of the first image, located substantially in a second region of the gesture sensor, and (b) a second ratio of (i) a third group of pixels, of a second image of the image information, located substantially in the first region, and (ii) a fourth group of pixels, of the second image, located substantially in the second region, exceeds a predetermined threshold, wherein the first region and the second region are opposing edges of the gesture sensor; and

causing a signal to be sent to a processor of the computing device, separate from the gesture sensor, as input to the computing device.

9. The method of claim 8, further comprising:

selectively activating at least one illumination element for capturing the image information based at least in part upon a detected intensity of ambient light.

10. The method of claim 9, wherein information is passed to the processor of the computing device only after detection by the gesture sensor of a predetermined type of motion.

11. A method of enabling gesture input for a computing device, comprising:

capturing first image information at a first resolution and a first frame rate using a subset of a set of pixels of a gesture sensor, wherein the set of pixels includes a plurality of pairs of pixels, each pair of pixels including a first pixel to be exposed to incident light and a second pixel to be shielded from the incident light and for storing charge received from the first pixel in response to the incident light;

analyzing, using a processor of the gesture sensor, the first image information to detect motion near the computing device by determining that a difference between (a) a first ratio of (i) a first group of pixels, of a first image of the first image information, located substantially in a first region of the gesture sensor, and (ii) a second group of pixels, of the first image, located substantially in a second region of the gesture sensor, and (b) a second ratio of (i) a third group of pixels, of a second image of the first image information, located substantially in the first region, and (ii) a fourth group of pixels, of the second image, located substantially in the second region, exceeds a predetermined threshold, wherein the first region and the second region are proximate opposing edges of the gesture sensor;

activating a gesture input mode to capture second image information at a second resolution and a second frame rate using the gesture sensor, wherein the second resolution is greater than the first resolution, and the second frame rate is greater than the first frame rate;

detecting one or more gestures in the second image information; and

sending a signal from the processor of the gesture sensor to a separate processor of the computing device,

wherein information for a detected gesture can be compared against a set of known gestures to provide input to the computing device.

12. The method of claim 11, further comprising:

deactivating the gesture input mode, or placing the gesture sensor in a low power mode, if no gesture is detected within a specified period of inactivity.

13. The method of claim 11, further comprising:

periodically flashing at least one illumination element during capture of the first image information.

14. The method of claim 11, wherein the second image information is captured at least in part by timing at least one of a capture time of each first pixel and a transfer time to each second pixel in a pair of pixels.

15. A computing device, comprising:

at least one device processor;

at least one image capture element;

at least one gesture sensor, the gesture sensor having a lower resolution than the at least one image capture element; and

a memory device including instructions operable to be executed by the processor to perform a set of actions, enabling the computing device to;

capture first image information at a first resolution and a first frame rate using a subset of a set of pixels of the gesture sensor, wherein the set of pixels includes a plurality of pairs of pixels, each pair of pixels including a first pixel to be exposed to incident light and a second pixel to be shielded from the incident light and for storing charge received from the first pixel exposed to the incident light;

analyze, using a processor of the gesture sensor, the first image information to detect motion near the computing device by determining that a difference between (a) a first ratio of (i) a first group of pixels, of a first image of the first image information, located substantially in a first region of the gesture sensor, and (ii) a second group of pixels, of the first image, located substantially in a second region of the gesture sensor, and (b) a second ratio of (i) a third group of pixels, of a second image of the first image information, located substantially in the first region, and (ii) a fourth group of pixels, of the second image, located substantially in the second region, exceeds a predetermined threshold, wherein the first region and the second region are opposing edges of the gesture sensor;

capture second image information at a second resolution and a second frame rate using the gesture sensor, wherein the second resolution is greater than the first resolution, and the second frame rate is greater than the first frame rate;

detect one or more gestures in the second image information; and

send a signal from the processor of the gesture sensor to the at least one device processor, the processor of the gesture sensor being separate from the at least one device processor,

wherein information for a detected gesture can be compared against a set of known gestures to provide input to the computing device.

16. The computing device of claim 15, further comprising:

at least one illumination element operable to selectively produce illumination during a time of image capture of the at least one gesture sensor.

17. The computing device of claim 15, further comprising:

at least one light sensor for determining an amount of ambient light, the amount of ambient light being used to determine, at least in part, when to produce illumination using the at least one illumination element.

18. The computing device of claim 15, wherein the at least one illumination element includes at least one of a light emitting diode, a liquid crystal display element, or a flashlamp.

19. The computing device of claim 15, wherein the second image information is captured at least in part by timing at least one of a capture time of each first pixel and a transfer time to each second pixel in a pair of pixels.

20. The computing device of claim 15, wherein the gesture sensor has a form factor corresponding to at least one of a complementary metal-oxide-semiconductor (CMOS) or a charge-coupled device (CCD) camera.

21. The computing device of claim 15, wherein the gesture sensor is a quarter video graphics array (QVGA) sensor.

22. A non-transitory computer-readable storage medium storing instructions for controlling an electronic device, the instructions when executed by at least one device processor of the electronic device causing the at least one device processor to:

capture first image information at a first resolution and a first frame rate using a subset of a set of pixels of a gesture sensor, wherein the set of pixels includes a plurality of pairs of pixels, each pair of pixels including a first pixel to be exposed to incident light and a second pixel to be shielded from the incident light and for storing charge received from the first pixel in response to the incident light;

analyze, using a processor of the gesture sensor, the first image information to to detect motion near the electronic computing device by determining that a difference between (a) a first ratio of (i) a first group of pixels, of a first image of the first image information, located substantially in a first region of the gesture sensor, and (ii) a second group of pixels, of the first image, located substantially in a second region of the gesture sensor, and (b) a third group of pixels, of a second image of the first image information, located substantially in the first region, and (ii) a fourth group of pixels, of the second image, located substantially in the second region, exceeds a predetermined threshold, wherein the first region and the second region are opposing edges of the gesture sensor;

capture second image information at a second resolution and a second frame rate using the gesture sensor, wherein the second resolution is greater than the first resolution, and the second frame rate is greater than the first frame rate;

detect one or more gestures in the second image information; and

send a signal from the processor of the gesture sensor to the at least one device processor, the processor of the gesture sensor being separate from the at least one device processor,

wherein information for a detected gesture can be compared against a set of known gestures to provide input to the electronic computing device.

23. The non-transitory computer-readable storage medium of claim 22, wherein the instructions when executed further cause the at least one device processor to:

selectively produce illumination during a time of image capture of the at least one gesture sensor using at least one illumination element.

24. The non-transitory computer-readable storage medium of claim 22, wherein the instructions when executed further cause the at least one device processor to:

determine an amount of ambient light using at least one light sensor for, the amount of ambient light being used to determine, at least in part, when to produce illumination using the at least one illumination element.