Low-power always-on face detection, tracking, recognition and/or analysis using events-based vision sensor

• US 9,870,506 B2
• Filed: 01/03/2017
• Issued: 01/16/2018
• Est. Priority Date: 09/30/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Gousev, Evgeni, Govil, Alok, Maitan, Jacek, Rasquinha, Nelson, Rangan, Venkat
• Primary Examiner(s): CHIO, TAT CHI
• Application Number:
  US15/397,566
• Publication Number:
  US 20170116478A1
• Time in Patent Office: 378 Days
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06V 10/20 Image preprocessing
• CPC Class Codes: G06V 10/449 Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/651 for reducing power consumption by affecting camera operations, e.g. sleep mode, hibernation mode or power off of selective parts of the camera
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

Techniques disclosed herein utilize a vision sensor that integrates a special-purpose camera with dedicated computer vision (CV) computation hardware and a dedicated low-power microprocessor for the purposes of detecting, tracking, recognizing, and/or analyzing subjects, objects, and scenes in the view of the camera. The vision sensor processes the information retrieved from the camera using the included low-power microprocessor and sends “events” (or indications that one or more reference occurrences have occurred, and, possibly, associated data) for the main processor only when needed or as defined and configured by the application. This allows the general-purpose microprocessor (which is typically relatively high-speed and high-power to support a variety of applications) to stay in a low-power (e.g., sleep mode) most of the time as conventional, while becoming active only when events are received from the vision sensor.

Claims

1. A smart sensor for sensing dynamic scene-based occurrences, the smart sensor comprising:

dedicated computer vision (CV) computation hardware configured to receive sensor data from a sensor array comprising more than one sensor pixel and capable of computing one or more CV features using readings from neighboring sensor pixels of the sensor array;

a classifier configured to receive an output of the dedicated CV computation hardware and perform operations based on the one or more computed CV features; and

a first processing unit configured to process an output of the classifier resulting from performing the operations based on the one or more computed CV features;

wherein the smart sensor is configured to generate an event to be received by a second processing unit in response to the processing of the output of the classifier by the first processing unit, the event configured to trigger an operation to be performed by the second processing unit and indicating a reference occurrence for the second processing unit.

2. The smart sensor of claim 1, wherein the reference occurrence is one or more of:

a coming into view of a human face,

a coming into view of a human body,

an emotion expressed on a human face,

coming into view of an non-human animal face,

coming into view of a non-human animal body,

coming into view of a human hand,

a hand gesture,

a coming into view of a reference object,

a change from an indoor environment to an outdoor environment,

a reference movement,

rapid movement in a scene indicating a fall,

motion toward an object indicating a risk of collision,

movement or objects in a scene indicating danger, or

any combination thereof.

3. The smart sensor of claim 1, wherein the classifier is implemented in software, executed by the first processing unit.

4. The smart sensor of claim 1, wherein the dedicated CV computation hardware is configured to provide Local Binary Patterns (LBPs).

5. The smart sensor of claim 1, wherein the classifier comprises a hardware cascade classifier configured to detect a presence of a reference object in a subset of the sensor data, and the reference occurrence is associated with the reference object.

6. The smart sensor of claim 5, wherein the first processing unit is configured to receive an indication from the hardware cascade classifier of the presence of the reference object when the presence of the reference object is detected by the cascade classifier hardware.

7. The smart sensor of claim 1, wherein the dedicated CV computation hardware is configured to receive raw sensor data from the sensor array and no image signal processing circuitry is disposed between the sensor array and the dedicated CV computation hardware.

8. The smart sensor of claim 1, wherein the dedicated CV computation hardware comprises an integrated circuit.

9. The smart sensor of claim 1, wherein the smart sensor is configured to disable at least part of the dedicated CV computation hardware while operating in a lower-power mode, detect a scene change in the lower-power mode, and to activate the dedicated CV computation hardware based on detection of the scene change.

10. The smart sensor of claim 1, wherein the event further includes data associated with the reference occurrence.

11. The smart sensor of claim 1, wherein the smart sensor is configured to:

perform a higher-power operation as at least part of the operations based on the one or more computed CV features, the higher-power operation consuming more power than a lower-power operation, and

provide a parameter for the lower-power operation based on the higher-power operation.

12. The smart sensor of claim 11, wherein the lower-power operation comprises one or more chosen from the list comprising:

ambient light sensing, proximity detection, motion detection, or change detection.

13. The smart sensor of claim 1, wherein the smart sensor is configured to:

perform a lower-power operation as at least part of the operations based on the one or more computed CV features, the lower-power operation consuming less power than a higher-power operation, and

provide a sensor reading for the higher-power operation based on the lower-power operation.

14. The smart sensor of claim 13, wherein the lower-power operation comprises one or more chosen from the list comprising:

ambient light sensing, proximity detection, motion detection, or change detection.

15. An apparatus for determining object detection events, the apparatus comprising:

a sensor array comprising more than one sensor pixel;

dedicated computer vision (CV) computation hardware capable of receiving sensor data from the sensor array and computing CV features using readings from neighboring sensor pixels of the sensor array;

a classifier configured to receive an output of the dedicated CV computation hardware and perform operations based on the one or more computed CV features; and

a first processing unit configured to;

determine, from an output of the classifier resulting from performing the operations based on the one or more computed CV features that an object has been detected, and

in response to the determination, generate an object-detected event to be received by a second processing unit and configured to trigger an operation to be performed by the second processing unit.

16. The apparatus of claim 15, further comprising the second processing unit, wherein the first processing unit is configured to communicate the object-detected event to the second processing unit while the second processing unit is operating in a low-power mode.

17. The apparatus of claim 15, wherein the dedicated CV computation hardware is configured to receive raw sensor data from the sensor array and no image signal processing circuitry is disposed between the sensor array and the dedicated CV computation hardware.

18. The apparatus of claim 15, further comprising peripheral circuitry configured to provide at least one of:

a timing operation,

a focusing operation,

an auto-exposure correction operation,

object detection,

object recognition,

storing a scanning window,

an event-queuing and/or processing operation,

analog processing,

analog-to-digital conversion,

an integration operation,

CV feature computation,

a histogram-based classification, or

memory buffering, or

any combination thereof.

19. A method for sensing dynamic scene-based occurrences with an apparatus, the method comprising:

receiving, with dedicated computer vision (CV) computation hardware, sensor data from a sensor array comprising more than one sensor pixel;

computing, with the dedicated CV computation hardware, one or more CV features using readings from neighboring sensor pixels of the sensor array;

performing, with a classifier, operations based on the one or more computed CV features;

processing, with a first processing unit, an output of the classifier resulting from performing the operations based on the one or more computed CV features; and

generating an event to be received by a second processing unit in response to the processing of the output of the classifier by the first processing unit, the event configured to trigger an operation to be performed by the second processing unit and indicating a reference occurrence for the second processing unit.

20. The method of claim 19, wherein the reference occurrence is one or more of:

a coming into view of a human face,

a coming into view of a human body,

an emotion expressed on a human face,

coming into view of an non-human animal face,

coming into view of a non-human animal body,

coming into view of a human hand,

a hand gesture,

a coming into view of a reference object,

a change from an indoor environment to an outdoor environment,

a reference movement,

rapid movement in a scene indicating a fall,

motion toward an object indicating a risk of collision,

movement or objects in a scene indicating danger, or

any combination thereof.

21. The method of claim 19, further comprising providing, with the dedicated CV computation hardware, Local Binary Patterns (LBPs).

22. The method of claim 19, wherein the classifier comprises hardware configured to detect a presence of a reference object in a subset of the sensor data, the reference occurrence being associated with the reference object.

23. The method of claim 22, further comprising, receiving, by the first processing unit, an indication from the classifier hardware of the presence of the reference object when the presence of the reference object is detected by the classifier hardware.

24. The method of claim 19, further comprising disabling at least part of the dedicated CV computation hardware to generate at least one lower-power optical sensor reading indicative of ambient light sensing, a proximity detection , a proximity to a reference object, a motion detection, or any combination thereof.

25. The method of claim 24, further comprising detecting the reference occurrence based on the at least one lower-power optical sensor reading.

26. The method of claim 19, further comprising:

disabling at least part of the dedicated CV computation hardware while operating in a lower-power mode;

detecting a scene change while operating in the lower-power mode; and

activating the disabled at least part of the dedicated CV computation hardware based on detection of the scene change.

27. The method of claim 19, wherein, receiving, with the dedicated CV computation hardware, the sensor data from the sensor array comprises receiving raw sensor data from the sensor array and wherein no image signal processing is performed on the sensor data prior to the receiving.

28. A method for determining object detection events, the method comprising:

receiving, with dedicated computer vision (CV) computation hardware, sensor data from a sensor array;

computing one or more CV features using readings from neighboring sensor pixels of the sensor array;

performing, with a classifier, operations based on the one or more computed CV features; and

using a first processing unit to;

determine, from an output of the classifier resulting from performing the operations based on the one or more computed CV features that an object has been detected, and

in response to the determination, generate an object-detected event to be received by a second processing unit and configured to trigger an operation to be performed by the second processing unit.

29. The method of claim 28, further comprising using the first processing unit to communicate the object-detected event to the second processing unit while the second processing unit is operating in a low-power mode.

30. The method of claim 28, wherein, receiving, with the dedicated CV computation hardware, the sensor data from the sensor array comprises receiving raw sensor data from the sensor array and wherein no image signal processing is performed on the sensor data prior to the receiving.