Single-processor computer vision hardware control and application execution

• US 10,515,284 B2
• Filed: 01/23/2017
• Issued: 12/24/2019
• Est. Priority Date: 09/30/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Gousev, Evgeni, Govil, Alok, Maitan, Jacek, Rasquinha, Nelson, Rangan, Venkat, Park, Edwin Chongwoo
• Primary Examiner(s): Chio, Tat C
• Application Number:
  US15/413,390
• Publication Number:
  US 20170161579A1
• Time in Patent Office: 1,065 Days
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/3215 Monitoring of peripheral devices
• CPC Class Codes: G06F 1/3231 Monitoring the presence, absence or movement of users
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 10/20 Image preprocessing
• CPC Class Codes: G06V 10/449 Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/651 for reducing power consumption by affecting camera operations, e.g. sleep mode, hibernation mode or power off of selective parts of the camera
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

Apparatuses, methods, and systems are presented for reacting to scene-based occurrences. Such an apparatus may comprise dedicated computer vision (CV) computation hardware configured to receive sensor data from a sensor array comprising a plurality of sensor pixels and capable of computing one or more CV features using readings from neighboring sensor pixels of the sensor array. The apparatus may further comprise a first processing unit configured to control operation of the dedicated CV computation hardware. The first processing unit may be further configured to execute one or more application programs and, in conjunction with execution of the one or more application programs, communicate with at least one input/output (I/O) device controller, to effectuate an I/O operation in reaction to an event generated based on operations performed on the one or more computed CV features.

Claims

1. An apparatus for reacting to scene-based occurrences, the apparatus comprising:

dedicated computer vision (CV) computation hardware configured to receive sensor data from a sensor array comprising a plurality of sensor pixels and capable of computing one or more CV features using readings from neighboring sensor pixels of the sensor array; and

a first processing unit configured to control operation of the dedicated CV computation hardware by specifying parameters associated with and affecting the computation of the one or more CV features,

wherein the first processing unit is configured to operate in a low-power state while the dedicated CV computation hardware computes the one or more CV features using readings from the neighboring sensor pixels of the sensor array, and

wherein the first processing unit is further configured to execute one or more application programs and, in conjunction with execution of the one or more application programs, communicate with at least one input/output (I/O) device controller, to effectuate an audio or visual I/O operation in reaction to a face detection event generated based on operations performed on the one or more computed CV features.

2. The apparatus of claim 1, further comprising:

a classifier configured to receive an output of the dedicated CV computation hardware and perform the operations based on the one or more computed CV features, and

wherein the face detection event is generated in response to an output of the classifier.

3. The apparatus of claim 1, wherein the dedicated CV computation hardware is configured to provide an indication of the face detection event to the first processing unit, in response to polling by the first processing unit.

4. The apparatus of claim 1, wherein the face detection event is generated by the first processing unit.

5. The apparatus of claim 1,

wherein the first processing unit comprises a first core and a second core,

wherein the first core is configured to control the operation of the dedicated CV computation hardware;

and

wherein the second core is configured to execute the one or more application programs.

6. The apparatus of claim 1,

wherein the first processing unit supports a software environment;

and

wherein the software environment supports the one or more application programs.

7. The apparatus of claim 1, wherein the I/O device controller comprises a media device controller.

8. The apparatus of claim 7, wherein the media device controller is configured to generate an audio output.

9. The apparatus of claim 7, wherein the media device controller is configured to generate a visual output.

10. The apparatus of claim 1, wherein the I/O device controller comprises a proportional-integral-derivative (PID) controller.

11. The apparatus of claim 10, wherein the PID controller is configured to generate a control signal to adjust a movable member to a desired position.

12. A method for reacting to scene-based occurrences, the method comprising:

receiving, at dedicated computer vision (CV) computation hardware, sensor data from a sensor array comprising a plurality of sensor pixels;

computing, at the dedicated CV computation hardware, one or more CV features using readings from neighboring sensor pixels of the sensor array;

controlling, using a first processing unit, operation of the dedicated CV computation hardware by specifying parameters associated with and affecting the computation of the one or more CV features,

wherein the first processing unit is operated in a low-power state while the one or more CV features are computed at the dedicated CV computation hardware using readings from the neighboring sensor pixels of the sensor array;

executing, at the first processing unit, one or more application programs; and

communicating, at the first processing unit and in conjunction with execution of the one or more application programs, with at least one input/output (I/O) device controller, to effectuate an audio or visual I/O operation in reaction to a face detection event generated based on operations performed on the one or more computed CV features.

13. The method of claim 12, further comprising:

receiving, at a classifier, an output of the dedicated CV computation hardware and performing the operations based on the one or more computed CV features, and

wherein the face detection event is generated in response to an output of the classifier.

14. The method of claim 12, wherein the dedicated CV computation hardware provides an indication of the face detection event to the first processing unit, in response to polling by the first processing unit.

15. The method of claim 12, wherein the I/O device controller comprises a media device controller.

16. The method of claim 15, wherein the media device controller is configured to generate an audio output.

17. The method of claim 15, wherein the media device controller is configured to generate a visual output.

18. The method of claim 12, wherein the I/O device controller comprises a proportional-integral-derivative (PID) controller.

19. The method of claim 18, wherein the PID controller is configured to generate a control signal to adjust a movable member to a desired position.

20. An apparatus for reacting to scene-based occurrences, the apparatus comprising:

dedicated computer vision (CV) computation hardware configured to receive sensor data from a sensor array comprising a plurality of sensor pixels and capable of computing one or more CV features using readings from neighboring sensor pixels of the sensor array; and

a first processing unit configured to control the dedicated CV computation hardware by specifying parameters associated with and affecting the computation of the one or more CV features,

wherein the first processing unit is configured to operate in a low-power state while the dedicated CV computation hardware computes the one or more CV features using readings from the neighboring sensor pixels of the sensor array, and

wherein the first processing unit is further configured communicate with at least one device controller to control an audio or visual device in reaction to a face detection event generated based on the one or more computed CV features.

21. The apparatus of claim 20, further comprising:

a classifier configured to receive an output of the dedicated CV computation hardware and perform operations based on the one or more computed CV features, and

wherein the face detection event is generated in response to an output of the classifier.

22. The apparatus of claim 20, wherein the face detection event is generated by the first processing unit.

23. The apparatus of claim 20, wherein the device controller comprises a proportional-integral-derivative (PID) controller.

24. The apparatus of claim 23, wherein the PID controller is configured to generate a control signal to control the actuator device and adjust a movable member coupled to the actuator device to a desired position.

25. A method for reacting to scene-based occurrences, the method comprising:

receiving, at dedicated computer vision (CV) computation hardware, sensor data from a sensor array comprising a plurality of sensor pixels;

computing, at the dedicated CV computation hardware, one or more CV features using readings from neighboring sensor pixels;

controlling, using a first processing, the dedicated CV computation hardware by specifying parameters associated with and affecting the computation of the one or more CV features,

wherein the first processing unit is operated in a low-power state while the one or more CV features are computed at the dedicated CV computation hardware using readings from the neighboring sensor pixels of the sensor array; and

communicating, using the first processing unit, with at least one device controller, to control an audio or visual device in reaction to a face detection event generated based on the one or more computed CV features.