Feature computation in a sensor element array

• US 9,838,635 B2
• Filed: 09/23/2015
• Issued: 12/05/2017
• Est. Priority Date: 09/30/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Gousev, Evgeni Petrovich, Govil, Alok, Kim, Soo Youn, Rasquinha, Nelson, Rangan, Venkat
• Primary Examiner(s): Tran, Nhan T
• Application Number:
  US14/862,319
• Publication Number:
  US 20160094800A1
• Time in Patent Office: 804 Days
• CPC Class Codes: G06F 1/3203 Power management, i.e. event-based initiation of a power-saving mode
• CPC Class Codes: G06V 10/20 Image preprocessing
• CPC Class Codes: G06V 10/449 Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/507 Summing image-intensity values; Histogram projection analysis
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/651 for reducing power consumption by affecting camera operations, e.g. sleep mode, hibernation mode or power off of selective parts of the camera
• CPC Class Codes: H04N 25/75 Circuitry for providing, modifying or processing image signals from the pixel array

Abstract

Techniques describe computing computer vision (CV) features based on sensor readings from a sensor and detecting macro-features based on the CV features. The sensor may include a sensor element array that includes a plurality of sensor elements. The sensor may also include in-pixel circuitry coupled to the sensor elements, peripheral circuitry and/or a dedicated microprocessor coupled to the sensor element array. The in-pixel circuitry, the peripheral circuitry or the dedicated microprocessor may include computation structures configured to perform analog or digital operations representative of a multi-pixel computation for a sensor element (or block of sensor elements), based on sensor readings generated by neighboring sensor elements in proximity to the sensor element, and to generate CV features. The dedicated microprocessor may process the CV features and detect macro-features. Furthermore, in certain embodiments, the dedicated microprocessor may be coupled to a second microprocessor through a wired or wireless interface.

Claims

1. A vision sensor comprising:

a sensor element array comprising a plurality of sensor elements, the plurality of sensor elements arranged along at least a first dimension and a second dimension of the sensor element array, each of the plurality of sensor elements capable of generating a signal based on light incident upon the plurality of sensor elements, the signals corresponding to the plurality of sensor elements representing an image;

dedicated computer vision (CV) computation hardware configured to compute a localized CV feature for a block of one or more subject sensor elements based on, at least in part, signals associated with a plurality of neighboring sensor elements in proximity to the block of the one or more subject sensor elements;

integration hardware for computation of combinations, sums, or averages of signals corresponding to blocks of sensor elements corresponding to a subset of the signals representing the image, wherein the dedicated CV computation hardware has access to the computed combinations, sums, or averages; and

a dedicated microprocessor coupled to the dedicated CV computation hardware, wherein the dedicated microprocessor includes an interface for communication with a second microprocessor.

2. The vision sensor of claim 1, wherein the interface for communication with the second microprocessor uses one of serial peripheral interface (SPI), Inter-Integrated Circuit (I2C), or low voltage differential signaling (LVDS).

3. The vision sensor of claim 1, wherein a mobile device comprises the vision sensor and the second microprocessor, wherein the dedicated microprocessor is coupled to the second microprocessor through a wired interface, and wherein the second microprocessor is a higher power processor than the dedicated microprocessor.

4. The vision sensor of claim 1, further comprising a wireless module for communication with the second microprocessor, wherein the wireless module for communication with the second microprocessor is coupled to the dedicated microprocessor using the interface for communication with the second microprocessor.

5. The vision sensor of claim 4, wherein the wireless module is configured to communicate using a Zigbee (IEEE 802.15.4 standard), Bluetooth®

, body area network (IEEE 802.15.6), wireless USB, Wi-Fi (802.11), Z-wave, or IrDA (IR-based communications).

6. The vision sensor of claim 1, wherein the block of one or more subject sensor elements for which the localized CV feature is computed is a single subject sensor element.

7. The vision sensor of claim 1, wherein the dedicated CV computation hardware computes a local binary pattern (LBP) label.

8. The vision sensor of claim 7, wherein the integration hardware is capable of computing an integral image of the image.

9. The vision sensor of claim 8, wherein the CV computation hardware is coupled to cascade classifier hardware configured to detect a presence or an absence of a reference object in the window of the image.

10. The vision sensor of claim 9, wherein the dedicated microprocessor is configured to receive an indication of the presence of the reference object when the presence of the reference object is detected.

11. The vision sensor of claim 1, wherein the dedicated microprocessor is configured to detect a presence or an absence of a reference object in a window of the image based on localized CV features received from the dedicated CV computation hardware.

12. The vision sensor of claim 1, wherein the dedicated CV computation hardware computes a histogram of signed gradients (HSG) feature.

13. The vision sensor of claim 1, wherein the localized CV features correspond to one or more of a spot, an edge, or a corner of a line edge.

14. The vision sensor of claim 1, wherein the sensor element array and the CV computation hardware are connected without intervening image signal processing circuitry.

15. The vision sensor of claim 14, wherein the signals received by the CV computation hardware from the sensor element array have not undergone one or more of defect correction, white balancing, color balancing, autofocus, lens roll off, demosaicing, debayering, and/or image sharpening.

16. A method comprising:

receiving, at a vision sensor, sensor readings based on light incident upon sensor elements from a plurality of sensor elements of the sensor element array the plurality of sensor elements arranged along at least a first dimension and a second dimension of the sensor element array and each capable of generating a signal based on light incident upon the plurality of sensor elements, the signals corresponding to the plurality of sensor elements representing an image;

computing, with dedicated computer vision (CV) hardware of the vision sensor, one or more localized CV features for a block of one or more subject sensor elements based on, at least in part, signals associated with a plurality of neighboring sensor elements in proximity to the block of the one or more sensor elements;

computing, with integration hardware of the vision sensor, combinations, sums, or averages of signals corresponding to blocks of sensor elements corresponding to a subset of the signals representing the image, wherein the dedicated CV computation hardware has access to the computed combinations, sums, or averages; and

generating, at a dedicated microprocessor of the vision sensor that is coupled to the dedicated CV hardware, an indication of a reference object detected based on the computed one or more localized CV features.

17. The method of claim 16, wherein the dedicated microprocessor includes an interface for communication with a second microprocessor.

18. The method of claim 16, wherein the interface is one of a wired interface or wireless interface.

19. The method of claim 18, further comprising sending, from the dedicated microprocessor, a signal to the second microprocessor based on the indication of the reference object detected.

20. A vision sensor comprising:

means for receiving sensor readings based on light incident upon sensor elements from a plurality of sensor elements of the sensor element array the plurality of sensor elements arranged along at least a first dimension and a second dimension of the sensor element array and each capable of generating a signal based on light incident upon the plurality of sensor elements, the signals corresponding to the plurality of sensor elements representing an image;

means for computing one or more localized computer vision (CV) features based on sensor readings from at least neighboring sensor elements for each block of one or more of the sensor elements;

means for computing combinations, sums, or averages of signals corresponding to blocks of sensor elements corresponding to a subset of the signals representing the image, wherein the dedicated CV computation hardware has access to the computed combinations, sums, or averages;

means for generating an indication of a reference object based on the computed one or more localized CV features; and

means for communicating the indication of the reference object to a second microprocessor.

21. The vision sensor of claim 20, wherein the means for computing a localized CV feature comprises means for computing a local binary pattern (LBP) label.

22. The vision sensor of claim 20, wherein the means for computing a localized CV feature comprises means for computing a histogram of signed gradients (HSG) label.

23. The vision sensor of claim 20, wherein the localized CV features correspond to one or more of a spot, an edge, or a corner of a line edge.

24. A vision sensor comprising a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises instructions executable by a dedicated computer vision (CV) microprocessor for:

controlling a sensor element array for generating sensor readings based on light incident upon sensor elements from a plurality of sensor elements of the sensor element array the plurality of sensor elements arranged along at least a first dimension and a second dimension of the sensor element array;

controlling dedicated computer vision (CV) hardware for computing one or more localized CV features for a block of one or more subject sensor elements based on, at least in part, signals associated with a plurality of neighboring sensor elements in proximity to the block of the one or more sensor elements;

controlling integration hardware of the vision sensor for computing combinations, sums, or averages of signals corresponding to blocks of sensor elements corresponding to a subset of the signals representing the image, wherein the dedicated CV computation hardware has access to the computed combinations, sums, or averages; and

generating, at a dedicated microprocessor coupled to the dedicated CV hardware, an indication of a reference object detected based on the computed one or more localized CV features.

25. The non-transitory computer-readable storage medium of claim 24, further comprising instructions for controlling a communication interface for communication with a second microprocessor.

26. The non-transitory computer-readable storage medium of claim 25, wherein the communication interface comprises one of a wired interface or wireless interface.

27. The non-transitory computer-readable storage medium of claim 25, further comprising instructions for sending, from the dedicated microprocessor to the second microprocessor, the generated indication of the reference object detected.