FEATURE COMPUTATION IN A SENSOR ELEMENT ARRAY

• US 20160094800A1
• Filed: 09/23/2015
• Published: 03/31/2016
• Est. Priority Date: 09/30/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Gousev, Evgeni Petrovich, Govil, Alok, Kim, Soo Youn, Rangan, Venkat, Rasquinha, Nelson
• Granted Patent: US 9,838,635 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 1/3203 Power management, i.e. event-based initiation of a power-saving mode
• CPC Class Codes: G06V 10/20 Image preprocessing
• CPC Class Codes: G06V 10/449 Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/507 Summing image-intensity values; Histogram projection analysis
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/651 for reducing power consumption by affecting camera operations, e.g. sleep mode, hibernation mode or power off of selective parts of the camera
• CPC Class Codes: H04N 25/75 Circuitry for providing, modifying or processing image signals from the pixel array

Abstract

Techniques describe computing computer vision (CV) features based on sensor readings from a sensor and detecting macro-features based on the CV features. The sensor may include a sensor element array that includes a plurality of sensor elements. The sensor may also include in-pixel circuitry coupled to the sensor elements, peripheral circuitry and/or a dedicated microprocessor coupled to the sensor element array. The in-pixel circuitry, the peripheral circuitry or the dedicated microprocessor may include computation structures configured to perform analog or digital operations representative of a multi-pixel computation for a sensor element (or block of sensor elements), based on sensor readings generated by neighboring sensor elements in proximity to the sensor element, and to generate CV features. The dedicated microprocessor may process the CV features and detect macro-features. Furthermore, in certain embodiments, the dedicated microprocessor may be coupled to a second microprocessor through a wired or wireless interface.

Claims

1. A vision sensor comprising:

a sensor element array comprising a plurality of sensor elements, the plurality of sensor elements arranged along at least a first dimension and a second dimension of the sensor element array, each of the plurality of sensor elements capable of generating a signal based on light incident upon the plurality of sensor elements, the signals corresponding to the plurality of sensor elements representing an image;

dedicated computer vision (CV) computation hardware configured to compute a localized CV feature for a block of one or more subject sensor elements based on, at least in part, signals associated with a plurality of neighboring sensor elements in proximity to the block of the one or more subject sensor elements; and

a dedicated microprocessor coupled to the dedicated CV computation hardware, wherein the dedicated microprocessor includes an interface for communication with a second microprocessor.

2. The vision sensor of claim 1, wherein the interface for communication with the second microprocessor uses one of serial peripheral interface (SPI), Inter-Integrated Circuit (I2C), or low voltage differential signaling (LVDS).

3. The vision sensor of claim 1, wherein a mobile device comprises the vision sensor and the second microprocessor, wherein the dedicated microprocessor is coupled to the second microprocessor through a wired interface, and wherein the second microprocessor is a higher power processor than the dedicated microprocessor.

4. The vision sensor of claim 1, further comprising a wireless module for communication with the second microprocessor, wherein the wireless module for communication with the second microprocessor is coupled to the dedicated microprocessor using the interface for communication with the second microprocessor.

5. The vision sensor of claim 4, wherein the wireless module is configured to communicate using a Zigbee (IEEE 802.15.4 standard), Bluetooth®

, body area network (IEEE 802.15.6), wireless USB, Wi-Fi (802.11), Z-wave, or IrDA (IR-based communications).

6. The vision sensor of claim 1, wherein the block of one or more subject sensor elements for which the localized CV feature is computed is a single subject sensor element.

7. The vision sensor of claim 1, wherein the dedicated CV computation hardware computes a local binary pattern (LBP) label.

8. The vision sensor of claim 7, further comprising integration hardware for computing an integral image of at least a part of the image based on at least a subset of the signals corresponding to a window of the image, wherein the dedicated CV computation hardware has access to the computed integral image for computation of combinations, sums, or averages of signals corresponding to blocks of sensor elements.

9. The vision sensor of claim 8, wherein the CV computation hardware is coupled to cascade classifier hardware configured to detect a presence or an absence of a reference object in the window of the image.

10. The vision sensor of claim 9, wherein the dedicated microprocessor is configured to receive an indication of the presence of the reference object when the presence of the reference object is detected.

11. The vision sensor of claim 1, wherein the dedicated microprocessor is configured to detect a presence or an absence of a reference object in a window of the image based on localized CV features received from the dedicated CV computation hardware.

12. The vision sensor of claim 1, wherein the dedicated CV computation hardware computes a histogram of signed gradients (HSG) feature.

13. The vision sensor of claim 1, wherein the localized CV features correspond to one or more of a spot, an edge, or a corner of a line edge.

14. The vision sensor of claim 1, wherein the sensor element array and the CV computation hardware are connected without intervening image signal processing circuitry.

15. The vision sensor of claim 14, wherein the signals received by the CV computation hardware from the sensor element array have not undergone one or more of defect correction, white balancing, color balancing, autofocus, lens roll off, demosaicing, debayering, and/or image sharpening.

16. A method comprising:

receiving sensor readings based on light incident upon sensor elements from a plurality of sensor elements, wherein the plurality of sensor elements are from a sensor element array, the plurality of sensor elements arranged along at least a first dimension and a second dimension of the sensor element array;

computing, with dedicated computer vision (CV) hardware, one or more localized CV features for a block of one or more subject sensor elements based on, at least in part, signals associated with a plurality of neighboring sensor elements in proximity to the block of the one or more sensor elements; and

obtaining, at a dedicated microprocessor coupled to the dedicated CV hardware, an indication of a reference object detected based on the computed one or more localized CV features.

17. The method of claim 16, wherein the dedicated microprocessor includes an interface for communication with a second microprocessor.

18. The method of claim 16, wherein the interface is one of a wired interface or wireless interface.

19. The method of claim 18, further comprising sending, from the dedicated microprocessor, a signal to the second microprocessor based on the indication of the reference object detected.

20. A vision sensor comprising:

means for receiving sensor readings based on light incident upon sensor elements from a plurality of sensor elements, wherein the plurality of sensor elements are from a sensor element array, the plurality of sensor elements arranged along at least a first dimension and a second dimension of the sensor element array;

means for computing one or more localized computer vision (CV) features based on sensor readings from at least neighboring sensor elements for each block of one or more of the sensor elements; and

means for detecting the presence or absence of a reference object based on the computed one or more localized CV features.

21. The vision sensor of claim 20, wherein the means for computing a localized CV feature comprises means for computing a local binary pattern (LBP) label.

22. The vision sensor of claim 20, wherein the means for computing a localized CV feature comprises means for computing a histogram of signed gradients (HSG) label.

23. The vision sensor of claim 20, wherein the localized CV features correspond to one or more of a spot, an edge, or a corner of a line edge.

24. A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises instructions executable by a dedicated computer vision (CV) microprocessor for:

receiving sensor readings based on light incident upon sensor elements from a plurality of sensor elements, wherein the plurality of sensor elements are from a sensor element array, the plurality of sensor elements arranged along at least a first dimension and a second dimension of the sensor element array;

computing, with dedicated computer vision (CV) hardware, one or more localized CV features for a block of one or more subject sensor elements based on, at least in part, signals associated with a plurality of neighboring sensor elements in proximity to the block of the one or more sensor elements; and

obtaining, at a dedicated microprocessor coupled to the dedicated CV hardware, an indication of a reference object detected based on the computed one or more localized CV features.

25. The non-transitory computer-readable storage medium of claim 24, wherein the means for computing a localized CV feature comprises means for computing a local binary pattern (LBP) label.

26. The non-transitory computer-readable storage medium of claim 24, wherein the means for computing a localized CV feature comprises means for computing a histogram of signed gradients (HSG) feature.

27. The non-transitory computer-readable storage medium of claim 24, wherein the localized CV features correspond to one or more of a spot, an edge, or a corner of a line edge.