Configurable hardware for computing computer vision features

• US 9,762,834 B2
• Filed: 09/21/2015
• Issued: 09/12/2017
• Est. Priority Date: 09/30/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Govil, Alok, Gousev, Evgeni Petrovich, Kim, Soo Youn
• Primary Examiner(s): Hogue, Dennis
• Application Number:
  US14/859,533
• Publication Number:
  US 20160091946A1
• Time in Patent Office: 722 Days
• CPC Class Codes: G06V 10/20 Image preprocessing
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 25/707 Pixels for event detection
• CPC Class Codes: H04N 25/76 Addressed sensors, e.g. MOS or CMOS sensors
• CPC Class Codes: H04N 25/77 Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components

Abstract

Techniques describe apparatus and method for generating computed results based on sensor readings for detecting features, such as edges, corners etc. The sensor apparatus may include a sensor element array that includes a plurality of sensor elements. The sensor elements may be arranged in a 2-dimensional array, such as columns and rows. The sensor elements may be capable of generating sensor reading based on environmental conditions. The sensor apparatus may include a dedicated computer vision (CV) computation hardware in in-pixel circuitry, peripheral circuitry or dedicated microprocessor coupled to the sensor element array and configured to receive output from one or more of sensor elements. The dedicated CV computation hardware may include configurable blocks for detecting features using CV operations, wherein the configurable blocks may be configured to switch between multiple CV operations, such as linear binary pattern (LBP) and/or histogram of signed gradient (HSG) computer vision operations.

Claims

1. A vision sensor comprising:

a sensor element array comprising a plurality of sensor elements, the sensor elements arranged along at least a first dimension and a second dimension of the sensor element array, each of the plurality of sensor elements capable of generating a signal based on light incident upon the sensor element; and

dedicated computer vision (CV) computation hardware capable of receiving image data from the sensor element array and configurable to serially compute CV features for one or more two-dimensional patches within the sensor element array based on signals from sensor elements in each of the one or more two-dimensional patches, the dedicated CV computation hardware including switches to allow the CV computation hardware to calculate a first type of CV feature in a first mode and to calculate a second type of CV feature in a second mode, wherein a portion of the dedicated CV computation hardware is bypassed using a bypass path to calculate the first type of CV feature or the second type of CV feature.

2. The vision sensor of claim 1, wherein the dedicated CV computation hardware is peripheral to the sensor element array.

3. The vision sensor of claim 1, wherein the first type of CV feature is a Local Binary Pattern (LBP) label.

4. The vision sensor of claim 3, wherein the second type of CV feature is a Histogram of Signed Gradients (HSG) label.

5. The vision sensor of claim 1, wherein the first type of CV feature is a first variation of an LBP label and the second type of CV feature is a second variation of an LBP label.

6. The vision sensor of claim 1, wherein the dedicated CV computation hardware is coupled to a dedicated microprocessor.

7. The vision sensor of claim 1, wherein the first type of CV feature is a feature from accelerated segment test (FAST) corner.

8. The vision sensor of claim 1, wherein the dedicated CV computation hardware is coupled to an application processor.

9. The vision sensor of claim 1, wherein the dedicated CV computation hardware comprises comparators.

10. The vision sensor of claim 1, wherein the dedicated CV computation hardware comprises circuitry for performing a weighted sum operation.

11. The vision sensor of claim 1, wherein the dedicated CV computation hardware comprises charge scaling circuitry.

12. The vision sensor of claim 1, wherein the dedicated CV computation hardware is configured to switch to a third mode for calculating the first type of CV feature and the second type of CV feature.

13. A method, comprising:

receiving sensor readings based on light incident upon a plurality of sensor elements forming a sensor element array, wherein the plurality of sensor elements are arranged along at least a first dimension and a second dimension of the sensor element array;

determining a mode to operate a dedicated computer vision (CV) computation hardware capable of receiving image data from the sensor element array and configurable to serially compute CV features for one or more two-dimensional patches within the sensor element array based on signals from sensor elements in each of the one or more two-dimensional patches, the dedicated CV computation hardware including switches to allow the CV computation hardware to calculate a first type of CV feature in a first mode and to calculate a second type of CV feature in a second mode, wherein a portion of the dedicated CV computation hardware is bypassed using a bypass path to calculate the first type of CV feature or the second type of CV feature; and

switching the dedicated CV computation hardware to the first mode or the second mode for computing the corresponding type of CV feature based on the determined mode.

14. The method of claim 13, wherein the dedicated CV computation hardware is peripheral to the sensor element array.

15. The method of claim 13, wherein the first type of CV feature is a Local Binary Pattern (LBP) label.

16. The method of claim 15, wherein the second type of CV feature is a Histogram of Signed Gradients (HSG) label.

17. The method of claim 13, wherein the first type of CV feature is a first variation of an LBP label and the second type of CV feature is a second variation of an LBP label.

18. The method of claim 13, wherein the dedicated CV computation hardware is coupled to a dedicated microprocessor.

19. The method of claim 13, wherein the first type of CV feature is a feature from accelerated segment test (FAST) corner.

20. The method of claim 13, wherein the dedicated CV computation hardware is coupled to an application processor.

21. The method of claim 13, wherein the dedicated CV computation hardware comprises comparators.

22. The method of claim 13, wherein the dedicated CV computation hardware comprises circuitry for performing a weighted sum operation.

23. The method of claim 13, wherein the dedicated CV computation hardware comprises charge scaling circuitry.

24. The method of claim 13, wherein the dedicated CV computation hardware is configured to switch to a third mode for calculating the first type of CV feature and the second type of CV feature.

25. An apparatus comprising:

means for receiving sensor readings based on light incident upon a plurality of sensor elements forming a sensor element array, wherein the plurality of sensor elements are arranged along at least a first dimension and a second dimension of the sensor element array;

means for determining a mode to operate a dedicated computer vision (CV) computation hardware capable of receiving image data from the sensor element array and configurable to serially compute CV features for one or more two-dimensional patches within the sensor element array based on signals from sensor elements in each of the one or more two-dimensional patches, the dedicated CV computation hardware including switches to allow the CV computation hardware to calculate a first type of CV feature in a first mode and to calculate a second type of CV feature in a second mode, wherein a portion of the dedicated CV computation hardware is bypassed using a bypass path to calculate the first type of CV feature or the second type of CV feature; and

means for switching the dedicated CV computation hardware to the first mode or the second mode for computing the corresponding type of CV feature based on the determined mode.

26. The apparatus of claim 25, wherein the first type of CV feature is a Local Binary Pattern (LBP) label.

27. The apparatus of claim 26, wherein the second type of CV feature is a Histogram of Signed Gradients (HSG) label.

28. A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises instructions executable by a processor for:

processing received sensor readings, the received sensor readings being based on light incident upon a plurality of sensor elements forming a sensor element array, wherein the plurality of sensor elements are arranged along at least a first dimension and a second dimension of the sensor element array;

determining a mode to operate a dedicated computer vision (CV) computation hardware capable of receiving image data from the sensor element array and configurable to serially compute CV features for one or more two-dimensional patches within the sensor element array based on signals from sensor elements in each of the one or more two-dimensional patches, the dedicated CV computation hardware including switches to allow the CV computation hardware to calculate a first type of CV feature in a first mode and to calculate a second type of CV feature in a second mode, wherein a portion of the dedicated CV computation hardware is bypassed using a bypass path to calculate the first type of CV feature or the second type of CV feature; and

switching the dedicated CV computation hardware to the first mode or the second mode for computing the corresponding type of CV feature based on the determined mode.