Hardware acceleration of computer vision feature detection

• US 9,923,004 B2
• Filed: 09/23/2015
• Issued: 03/20/2018
• Est. Priority Date: 09/30/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Govil, Alok, Kim, Soo Youn, Gousev, Evgeni Petrovich
• Primary Examiner(s): Tran, Nhan T
• Application Number:
  US14/862,314
• Publication Number:
  US 20160091947A1
• Time in Patent Office: 909 Days
• Field of Search: 348294, 348308, 348310
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: H01L 27/1461 characterised by the photosensitive area
• CPC Class Codes: H01L 27/14643 Photodiode arrays; MOS imagers
• CPC Class Codes: H04N 25/70 SSIS architectures; Circuits associated therewith

Abstract

Techniques describe apparatus and method for generating local binary pattern (LBP) labels based on sensor readings from sensor elements. The sensor apparatus may include a sensor element array that includes a plurality of sensor elements and may also include in-pixel circuitry coupled directly to the sensor element, peripheral circuitry coupled to the sensor element array and configured to receive output from one or more of sensor elements and digital circuitry. The in-pixel circuitry and/or peripheral circuitry may include an analog and/or digital computation structure configured to generate an LBP label for each of the sensor elements readings, by currently comparing the sensor readings for the referenced sensor element with the sensor readings of four or less neighboring sensor elements, and using previously, or subsequently, generated comparisons for the remaining neighboring sensor elements.

Claims

1. An apparatus for generating computed results based on sensor readings, the apparatus comprising:

a sensor element array comprising a plurality of sensor elements, the plurality of sensor elements arranged along at least a first dimension and a second dimension of the sensor element array, each of the plurality of sensor elements capable of generating a sensor reading based on light incident upon the sensor element;

a memory;

at least one computation structure configured to generate a computer vision (CV) label for a subject sensor element from the plurality of sensor elements, by, when generating the CV label for the subject sensor element,

only performing comparisons of the sensor reading for the subject sensor element with the sensor reading of each of a subset of a plurality of neighboring sensor elements to the subject sensor element, the plurality of neighboring sensor elements being adjacent to the subject sensor element, and

retrieving from the memory previously performed comparisons of the sensor reading for the subject sensor element with the sensor reading of each of remaining neighboring sensor elements to the subject sensor element, wherein the previously performed comparisons were performed when generating the CV label for sensor elements other than the subject sensor element and wherein the comparisons from the subset of the plurality of neighboring sensor elements and the comparisons from the remaining neighboring sensor elements are together required for generating the CV label for the subject sensor element and wherein the subset is smaller than the plurality of neighboring sensor elements.

2. The apparatus of claim 1, wherein the CV label for the subject sensor element is a local binary pattern (LBP) label.

3. The apparatus of claim 2, wherein the comparisons from the subset of the plurality of neighboring sensor elements and the comparisons from the remaining neighboring sensor elements comprise eight computed comparisons required for generating the LBP label.

4. The apparatus of claim 3, wherein the previously performed comparisons performed when generating the CV label for a sensor element other than the subject sensor element comprise at least four comparisons computed while generating LBP labels for the sensor elements other than the subject sensor element.

5. The apparatus of claim 1, wherein the plurality of neighboring sensor elements to the subject sensor element are immediately adjacent sensor elements to the subject sensor element.

6. The apparatus of claim 1, wherein the sensor element array comprises an array of pixels wherein each pixel comprises the sensor element and in-pixel circuitry, and the at least one computation structure is part of in-pixel circuitry coupled to the sensor element.

7. The apparatus of claim 1, wherein the at least one computation structure is part of peripheral circuitry coupled to the sensor element array and configured to receive output from the plurality of sensor elements including the subject sensor element and the plurality of neighboring sensor elements.

8. The apparatus of claim 1, wherein the computation structure comprises one or more comparators configured to:

perform the comparisons of the sensor reading for the subject sensor element with the sensor readings of each of the subset of the plurality of neighboring sensor elements to determine threshold values for the respective neighboring sensor elements with respect to the subject sensor element; and

retrieve from the memory the threshold values generated for the remaining neighboring sensor elements from the previously performed comparisons of the sensor reading for the subject sensor element with the sensor reading of each of the remaining neighboring sensor elements to the subject sensor element.

9. The apparatus of claim 8, wherein a binary complement for the retrieved threshold values for the remaining neighboring sensor elements of the plurality of the neighboring sensor elements is used for generating the CV label.

10. The apparatus of claim 1, wherein the sensor element array is a vision sensor, and each of the sensor elements comprises a photodiode.

11. The apparatus of claim 1, wherein the at least one computation structure comprises multiple parallel computation structures configured to compare the sensor reading for each of a plurality of subject sensor elements to a sensor element neighboring the each of the plurality of subject sensor elements simultaneously.

12. The apparatus of claim 1, wherein the CV label for the subject sensor element is generated for detecting features for a CV application.

13. The apparatus of claim 1, wherein the at least one computation structure is a digital computation structure.

14. A method for generating computed results based on sensor readings, comprising:

receiving a sensor reading based on light incident upon a subject sensor element from a plurality of sensor elements forming a sensor element array, wherein the plurality of sensor elements are arranged along at least a first dimension and a second dimension of the sensor element array;

performing comparisons, at a computation structure, of the sensor reading for the subject sensor element with the sensor reading of each of only a subset of a plurality of neighboring sensor elements to the subject sensor element to generate currently performed comparisons, the plurality of neighboring sensor elements being adjacent to the subject sensor element and the subset being smaller than the plurality of neighboring sensor elements;

retrieving, from one or more buffers, previously performed comparisons of the sensor reading for the subject sensor element with the sensor reading of each of remaining neighboring sensor elements to the subject sensor element, wherein the previously performed comparisons were performed when generating the CV label for sensor elements other than the subject sensor element and wherein the comparisons from the subset of the plurality of neighboring sensor elements and the comparisons from the remaining neighboring sensor elements are together required for generating the CV label for the subject sensor element; and

generating, at the computation structure, the CV label for the subject sensor element by using the currently performed comparisons and the previously performed comparisons.

15. The method of claim 14, wherein the CV label for the subject sensor element is a local binary pattern (LBP) label.

16. The method of claim 15, wherein the comparisons from the subset of the plurality of neighboring sensor elements and the comparisons from the remaining neighboring sensor elements comprise eight computed comparisons required for generating the LBP label.

17. The method of claim 16, wherein the previously performed comparisons performed when generating the CV label for a sensor element other than the subject sensor element comprise at least four comparisons computed while generating LBP labels for the sensor elements other than the subject sensor element.

18. The method of claim 14, wherein the plurality of neighboring sensor elements to the subject sensor element are immediately adjacent sensor elements to the subject sensor element.

19. The method of claim 14, wherein the sensor element array comprises an array of pixels, wherein each pixel comprises the sensor element and in-pixel circuitry, and the computation structure is part of the in-pixel circuitry coupled to the sensor element.

20. The method of claim 14, wherein the computation structure is part of peripheral circuitry coupled to the sensor element array.

21. The method of claim 14, wherein the sensor element array is a vision sensor, and each of the plurality of sensor elements comprises a photodiode.

22. The method of claim 14, wherein performing comparisons, at the computation structure, of the sensor reading for the subject sensor element with the sensor reading of each of only the subset of the plurality of neighboring sensor elements comprises comparing, at a parallel computation structure, the sensor reading for each of a plurality of subject sensor elements to a sensor element neighboring the each of the plurality of subject sensor elements simultaneously.

23. The method of claim 14, wherein CV labels for each of the plurality of sensor elements are generated for detecting features for a CV application.

24. The method of claim 14, wherein the at least one computation structure is a digital computation structure.

25. An apparatus for generating computed results based on sensor readings comprising:

means for receiving a sensor reading based on light incident upon a subject sensor element from a plurality of sensor elements forming a sensor element array, wherein the plurality of sensor elements are arranged along at least a first dimension and a second dimension of the sensor element array;

means for performing comparisons of the sensor reading for the subject sensor element with the sensor reading of each of only a subset of a plurality of neighboring sensor elements to generate currently performed comparisons, the plurality of neighboring sensor elements being adjacent to the subject sensor element and the subset being smaller than the plurality of neighboring sensor elements;

means for retrieving previously performed comparisons of the sensor reading for the subject sensor element with the sensor reading of each of remaining neighboring sensor elements to the subject sensor element, wherein the previously performed comparisons were performed when generating the CV label for sensor elements other than the subject sensor element and wherein the comparisons from the subset of the plurality of neighboring sensor elements and the comparisons from the remaining neighboring sensor elements are together required for generating the CV label for the subject sensor element;

means for generating the CV label for the subject sensor element by using the currently performed comparisons and the previously performed comparisons.

26. The apparatus of claim 25, wherein the CV label for the subject sensor element is a local binary pattern (LBP) label.

27. The apparatus of claim 26, wherein the comparisons from the subset of the plurality of neighboring sensor elements and the comparisons from the remaining neighboring sensor elements comprise eight computed comparisons required for generating the LBP label.

28. The apparatus of claim 25, wherein the plurality of neighboring sensor elements to the subject sensor element are immediately adjacent sensor elements to the subject sensor element.

29. A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises instructions executable by a processor for:

receiving a sensor reading based on light incident upon a subject sensor element from a plurality of sensor elements forming a sensor element array, wherein the plurality of sensor elements are arranged along at least a first dimension and a second dimension of the sensor element array;

receiving comparisons of the sensor reading for the subject sensor element with the sensor reading of each of only a subset of a plurality of neighboring sensor elements to generate currently performed comparisons, the plurality of neighboring sensor elements being adjacent to the subject sensor element and the subset being smaller than the plurality of neighboring sensor elements;

retrieving previously performed comparisons of the sensor reading for the subject sensor element with the sensor reading of each of remaining neighboring sensor elements to the subject sensor element, wherein the previously performed comparisons were performed when generating the CV label for sensor elements other than the subject sensor element and wherein the comparisons from the subset of the plurality of neighboring sensor elements and the comparisons from the remaining neighboring sensor elements are together required for generating the CV label for the subject sensor element;

generating the CV label for the subject sensor element by using the currently performed comparisons and the previously performed comparisons.

30. The non-transitory computer-readable storage medium of claim 29, wherein the CV label for the subject sensor element is a local binary pattern (LBP) label.