Scanning window for isolating pixel values in hardware for computer vision operations

• US 9,940,533 B2
• Filed: 09/25/2015
• Issued: 04/10/2018
• Est. Priority Date: 09/30/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Govil, Alok, Rangan, Venkat, Rasquinha, Nelson, Seo, Hae-Jong
• Primary Examiner(s): ANSARI, TAHMINA N
• Application Number:
  US14/866,739
• Publication Number:
  US 20160092735A1
• Time in Patent Office: 928 Days
• Field of Search: None
• CPC Class Codes: G06V 10/20 Image preprocessing
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 25/443 by reading pixels from selected 2D regions of the array, e.g. for windowing or digital zooming
• CPC Class Codes: H04N 25/46 by combining or binning pixels
• CPC Class Codes: H04N 25/75 Circuitry for providing, modifying or processing image signals from the pixel array

Abstract

An apparatus includes a hardware sensor array including a plurality of pixels arranged along at least a first dimension and a second dimension of the array, each of the pixels capable of generating a sensor reading. A hardware scanning window array includes a plurality of storage elements arranged along at least a first dimension and a second dimension of the hardware scanning window array, each of the storage elements capable of storing a pixel value based on one or more sensor readings. Peripheral circuitry for systematically transfers pixel values, based on sensor readings, into the hardware scanning window array, to cause different windows of pixel values to be stored in the hardware scanning window array at different times. Control logic coupled to the hardware sensor array, the hardware scanning window array, and the peripheral circuitry, provides control signals to the peripheral circuitry to control the transfer of pixel values.

Claims

1. An apparatus for isolating pixel values in hardware for computer vision operations comprising:

a hardware sensor array comprising a plurality of pixels arranged along at least a first dimension and a second dimension of the hardware sensor array, each of the pixels capable of generating a sensor reading based on environmental conditions;

a hardware scanning window array comprising a plurality of storage elements arranged along at least a first dimension and a second dimension of the hardware scanning window array, each of the storage elements capable of storing a pixel value based on one or more sensor readings from the hardware sensor array;

peripheral circuitry for systematically transferring pixel values, based on sensor readings from the hardware sensor array, into the hardware scanning window array; and

control logic coupled to the hardware sensor array, the hardware scanning window array, and the peripheral circuitry, the control logic operable to provide control signals to the peripheral circuitry to control the transfer of pixel values into the hardware scanning window array,

wherein the transfer of pixel values causes a first window of pixels values to be stored in the hardware scanning window array at a first time and a second window of pixel values to be stored in the hardware scanning window array at a second time, the first window corresponding to a different position than the second window within the hardware sensor array.

2. The apparatus of claim 1, wherein the hardware sensor array further comprises:

configurable combining circuitry coupled to the plurality of pixels, the configurable combining circuitry operable to combine, in hardware, multiple sensor readings from the plurality of pixels to generate the pixel values, according to at least one combining pattern, wherein the configurable combining circuitry comprises;

a first plurality of controllable connections operable to connect adjacent pixels along the first dimension of the hardware sensor array; and

a second plurality of controllable connections operable to connect adjacent pixels along the second dimension of the hardware sensor array,

wherein connecting adjacent pixels generates averaged pixel values at each of the adjacent pixels being connected.

3. The apparatus of claim 2, wherein the sensor reading is an analog sensor reading, wherein the pixel value is an analog pixel value, and wherein the configurable combining circuitry is configurable analog combining circuitry.

4. The apparatus of claim 1, wherein the peripheral circuitry comprises:

a line buffer comprising a plurality of storage elements arranged along at least a first dimension and a second dimension of the line buffer, the first dimension of the line buffer being smaller than the first dimension of the hardware sensor array, the second dimension of the line buffer being equal to the second dimension of the hardware sensor array,

wherein the line buffer is capable of storing selected rows of pixel values from the hardware sensor array.

5. The apparatus of claim 4, wherein the control logic is capable of storing a next row of pixel values from the hardware sensor array into the line buffer, by replacing an oldest row of pixel values in the line buffer, thereby introducing a discontinuity along a row direction in an image represented by the pixel values stored in the line buffer.

6. The apparatus of claim 5, wherein the peripheral circuitry further comprises a multiplexer circuit positioned between the line buffer and the hardware scanning window array, the multiplexer circuit capable of switching row ordering while transferring pixel values from the line buffer to the hardware scanning window array, thereby removing the discontinuity along a row direction in an image represented by the pixel values stored in the line buffer.

7. The apparatus of claim 5, wherein the peripheral circuitry further comprises a multiplexer circuit positioned between the hardware scanning window array and a hardware computer vision feature computation block, the multiplexer circuit capable of switching column ordering while transferring pixel values from the hardware scanning window array to the hardware computer vision feature computation block, thereby removing the discontinuity along a column direction in an image represented by the pixel values stored in the hardware scanning window array.

8. The apparatus of claim 4, wherein the control logic is capable of storing a next column of pixel values from the line buffer into the hardware scanning window array, by replacing an oldest column of pixel values in the hardware scanning window array, thereby introducing a discontinuity along a column direction in an image represented by the pixel values stored in the hardware scanning window array.

9. The apparatus of claim 4, wherein the peripheral circuitry further comprises a hardware computer vision feature computation block operable to perform, in hardware, computer vision computations based on values stored in the hardware scanning window array.

10. The apparatus of claim 9, further comprising configurable combining circuitry coupled to the plurality of storage elements of the hardware scanning window, wherein the configurable combining circuitry is capable of performing combining, in hardware, of some multiple pixel values to generate a first set of combined pixel values according to a first combining pattern, while maintaining the multiple pixel values for subsequent combining of some other multiple pixel values to generate a second set of combined pixel values according to a second combining pattern, and wherein the first set of combined pixel values and the second set of combined pixel values are used by the hardware computer vision feature computation block to compute a multi-block local binary pattern computer vision feature.

11. The apparatus of claim 9, wherein the peripheral circuitry further comprises an integration block capable of computing an integral image based on the sensor readings from the hardware sensor array and storing the computed integral image in the scanning window array, and wherein combined pixel values generated from the integral image stored in the scanning window array are used by the hardware computer vision feature computation block to compute a multi-block local binary pattern computer vision feature.

12. The apparatus of claim 9, wherein the configurable combining circuitry comprises:

a first plurality of controllable connections operable to connect adjacent circuit elements along the first dimension of the hardware scanning window array; and

a second plurality of controllable connections operable to connect adjacent pixels along the second dimension of the hardware scanning window array,

wherein connecting adjacent pixels generates averaged pixel values at each of the adjacent pixels being connected.

13. The apparatus of claim 9, wherein the different ones of the first and second combining patterns allow for selection of the plurality of storage elements at different locations and dimensions within the hardware scanning window array.

14. The apparatus of claim 9, wherein the hardware computer vision feature computation block comprises circuitry operable to perform corner detection.

15. The apparatus of claim 14, further comprising a second hardware scanning window array coupled to the control logic, the control logic operable to provide control signals to control the transfer of pixel values from the hardware scanning window array to the second hardware scanning window array based on detection of a corner by the hardware computer vision feature computation block in an image represented by the pixel values stored in the hardware scanning window array, the peripheral circuitry further comprising a multiplexer circuit positioned between the hardware scanning window array and the second hardware scanning window array, the multiplexer circuit capable of switching column ordering while transferring the pixel values from the hardware scanning window array to the second hardware scanning window array, thereby removing a discontinuity along a column direction in the image.

16. The apparatus of claim 15, further comprising a second hardware computer vision feature computation block operable to perform, in hardware, computer vision computations based on values stored in the second hardware scanning window array.

17. The apparatus of claim 1, wherein at least one pixel of the plurality of pixels comprises a sensor element and in-pixel circuitry.

18. The apparatus of claim 1, wherein the pixel values each based on the one or more sensor readings from the hardware sensor array comprise raw pixel values each based on one or more raw sensor readings.

19. The apparatus of claim 1, wherein no image signal processing circuitry is disposed between the hardware sensor array and the hardware scanning window array.

20. A method for isolating pixel values in hardware for computer vision operations, comprising:

generating, via a plurality of pixels within a hardware sensor array, a sensor reading based on environmental conditions, wherein the plurality of pixels are arranged along at least a first dimension and a second dimension of the hardware sensor array;

storing, within a plurality of storage elements within a hardware scanning window array, a pixel value based on one or more sensor readings from the hardware sensor array, wherein the plurality of storage elements are arranged along at least a first dimension and a second dimension of the hardware scanning window array;

causing, via peripheral circuitry, systematic transfer of pixel values, based on sensor readings from the hardware sensor array, into the hardware scanning window array; and

providing, via control logic coupled to the hardware sensor array, the hardware scanning window array, and the peripheral circuitry, signals to the peripheral circuitry to control the transfer of pixel values into the hardware scanning window array,

wherein the transfer of pixel values causes a first window of pixels values to be stored in the hardware scanning window array at a first time and a second window of pixel values to be stored in the hardware scanning window array at a second time, the first window corresponding to a different position than the second window within the hardware sensor array.

21. The method of claim 20, further comprising combining, in hardware, via configurable combining circuitry coupled to the plurality of pixels, multiple sensor readings from the plurality of pixels to generate the pixel values according to at least one combining pattern, wherein the configurable combining circuitry comprises:

a first plurality of controllable connections operable to connect adjacent pixels along the first dimension of the hardware sensor array; and

a second plurality of controllable connections operable to connect adjacent pixels along the second dimension of the hardware sensor array,

wherein connecting adjacent pixels generates averaged pixel values at each of the adjacent pixels being connected.

22. The method of claim 21, wherein the sensor reading is an analog sensor reading, wherein the pixel value is an analog pixel value, and wherein the configurable combining circuitry is configurable analog combining circuitry.

23. The method of claim 20, wherein the causing, via the peripheral circuitry for systematically transferring pixel values, the different windows of the pixel values to be stored in the hardware scanning window array comprises storing selected rows of pixel values from the hardware sensor array in a line buffer comprising a plurality of storage elements arranged along at least a first dimension and a second dimension of the line buffer, the first dimension of the line buffer being smaller than the first dimension of the hardware sensor array, the second dimension of the line buffer being equal to the second dimension of the hardware sensor array,

wherein the line buffer is capable of storing selected rows of pixel values from the hardware sensor array.

24. The method of claim 23, further comprising storing, via the control logic, a next row of pixel values from the hardware sensor array into the line buffer, by replacing an oldest row of pixel values in the line buffer, thereby introducing a discontinuity along a row direction in an image represented by the pixel values stored in the line buffer.

25. The method of claim 24, further comprising switching row ordering, via a multiplexer circuit positioned between the line buffer and the hardware scanning window array, while transferring pixel values from the line buffer to the hardware scanning window array, thereby removing the discontinuity along a row direction in an image represented by the pixel values stored in the line buffer.

26. The method of claim 24, further comprising switching column ordering, via a multiplexer circuit positioned between the hardware scanning window array and a computer vision feature computation block, while transferring pixel values from the hardware scanning window array to the computer vision feature computation block, thereby removing the discontinuity along a column direction in an image represented by the pixel values stored in the hardware scanning window array.

27. The method of claim 23, further comprising storing, via the control logic, a next column of pixel values from the line buffer into the hardware scanning window array, by replacing an oldest column of pixel values in the hardware scanning window array, thereby introducing a discontinuity along a column direction in an image represented by the pixel values stored in the hardware scanning window array.

28. The method of claim 23, further comprising performing, via computer vision computation circuitry, corner detection based on pixel values stored in the hardware scanning window array.

29. The method of claim 28, wherein, based on the detection of a corner by the hardware computer vision computation circuitry in an image represented by pixel values stored in the hardware scanning window array, causing, via the peripheral circuitry for systematically transferring pixel values, pixel values to be transferred from the hardware scanning window array to a second hardware scanning window array, the peripheral circuitry further comprising a multiplexer circuit positioned between the hardware scanning window array and the second hardware scanning window array, the multiplexer circuit capable of switching column ordering while transferring the pixel values from the hardware scanning window array to the second hardware scanning window array, thereby removing a discontinuity along a column direction in the image.

30. The method of claim 29, further comprising performing, via second hardware computer vision computation circuitry, computer vision computations based on values stored in the second hardware scanning window array.

31. The method of claim 20, further comprising performing combining, via configurable combining circuitry coupled to the plurality of storage elements, of some multiple pixel values to generate a first set of combined pixel values according to a first combining pattern, while maintaining the multiple pixel values for subsequent combining of some other multiple pixel values to generate a second set of combined pixel values according to a second combining pattern.

32. The method of claim 31, further comprising performing, via computer vision computation circuitry coupled to the configurable combining circuitry, computer vision computations.

33. The method of claim 32, wherein performing, via the computer vision computation circuitry coupled to the configurable combining circuitry, computer vision computations comprises performing a first computer vision computation based on the first set of combined pixel values and performing a second computer vision computation based on the second set of combined pixel values.

34. The method of claim 33, wherein the computer vision computation circuitry is capable of performing the first and second computer vision computations as part of a multi-block local binary (LBP) operation.

35. The method of claim 31, wherein the pixel values stored in the hardware scanning window array represent an image having a high resolution and the first combining pattern represents an averaging of neighboring pixel values to generate an image having a first lower resolution lower than the high resolution.

36. The method of claim 35, wherein the second combining pattern represents an averaging of neighboring pixel values to generate an image having a second lower resolution lower than the first lower resolution.

37. The method of claim 20, wherein at least one pixel of the plurality of pixels comprises a sensor element and in-pixel circuitry.

38. The method of claim 20, wherein the pixel values each based on the one or more sensor readings from the hardware sensor array comprise raw pixel values each based on one or more raw sensor readings.

39. The method of claim 20, wherein no image signal processing circuitry is performed between generating the sensor reading within the hardware sensor array and storing the pixel value within the hardware scanning window array.