Apparatus and method for low-power object-detection in images using hardware scanning window

• US 9,471,840 B2
• Filed: 12/30/2015
• Issued: 10/18/2016
• Est. Priority Date: 09/30/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Govil, Alok, Gousev, Evgeni Petrovich, Rangan, Venkat, Rasquinha, Nelson
• Primary Examiner(s): ANSARI, TAHMINA N
• Application Number:
  US14/985,144
• Publication Number:
  US 20160110603A1
• Time in Patent Office: 293 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06V 10/20 Image preprocessing
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 25/443 by reading pixels from selected 2D regions of the array, e.g. for windowing or digital zooming
• CPC Class Codes: H04N 25/46 by combining or binning pixels
• CPC Class Codes: H04N 25/75 Circuitry for providing, modifying or processing image signals from the pixel array

Abstract

An apparatus includes a hardware sensor array including a plurality of pixels arranged along at least a first dimension and a second dimension of the array, each of the pixels capable of generating a sensor reading. A hardware scanning window array includes a plurality of storage elements arranged along at least a first dimension and a second dimension of the hardware scanning window array, each of the storage elements capable of storing a pixel value based on one or more sensor readings. Peripheral circuitry for systematically transfers pixel values, based on sensor readings, into the hardware scanning window array, to cause different windows of pixel values to be stored in the hardware scanning window array at different times. Control logic coupled to the hardware sensor array, the hardware scanning window array, and the peripheral circuitry, provides control signals to the peripheral circuitry to control the transfer of pixel values.

Claims

1. A low-power vision sensor, comprising:

an image sensor array capable of capturing an image, the image sensor array having a plurality of rows and a plurality of columns;

one or more line buffers for transferring image data from one or more rows of the plurality of rows of the image sensor array;

a hardware scanning window array for storing data based on a sample window of the image, the data to be used in a computer vision computation, the hardware scanning window array having a number of columns fewer than a number of columns of the one or more line buffers, wherein the hardware scanning window array is configured to store data based on different sample windows of the image at different times; and

computer vision feature computation hardware configured to compute a computer vision feature based on the data stored in the hardware scanning window array.

8. The low-power vision sensor of claim 1, wherein the computer vision feature computation hardware is configured to compute a local binary patterns (LBP) feature.

9. The low-power vision sensor of claim 8, wherein the LBP feature is a multi-block LBP feature.

10. The low-power vision sensor of claim 1, further comprising a hardware classifier configured to detect the presence of a reference object within the sample window stored in the hardware scanning window array based on computer vision features computed by the computer vision computation hardware.

11. The low-power vision sensor of claim 1, wherein no image signal processing circuitry is disposed between the image sensor array and the hardware scanning window array.

2. A method for detecting an object in a low-power vision sensor, comprising:

capturing, via an image sensor array, an image, the image sensor array having a plurality of rows and a plurality of columns;

transferring, via one or more line buffers, image data from one or more rows of the plurality of rows of the image sensor array;

storing, via a hardware scanning window array, data based on a sample window of the image, the data to be used in a computer vision computation, the hardware scanning window array having a number of columns fewer than a number of columns of the one or more line buffers, wherein the hardware scanning window array is configured to store data based on different sample windows of the image at different times; and

computing, via computer vision feature computation hardware, a computer vision feature based on the data stored in the hardware scanning window array.

3. One or more non-transitory computer-readable media storing computer-executable instructions for detecting an object in a low-power vision sensor that, when executed, cause one or more computing devices to:

capture, via an image sensor array, an image, the image sensor array having a plurality of rows and a plurality of columns;

transfer, via one or more line buffers, image data from one or more rows of the plurality of rows of the image sensor array;

store, via a hardware scanning window array, data based on a sample window of the image, the data to be used in a computer vision computation, the hardware scanning window array having a number of columns fewer than a number of columns of the one or more line buffers, wherein the hardware scanning window array is configured to store data based on different sample windows of the image at different times; and

compute, via computer vision feature computation hardware, a computer vision feature based on the data stored in the hardware scanning window array.

4. An apparatus for detecting an object in a low-power vision sensor, comprising:

hardware means for capturing, via an image sensor array, an image, the image sensor array having a plurality of rows and a plurality of columns;

hardware means for transferring, via one or more line buffers, image data from one or more rows of the plurality of rows of the image sensor array;

hardware means for storing, via a hardware scanning window array, data based on a sample window of the image, the data to be used in a computer vision computation, the hardware scanning window array having a number of columns fewer than a number of columns of the one or more line buffers, wherein the hardware scanning window array is configured to store data based on different sample windows of the image at different times; and

hardware means for computing, via computer vision feature computation hardware, a computer vision feature based on the data stored in the hardware scanning window array.

5. The low-power vision sensor of claim 4, wherein the integration hardware comprises two-dimension (2-D) integration hardware.

6. The low-power vision sensor of claim 4, wherein the integration hardware is coupled to the frame buffer.

7. The low-power vision sensor of claim 4, further comprising an analog-to-digital converter (ADC) coupled to the one or more line buffers and the integration hardware.

12. A method for detecting an object in a low-power vision sensor, comprising:

capturing, via an image sensor array, an image, the image sensor array having a plurality of rows and a plurality of columns;

transferring, via one or more line buffers, image data from one or more rows of the plurality of rows of the image sensor array;

storing, via a hardware scanning window array, data based on a sample window of the image, the data to be used in a computer vision computation, the hardware scanning window array having a number of columns fewer than a number of columns of the one or more line buffers, wherein the hardware scanning window array is configured to store data based on different sample windows of the image at different times; and

computing, via computer vision feature computation hardware, a computer vision feature based on the data stored in the hardware scanning window array.

13. The method of claim 12, further comprising transferring, via a frame buffer, data to the hardware scanning window array.

14. The method of claim 13, wherein the frame buffer is at least one of an image frame buffer or an integral image frame buffer.

15. The method of claim 12, further comprising computing, via integration hardware, an integral image based on the image data transferred by the one or more line buffers.

16. The method of claim 12, further comprising computing, via the computer vision feature computation hardware, a local binary patterns (LBP) feature.

17. The method of claim 16, wherein computing the LBP feature comprises computing a multi-block local binary patterns LBP feature.

18. The method of claim 12, further comprising detecting, via a hardware classifier, a reference object within the sample window stored in the hardware scanning window array based on computer vision features computed by the computer vision computation hardware.

19. One or more non-transitory computer-readable media storing computer-executable instructions for detecting an object in a low-power vision sensor that, when executed, cause one or more computing devices to:

capture, via an image sensor array, an image, the image sensor array having a plurality of rows and a plurality of columns;

transfer, via one or more line buffers, image data from one or more rows of the plurality of rows of the image sensor array;

store, via a hardware scanning window array, data based on a sample window of the image, the data to be used in a computer vision computation, the hardware scanning window array having a number of columns fewer than a number of columns of the one or more line buffers, wherein the hardware scanning window array is configured to store data based on different sample windows of the image at different times; and

compute, via computer vision feature computation hardware, a computer vision feature based on the data stored in the hardware scanning window array.

20. The non-transitory computer-readable media of claim 19, wherein the computer-executable instructions, when executed, further cause the one or more computing devices to transfer, via frame buffer, data to the hardware scanning window array.

21. The non-transitory computer-readable media of claim 20, wherein the frame buffer is at least one of an image frame buffer or an integral image frame buffer.

22. The non-transitory computer-readable media of claim 19, wherein the computer-executable instructions, when executed, further cause the one or more computing devices to compute, via integration hardware, an integral image based on the image data transferred by the one or more line buffers.

23. The non-transitory computer-readable media of claim 22, wherein the integration hardware is coupled to the frame buffer.

24. The non-transitory computer-readable media of claim 19, wherein the computer-executable instructions, when executed, further cause the one or more computing devices to compute, via the computer vision feature computation hardware, a local binary patterns (LBP) feature, wherein computing the LBP feature comprises computing a multi-block LBP feature.

25. The non-transitory computer-readable media of claim 19, wherein the computer-executable instructions, when executed, further cause the one or more computing devices to detect, via a hardware classifier, a reference object within the sample window stored in the hardware scanning window array based on computer vision features computed by the computer vision computation hardware.

26. An apparatus for detecting an object in a low-power vision sensor, comprising:

hardware means for capturing, via an image sensor array, an image, the image sensor array having a plurality of rows and a plurality of columns;

hardware means for transferring, via one or more line buffers, image data from one or more rows of the plurality of rows of the image sensor array;

hardware means for storing, via a hardware scanning window array, data based on a sample window of the image, the data to be used in a computer vision computation, the hardware scanning window array having a number of columns fewer than a number of columns of the one or more line buffers, wherein the hardware scanning window array is configured to store data based on different sample windows of the image at different times; and

hardware means for computing, via computer vision feature computation hardware, a computer vision feature based on the data stored in the hardware scanning window array.

27. The apparatus of claim 26, further comprising hardware means for transferring, via a frame buffer, data to the hardware scanning window array,

wherein the frame buffer is at least one of an image frame buffer or an integral image frame buffer.

28. The apparatus of claim 26, further comprising hardware means for computing, via integration hardware coupled to the frame buffer, an integral image based on the image data transferred by the one or more line buffers.

29. The apparatus of claim 26, further comprising hardware means for computing, via the computer vision feature computation hardware, a local binary patterns (LBP) feature, wherein the hardware means for computing the LBP feature comprises computing multi-block LBP feature.

30. The apparatus of claim 26, further comprising hardware means for detecting, via a hardware classifier, a reference object within the sample window stored in the hardware scanning window array based on computer vision features computed by the computer vision computation hardware.