Apparatus and method for low-power object-detection in images using computer vision feature computation hardware

• US 9,977,977 B2
• Filed: 10/17/2016
• Issued: 05/22/2018
• Est. Priority Date: 09/30/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Govil, Alok, Gousev, Evgeni Petrovich, Rangan, Venkat, Rasquinha, Nelson
• Primary Examiner(s): ANSARI, TAHMINA N
• Application Number:
  US15/295,960
• Publication Number:
  US 20170032216A1
• Time in Patent Office: 582 Days
• Field of Search: None
• CPC Class Codes: G06V 10/20 Image preprocessing
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 25/443 by reading pixels from selected 2D regions of the array, e.g. for windowing or digital zooming
• CPC Class Codes: H04N 25/46 by combining or binning pixels
• CPC Class Codes: H04N 25/75 Circuitry for providing, modifying or processing image signals from the pixel array

Abstract

An apparatus includes a hardware sensor array including a plurality of pixels arranged along at least a first dimension and a second dimension of the array, each of the pixels capable of generating a sensor reading. A hardware scanning window array includes a plurality of storage elements arranged along at least a first dimension and a second dimension of the hardware scanning window array, each of the storage elements capable of storing a pixel value based on one or more sensor readings. Peripheral circuitry for systematically transfers pixel values, based on sensor readings, into the hardware scanning window array, to cause different windows of pixel values to be stored in the hardware scanning window array at different times. Control logic coupled to the hardware sensor array, the hardware scanning window array, and the peripheral circuitry, provides control signals to the peripheral circuitry to control the transfer of pixel values.

Claims

1. A low-power object detection hardware, comprising:

a hardware scanning window array for storing pixel values or blocks of pixel values based on a two-dimensional sample window of an image, wherein the hardware scanning window array is configured to store pixel values or blocks of pixel values based on different two-dimensional sample windows of the image at different times;

computer vision feature computation hardware configured to compute one or more computer vision features based on one or more comparisons of the pixel values or blocks of pixel values stored in the hardware scanning window array, wherein each of the one or more comparisons comprises a comparison between a value for a pixel or a block of pixels and a value for another pixel or block of pixels within the two-dimensional sample window of the image; and

a classifier configured to receive the one or more computer vision features computed by the computer vision feature computation hardware and to determine whether a reference object is detected within the two-dimensional sample window of the image.

2. The low-power object detection hardware of claim 1, further comprising integration hardware capable of computing an integral image, wherein the data stored in the hardware scanning window is based on the integral image.

3. The low-power object detection hardware of claim 2, wherein the integral image is an integral image of the two-dimensional sample window of the image.

4. The low-power object detection hardware of claim 1, further comprising an image sensor array capable of capturing the image.

5. The low-power object detection hardware of claim 4, wherein no image signal processing circuitry is disposed between the image sensor array and the hardware scanning window array.

6. The low-power object detection hardware of claim 1, wherein the classifier is a hardware classifier.

7. The low-power object detection hardware of claim 1, wherein the classifier is a cascade classifier.

8. The low-power objection detection hardware of claim 1, wherein the computer vision feature computation hardware is implemented as in-pixel circuitry.

9. The low-power objection detection hardware of claim 1, wherein the computer vision feature computation hardware is implemented as peripheral circuitry.

10. The low-power object detection hardware of claim 1, further comprising a dedicated microprocessor configured to receive the computed one or more computer vision features and capable of indicating, responsive to a detection of the reference object, the detection of the reference object to a second microprocessor.

11. The low-power object detection hardware of claim 10, wherein the dedicated microprocessor is capable of performing one or more high-level computer vision operations based on the received computed one or more computer vision features.

12. The low-power object detection hardware of claim 11, wherein the classifier is implemented in the dedicated microprocessor and wherein the one or more high-level computer vision operations based on the received computed one or more computer vision features include determining whether the reference object is detected within the two-dimensional sample window of the image using the classifier.

13. The low-power object detection hardware of claim 1, wherein the classifier is implemented in hardware dedicated to perform cascade classifier functions.

14. A method for detecting an object via low-power object detection hardware, comprising:

storing, via a hardware scanning window array, pixel values or blocks of pixel values based on a two-dimensional sample window of an image, wherein the hardware scanning window array is configured to store pixel values or blocks of pixel values based on different two-dimensional sample windows of the image at different times;

computing, via computer vision feature computation hardware, one or more computer vision features based on one or more comparisons of the pixel values or blocks of pixel values stored in the hardware scanning window array, wherein each of the one or more comparisons comprises a comparison between a value for a pixel or a block of pixels and a value for another pixel or block of pixels within the two-dimensional sample window of the image;

receiving, via a classifier, the one or more computer vision features computed by the computer vision feature computation hardware; and

determining, via the classifier, whether a reference object is detected within the two-dimensional sample window of the image.

15. The method of claim 14, further comprising computing, via integration hardware, an integral image, wherein the data stored in the hardware scanning window is based on the integral image.

16. The method of claim 15, wherein the integral image is an integral image of the two-dimensional sample window of the image.

17. The method of claim 14, further comprising capturing, via an image sensor array, the image.

18. The method of claim 17, wherein no image signal processing circuitry is disposed between the image sensor array and the hardware scanning window array.

19. The method of claim 14, wherein the classifier is a hardware classifier.

20. The method of claim 14, wherein the classifier is a cascade classifier.

21. The method of claim 14, wherein the computer vision feature computation hardware is implemented as in-pixel circuitry.

22. The method of claim 14, wherein the computer vision feature computation hardware is implemented as peripheral circuitry.

23. The method of claim 14, further comprising receiving, via a dedicated microprocessor, the computed one or more computer vision features;

and

indicating, via the dedicated microprocessor, responsive to a detection of the reference object, the detection of the reference object to a second microprocessor.

24. The method of claim 23, wherein the dedicated microprocessor is capable of computing one or more high-level computer vision operations based on the received computed one or more computer vision features.

25. The method of claim 24, wherein the classifier is implemented in the dedicated microprocessor and wherein the one or more high-level computer vision operations based on the received computed one or more computer vision features include determining whether the reference object is detected within the two-dimensional sample window of the image using the classifier.

26. The method of claim 14, wherein the classifier is implemented in hardware dedicated to perform cascade classifier functions.

27. One or more non-transitory computer-readable media storing computer-executable instructions for detecting an object via low-power object detection hardware that, when executed, cause one or more computing devices to:

store, via a hardware scanning window array, pixel values or blocks of pixel values based on a two-dimensional sample window of an image, wherein the hardware scanning window array is configured to store pixel values or blocks of pixel values based on different two-dimensional sample windows of the image at different times;

compute, via computer vision feature computation hardware, one or more computer vision features based on one or more comparisons of the pixel values or blocks of pixel values stored in the hardware scanning window array, wherein each of the one or more comparisons comprises a comparison between a value for a pixel or a block of pixels and a value for another pixel or block of pixels within the two-dimensional sample window of the image;

receive, via a classifier, the one or more computer vision features computed by the computer vision feature computation hardware; and

determine, via the classifier, whether a reference object is detected within the two-dimensional sample window of the image.

28. The non-transitory computer-readable media of claim 27, wherein the computer-executable instructions, when executed, further cause the one or more computing devices to compute, via integration hardware, an integral image, wherein the data stored in the hardware scanning window is based on the integral image.

29. The non-transitory computer-readable media of claim 28, wherein the integral image is an integral image of the two-dimensional sample window of the image.

30. The non-transitory computer-readable media of claim 27, wherein the computer-executable instructions, when executed, further cause the one or more computing devices to capture, via an image sensor array, the image.

31. The non-transitory computer-readable media of claim 30, wherein no image signal processing circuitry is disposed between the image sensor array and the hardware scanning window array.

32. The non-transitory computer-readable media of claim 27, wherein the classifier is a hardware classifier.

33. The non-transitory computer-readable media of claim 27, wherein the classifier is a cascade classifier.

34. The non-transitory computer-readable media of claim 27, wherein the computer vision feature computation hardware is implemented as in-pixel circuitry.

35. The non-transitory computer-readable media of claim 27, wherein the computer vision feature computation hardware is implemented as peripheral circuitry.

36. The non-transitory computer-readable media of claim 27, wherein the computer-executable instructions, when executed, further cause the one or more computing devices to receive, via a dedicated microprocessor, the computed one or more computer vision features;

and

indicate, via the dedicated microprocessor, responsive to a detection of the reference object, the detection of the reference object to a second microprocessor.

37. The non-transitory computer-readable media of claim 36, wherein the dedicated microprocessor is capable of computing one or more high-level computer vision operations based on the received computed one or more computer vision features.

38. The non-transitory computer-readable media of claim 37, wherein the classifier is implemented in the dedicated microprocessor and wherein the one or more high-level computer vision operations based on the received computed one or more computer vision features include determining whether the reference object is detected within the two-dimensional sample window of the image using the classifier.

39. The non-transitory computer-readable media of claim 27, wherein the classifier is implemented in hardware dedicated to perform cascade classifier functions.

40. An apparatus for detecting an object via low-power object detection hardware, comprising:

hardware means for storing, via a hardware scanning window array, pixel values or blocks of pixel values based on a two-dimensional sample window of an image, wherein the hardware scanning window array is configured to store pixel values or blocks of pixel values based on different two-dimensional sample windows of the image at different times;

hardware means for computing, via computer vision feature computation hardware, one or more computer vision features based on one or more comparisons of the pixel values or blocks of pixel values stored in the hardware scanning window array, wherein each of the one or more comparisons comprises a comparison between a value for a pixel or a block of pixels and a value for another pixel or block of pixels within the two-dimensional sample window of the image;

hardware means for receiving, via a classifier, the one or more computer vision features computed by the computer vision feature computation hardware; and

hardware means for determining, via the classifier, whether a reference object is detected within the two-dimensional sample window of the image.

41. The apparatus of claim 40, further comprising hardware means for computing, via integration hardware, an integral image, wherein the data stored in the hardware scanning window is based on the integral image.

42. The apparatus of claim 41, wherein the integral image is an integral image of the two-dimensional sample window of the image.

43. The apparatus of claim 40, further comprising hardware means for capturing, via an image sensor array, the image.

44. The apparatus of claim 43, wherein no image signal processing circuitry is disposed between the image sensor array and the hardware scanning window array.

45. The apparatus of claim 40, wherein the classifier is a hardware classifier.

46. The apparatus of claim 40, wherein the classifier is a cascade classifier.

47. The apparatus of claim 40, wherein the computer vision feature computation hardware is implemented as in-pixel circuitry.

48. The apparatus of claim 40, wherein the computer vision feature computation hardware is implemented as peripheral circuitry.

49. The apparatus of claim 40, further comprising hardware means for receiving, via a dedicated microprocessor, the computed one or more computer vision features;

and

indicating, via the dedicated microprocessor, responsive to a detection of the reference object, the detection of the reference object to a second microprocessor.

50. The apparatus of claim 49, wherein the dedicated microprocessor is capable of computing one or more high-level computer vision operations based on the received computed one or more computer vision features.

51. The apparatus of claim 50, wherein the classifier is implemented in the dedicated microprocessor and wherein the one or more high-level computer vision operations based on the received computed one or more computer vision features include determining whether the reference object is detected within the two-dimensional sample window of the image using the classifier.

52. The apparatus of claim 40, wherein the classifier is implemented in hardware dedicated to perform cascade classifier functions.