CMOS image sensors with feature extraction

• US 9,936,132 B2
• Filed: 10/28/2013
• Issued: 04/03/2018
• Est. Priority Date: 10/26/2012
• Status: Active Grant

• Current Assignee: Board of Regents of the University of Michigan (University of Michigan)
• Original Assignee: Board of Regents of the University of Michigan (University of Michigan)
• Inventors: Yoon, Euisik, Choi, Jaehyuk
• Primary Examiner(s): Tran, Sinh
• Assistant Examiner(s): Wu, Zhenzhen
• Application Number:
  US14/064,311
• Publication Number:
  US 20140118592A1
• Time in Patent Office: 1,618 Days
• Field of Search: 348308
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/681 Motion detection
• CPC Class Codes: H04N 25/42 by switching between different modes of operation using different resolutions or aspect ratios, e.g. switching between interlaced and non-interlaced mode
• CPC Class Codes: H04N 25/589 with different integration times, e.g. short and long exposures
• CPC Class Codes: H04N 25/709 Circuitry for control of the power supply
• CPC Class Codes: H04N 25/75 Circuitry for providing, modifying or processing image signals from the pixel array
• CPC Class Codes: H04N 25/771 comprising storage means other than floating diffusion

Abstract

A CMOS imaging sensor with embedded feature extraction capability operatable in different modes by a method that includes the steps of: (a) operating the CMOS imaging sensor in a motion-detecting mode at a first power level using circuitry on the imaging sensor that generates motion data based on received images detected by pixels in the pixel array; (b) switching the imaging sensor from the motion-detecting mode to a feature extraction mode in response to detecting motion; and (c) operating the imaging sensor in the feature extraction mode at a second power level that is higher than the first power level.

Claims

1. A method of operating a complementary metal oxide semiconductor (CMOS) imaging sensor having a pixel array and pixel processing circuitry, comprising the steps of:

(a) operating the CMOS imaging sensor in an in-pixel motion-detecting mode at a first power level using circuitry on the imaging sensor that generates motion data based on received images detected by pixels in the pixel array, wherein when in the in-pixel motion-detecting mode, a pair of pixels in the pixel array are merged and a previous frame signal generated by the merged pixels and a current frame signal generated by the merged pixels are compared by an in-pixel comparator to determine a frame difference therebetween, and further wherein a quantized frame-difference signal is generated by the in-pixel comparator when motion is detected based on the frame difference;

(b) switching the imaging sensor from the motion-detecting mode to a feature extraction mode in response to detecting motion; and

(c) operating the imaging sensor in the feature extraction mode at a second power level that is higher than the first power level, wherein operating in the feature extraction mode comprises;

receiving pixel data from the pixel array;

calculating spatial gradients based on the received pixel data;

determining angular information associated with the spatial gradients using a gradient-to-analog converter comprising mixed-signal processing circuitry by;

converting the spatial gradients into analog values;

generating an analog trigonometric value of the spatial gradients; and

determining angular information associated with the spatial gradients based on the analog trigonometric value; and

determining magnitudes of the spatial gradients associated with the angular information;

wherein the imaging sensor generates extracted feature data from one or more received images when operating in the feature extraction mode.

2. The method of claim 1, further comprising the step of (d) switching from the feature extraction mode to an image capture mode based on analysis of the extracted feature data.

3. The method of claim 2, further comprising the steps of (e) operating the imaging sensor in the image capture mode and storing or transmitting received images at a third power level that is higher than the second power level.

4. The method of claim 2, wherein step (d) is carried out in response to detecting an object in one of the received images.

5. The method of claim 2, wherein the extracted feature data comprises histograms of oriented gradients generated from the one or more received images.

6. The method of claim 1, wherein step (a) further comprises generating 1 bit motion data for at least some of the pixels in the pixel array.

7. The method of claim 6, wherein step (a) further comprises generating the 1 bit motion data for a subset of the pixels used for generating the extracted feature data.

8. An imaging device having a CMOS imaging sensor that operates according to the method of claim 1.

9. The method of claim 1, wherein receiving pixel data comprises receiving a plurality of rows or columns of pixel data from the pixel array;

and wherein

calculating spatial gradients based on the received pixel data comprises

calculating spatial gradients based on the received rows or columns of pixel data.

10. The method of claim 9, wherein determining magnitudes of the spatial gradients associated with the angular information comprises generating a histogram of gradients.

11. The method of claim 1, wherein when in the motion-detecting mode, the method further comprises:

sampling, by a first in-pixel capacitor of one of the pair of pixels, the previous frame signal; and

sampling, by a second in-pixel capacitor of the other of the pair of pixels, the current frame signal, and

further wherein the in-pixel comparator determines the frame difference between the previous and current frame signals stored by the first and second in-pixel capacitors, respectively.

12. The method of claim 11, wherein the previous frame signal is sampled by connecting the two pixels of the pair of pixels to the first in-pixel capacitor, and the current frame signal is sampled by connecting the two pixels of the pair of pixels to the second in-pixel capacitor.

13. A method of operating a complementary metal oxide semiconductor (CMOS) imaging sensor having a pixel array and pixel processing circuitry in an in-pixel motion detecting mode using circuitry on the imaging sensor that generates motion data based on received images detected by pixels in the pixel array, comprising:

sampling, by a first in-pixel capacitor of one of a pair of pixels, an analog previous frame signal, wherein the analog previous frame signal is sampled by connecting the two pixels of the pair of pixels to the first in-pixel capacitor;

sampling, by a second in-pixel capacitor of the other of the pair of pixels, an analog current frame signal, wherein the analog current frame signal is sampled by connecting the two pixels of the pair of pixels to the second in-pixel capacitor;

determining, by one or more in-pixel transistors reconfigured as an in-pixel comparator, a frame difference between the analog previous and current frame signals sampled by the first and second in-pixel capacitors, respectively; and

generating, by the one or more in-pixel transistors reconfigured as the in-pixel comparator, a quantized digital frame-difference signal when motion is detected based on the frame difference.

14. The method of claim 13, wherein the generating step comprises generating a 1-bit frame-difference signal when motion is detected.

15. An imaging device having a CMOS imaging sensor that operates according to the method of claim 13.