CMOS IMAGE SENSORS WITH FEATURE EXTRACTION

• US 20140118592A1
• Filed: 10/28/2013
• Published: 05/01/2014
• Est. Priority Date: 10/26/2012
• Status: Active Grant

• Current Assignee: Board of Regents of the University of Michigan (University of Michigan)
• Original Assignee: Board of Regents of the University of Michigan (University of Michigan)
• Inventors: Yoon, Euisik, Choi, Jaehyuk
• Granted Patent: US 9,936,132 B2
• Time in Patent Office: Days
• US Class Current: 348/308
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/681 Motion detection
• CPC Class Codes: H04N 25/00 Circuitry of solid-state image sensors [SSIS]; Control thereof
• CPC Class Codes: H04N 25/42 by switching between different modes of operation using different resolutions or aspect ratios, e.g. switching between interlaced and non-interlaced mode
• CPC Class Codes: H04N 25/47 Image sensors with pixel address output; Event-driven image sensors; Selection of pixels to be read out based on image data
• CPC Class Codes: H04N 25/589 with different integration times, e.g. short and long exposures
• CPC Class Codes: H04N 25/707 Pixels for event detection
• CPC Class Codes: H04N 25/708 Pixels for edge detection
• CPC Class Codes: H04N 25/709 Circuitry for control of the power supply
• CPC Class Codes: H04N 25/75 Circuitry for providing, modifying or processing image signals from the pixel array
• CPC Class Codes: H04N 25/771 comprising storage means other than floating diffusion
• CPC Class Codes: H04N 25/78 Readout circuits for addressed sensors, e.g. output amplifiers or A/D converters

Abstract

A CMOS imaging sensor with ernbedded feature extraction capability operatable in different modes by a method that includes the steps of: (a) operating the CMOS imaging sensor in a motion-detecting mode at a first power level using circuitry on the imaging sensor that generates motion data based on received images detected by pixels in the pixel array; (b) switching the imaging sensor from the motion-detecting mode to a feature extraction mode in response to detecting motion; and (c) operating the imaging sensor in the feature extraction mode at a second power level that is higher than the first power level.

Claims

1. A method of operating a complementary metal oxide semiconductor (CMOS) imaging sensor having a pixel array and pixel processing circuitry, comprising the steps of:

(a) operating the CMOS imaging sensor in a motion-detecting mode at a first power level using circuitry on the imaging sensor that generates motion data based on received images detected by pixels in the pixel array;

(b) switching the imaging sensor from the motion-detecting mode to a feature extraction mode in response to detecting motion; and

(c) operating the imaging sensor in the feature extraction mode at a second power level that is higher than the first power level;

wherein the imaging sensor generates extracted feature data from one or more received images when operating in the feature extraction mode.

2. The method of claim 1, further comprising the step of (d) switching from the feature extraction mode to an image capture mode based on analysis of the extracted feature data.

3. The method of claim 2, further comprising the steps of (e) operating the imaging sensor in the image capture mode and storing or transmitting received images at a third power level that is higher than the second power level.

4. The method of claim 2, wherein step (d) is carried out in response to detecting an object in one of the received images.

5. The method of claim 2, wherein the extracted feature data comprises histograms of oriented gradients generated from the one or more received images.

6. The method of claim 1, wherein step (a) further comprises generating 1 bit motion data for at least some of the pixels in the pixel array.

7. The method of claim 6, wherein step (a) further comprises generating the 1 bit motion data for a subset of the pixels used for generating the extracted feature data.

8. An imaging device having a CMOS imaging sensor that operates according to the method of claim 1.

9. A method of operating a complementary metal oxide semiconductor (CMOS) imaging sensor having a pixel array and pixel processing circuitry, comprising carryout out the following steps using the pixel processing circuitry:

(a) receiving a plurality of rows or columns of pixel data from the pixel array;

(b) calculating spatial gradients based on the received rows or columns of pixel data;

(c) determining angular information associated with the spatial gradients; and

(d) determining magnitudes of the spatial gradients associated with the angular information.

10. The method of claim 9, wherein step (c) is carried out using a gradient-to-angle converter using mixed-mode processing circuitry.

11. The method of claim 9, wherein step (d) comprises generating a histogram of gradients.

12. A complementary metal oxide semiconductor (CMOS) imaging sensor having a pixel array, comprising:

a first pixel having a first in-pixel capacitor;

a second pixel located adjacent the first pixel having a second in-pixel capacitor; and

a common electrical communication line electrically linking the first pixel and the second pixel, wherein the first pixel and the second pixel each include a source follower readout transistor connected to the common electrical communication line and connected to the respective first in-pixel capacitor or second in-pixel capacitor to provide a differential readout.

13. The CMOS imaging sensor of claim 12, wherein the first in-pixel capacitor is disposed on top of a first pixel circuit located in the first pixel.

14. The CMOS imaging sensor of claim 12, wherein the second pixel capacitor disposed on top of a second pixel circuit located in the second pixel.

15. The CMOS imaging sensor of claim 12, wherein the first in-pixel capacitor, the second in-pixel capacitor, or both further comprise a metal-insulator-metal (MIM) capacitor.

16. The CMOS imaging sensor of claim 12, wherein the first in-pixel capacitor, the second in-pixel capacitor, or both act as frame buffer memory for both sensing motion and low-power imaging.

17. The CMOS imaging sensor of claim 12, further comprising a merging switch that connects the first in-pixel capacitor and the second in-pixel capacitor.

18. The CMOS imaging sensor of claim 17, wherein the first in-pixel capacitor and the second in-pixel capacitor are electrically connected via the merging switch to operate in a motion-sensing mode.

19. The CMOS imaging sensor of claim 12, wherein the pixel array is selectively switched between a motion-sensing mode and an image-generating mode.