Video imaging system including a digital image sensor and a digital signal processor

• US 7,483,058 B1
• Filed: 08/04/2003
• Issued: 01/27/2009
• Est. Priority Date: 08/04/2003
• Status: Active Grant

• Current Assignee: Pixim, Inc. (Sony Group Corp.)
• Original Assignee: Pixim, Inc. (Sony Group Corp.)
• Inventors: Yang, David Xiao Dong, Ewedemi, Odutola Oluseye, Frank, Michael, Bidermann, William R., Reyneri, Justin, Motta, Ricardo J.
• Primary Examiner(s): Ometz; David L
• Assistant Examiner(s): Giles; Nicholas G
• Application Number:
  US10/634,302
• Time in Patent Office: 2,003 Days
• Field of Search: 348/311.4, 348/222.1, 348/211.14, 348/207.1, 348/211.4
• US Class Current: 348/222.1
• CPC Class Codes: H04N 23/651 for reducing power consumption by affecting camera operations, e.g. sleep mode, hibernation mode or power off of selective parts of the camera
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/772 comprising A/D, V/T, V/F, I/T or I/F converters

Abstract

A video imaging system includes a digital image sensor for performing image capture operations and a digital image processor for performing image processing operations. The digital image sensor includes a sensor array outputting digital pixel data, an image buffer for storing the pixel data, a first processor and a first interface circuit for transferring the pixel data onto a pixel bus. The digital image processor includes a second interface circuit coupled to receive the pixel data from the pixel bus, a frame buffer coupled to store the pixel data, an image processing pipeline for processing the pixel data stored in the frame buffer into video data, and a second processor. The digital image sensor and the digital image processor transfer control information over a control interface bus and the digital image sensor performs the image capture operations independent of the image processing operations performed by the digital image processor.

Claims

1. A video imaging system, comprising:

a digital image sensor for performing image capture operations, comprising;

a sensor array comprising a two-dimensional array of digital pixels, each digital pixel outputting digital signals as pixel data representing an image of a scene;

an image buffer, in communication with said sensor array, for storing said pixel data;

a first processor, in communication with said image buffer and said sensor array, for controlling image capture and pixel data processing operations; and

a first interface circuit, in communication with said image buffer, for transferring said pixel data onto a pixel bus; and

a digital image processor for performing image processing operations, comprising;

a second interface circuit coupled to receive said pixel data from said pixel bus;

a frame buffer, in communication with said second interface circuit, coupled to store said pixel data;

an image processing pipeline for processing said pixel data stored in said frame buffer into video data corresponding to a video format selected from a group of video formats; and

a second processor, in communication with said frame buffer and said image processing pipeline, for directing said image processing pipeline to process said pixel data stored in said frame buffer;

wherein said digital image sensor and said digital image processor transfer control information over a control interface bus coupled between said first interface circuit and said second interface circuit and separate from said pixel bus and said digital image sensor performs said image capture operations independent of said image processing operations performed by said digital image processor.

2. The video imaging system of claim 1, wherein said image processing pipeline comprises an interpolator module, in communication with said frame buffer, for interpolating said pixel data to generate video data in at least three color planes and having a vertical resolution corresponding to said selected video format.

3. The video imaging system of claim 2, wherein said image processing pipeline further comprises an image processing circuit coupled to receive said video data from said interpolator module and for performing image enhancement functions on said video data.

4. The video imaging system of claim 3, wherein said image processing circuit performs tone correction operations on said video data.

5. The video imaging system of claim 2, wherein said interpolator module performs vertical interpolation and demosaic operations on said pixel data.

6. The video imaging system of claim 2, wherein said digital image processor further comprises a television encoder, in communication with said interpolator module, for encoding said video data in said selected video format and for providing control signals to said interpolator module for directing said interpolator module to process video data.

7. The video imaging system of claim 1, wherein said group of video formats comprises NTSC, PAL and digital TV video formats.

8. The video imaging system of claim 1, wherein each digital pixel in said sensor array comprises a photodetector producing an analog signal indicative of the amount of light impinging on said sensor array, and said two dimensional array of digital pixels further comprises a plurality of analog-to-digital conversion (ADC) circuits, each of said ADC circuits being connected to one or more photodetectors of said digital pixels for converting said analog signal to said digital pixel data.

9. The video imaging system of claim 1, wherein said digital image sensor further comprises a single-instruction-multiple-data engine, in communication with said image buffer and said first processor, for providing image processing operations on said pixel data stored in said image buffer.

10. The video imaging system of claim 9, wherein image processing operations comprise CDS subtract operations, Gray code to linear conversion and companding operations.

11. The video imaging system of claim 9, wherein said single-instruction-multiple-data engine comprises a lookup table coupled to convert said pixel data into corresponding output data.

12. The video imaging system of claim 1, wherein said digital image processor further comprises a lookup table coupled to receive pixel data stored in said frame buffer and output codewords corresponding to said pixel data.

13. The video imaging system of claim 1, wherein said pixel bus comprises a low voltage differential signalling data bus.

14. The video imaging system of claim 1, wherein said control interface bus comprises a multi-bit serial bi-directional data bus.

15. The video imaging system of claim 1, wherein said digital image sensor and said digital image processor are formed as separate integrated circuits, said digital image sensor and said digital image processor communicating over said pixel bus and said control interface bus.

16. The video imaging system of claim 1, wherein said digital image sensor and said digital image processor are formed as a single integrated circuit.

17. The video imaging system of claim 1, wherein said digital image sensor and said digital image processor are formed as separate integrated circuits on a common substrate.

18. The video imaging system of claim 1, wherein said digital image sensor and said digital image processor communicate using only digital data over said pixel bus and said control interface bus.

19. The video imaging system of claim 1, wherein said digital image processor transmits a first control signal over said control interface bus to cause said digital image sensor to initiate an image capture operation.

20. The video imaging system of claim 19, wherein said digital image sensor transmits a second control signal over said control interface bus to indicate to said digital image processor the completion of an image capture operation.

21. The video imaging system of claim 20, wherein, upon receipt of said second control signal, said digital image processor transmits a third control signal over said control interface bus to initiate transfer of pixel data stored in said image buffer of said digital image sensor to said frame buffer of said digital image processor.

22. The video imaging system of claim 1, wherein said second interface circuit comprises a noise reduction circuit performing signal processing for noise reduction.

23. The video imaging system of claim 22, wherein said noise reduction circuit performs an infinite impulse filtering operation using a fixed blending coefficient.

24. The video imaging system of claim 23, wherein said noise reduction circuit performs said infinite impulse filtering operation by averaging multiple number of frames of pixel data provided by said digital image sensor using said fixed blending coefficient.

25. The video imaging system of claim 24, wherein said noise reduction circuit calculates new pixel data for each frame of pixel data received using the equation:

new data=α

*input data+(1−

α

)*old data, where “

new data”

represents the final pixel data, “

input data”

represents the pixel data of the current frame to be averaged, “

old data”

represents the pixel data previously averaged, and “

α

”

represents said fixed blending coefficient.

26. The video imaging system of claim 22, wherein said noise reduction circuit performs a multisample averaging operation using a data and exposure time dependent blending coefficient.

27. The video imaging system of claim 26, wherein said noise reduction circuit performs said multisample averaging operation by averaging multiple reads of the same frame of pixel data provided by said digital image sensor and applying said data and exposure time dependent blending coefficient.

28. The video imaging system of claim 27, wherein said noise reduction circuit calculates new pixel data for each frame of pixel data received using the equation:

new data=α

*input data+(1−

α

)*old data, where “

new data”

represents the final pixel data, “

input data”

represents the pixel data of the current frame to be averaged, “

old data”

represents the pixel data previously averaged, and “

α

”

represents said data and exposure time dependent blending coefficient.

29. The video imaging system of claim 1, wherein said frame buffer includes a plurality of row lines, each row line being organized into a plurality of access units, each access unit including pixel data for a group of pixels and a link field indicating the location of the next access unit including a defective pixel.

30. The video imaging system of claim 29, wherein said link field for a first access unit includes a reserved codeword indicating an end of chain condition when access units following the first access unit do not contain defective pixels.

31. The video imaging system of claim 29, wherein said link field comprises k bits, and when a first access unit including a defective pixel is separated by more than 2^k access units from a second access unit including a defective pixel, one or more intermediate access units, each intermediate access unit being 2^k access units from the first access unit or 2^k access units apart from each other, are used to indicate the location of said second access unit.