METHODS AND APPARATUS FOR DEMOSAICING IMAGES WITH HIGHLY CORRELATED COLOR CHANNELS

• US 20120206582A1
• Filed: 02/14/2011
• Published: 08/16/2012
• Est. Priority Date: 02/14/2011
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Scott, David D., Zhao, Wenyi, DiCarlo, Jeffrey
• Granted Patent: US 8,698,885 B2
• Time in Patent Office: Days
• US Class Current: 348/71
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: A61B 1/044 for absorption imaging
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 2209/046 Colour interpolation to calculate the missing colour values
• CPC Class Codes: H04N 23/12 with one sensor only
• CPC Class Codes: H04N 23/13 with multiple sensors
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements

Abstract

In one embodiment of the invention, an apparatus is disclosed including an image sensor, a color filter array, and an image processor. The image sensor has an active area with a matrix of camera pixels. The color filter array is in optical alignment over the matrix of the camera pixels. The color filter array assigns alternating single colors to each camera pixel. The image processor receives the camera pixels and includes a correlation detector to detect spatial correlation of color information between pairs of colors in the pixel data captured by the camera pixels. The correlation detector further controls demosaicing of the camera pixels into full color pixels with improved resolution. The apparatus may further include demosaicing logic to demosaic the camera pixels into the full color pixels with improved resolution in response to the spatial correlation of the color information between pairs of colors.

Claims

1. A method of image processing comprising:

capturing a first captured image with at least a first image sensor, the first captured image including a first frame of initial pixel data captured by a first plurality of camera pixels;

assigning a color mosaic of color information to the initial pixel data to generate alternating single-color pixel data of known color information for each of the first plurality of camera pixels to form at least two color channels;

with a processor, estimating missing color information for each of the first plurality of camera pixels in the first frame;

for each array of a plurality of arrays of camera pixels in the first frame, generating correlation information to determine if a pair of the at least two color channels are substantially correlated in response to the known color information and the estimated color information; and

if a pair of color channels are substantially correlated within an array then

demosaicing each camera pixel in the array in response to its single-color pixel data and the correlation information to generate full color pixel data for each camera pixel within the array and form a portion of a first full color image of the first captured image.

2. The method of claim 1, wherein

the estimates for the missing color information form a correlation map for the frame of camera pixels, and

missing color information is estimated by averaging known color information of respective neighboring camera pixels associated with the same respective color.

3. The method of claim 1, further comprising:

selecting the plurality of arrays in the first frame in response to the correlation map; and

outputting the full color image of the first image.

4. The method of claim 1, wherein

a color filter array assigns the mosaic of color information to the initial grey-scale pixel data of each of the plurality of camera pixels.

5. The method of claim 1, wherein

each and every neighboring camera pixel adjacent to a selected camera pixel is further used to demosaic the selected camera pixel in the array.

6. The method of claim 5, wherein

the neighboring camera pixels include northwest, southwest, northeast, and southeast adjacent camera pixels.

7. The method of claim 1, wherein

the single-color pixel data of a given camera pixel is used alone with the correlation information to demosaic the given camera pixel into a full color pixel.

8. The method of claim 1, wherein

a pair of color channels is substantially correlated within the array if a change in level of color information for the pair of color channels is substantially similar from camera pixel to camera pixel in the array.

9. The method of claim 8, wherein

the pair of color channels are substantially correlated within the array, and

the demosaicing of each camera pixel includes

interpolating a first missing color of the highly correlated color channel for a given camera pixel by scaling the single-color pixel data of the given camera pixel in response to the correlation.

10. The method of claim 9, wherein

the demosaicing of each camera pixel further includes

interpolating the neighboring pixel data to determine a second missing color of the given camera pixel.

11. The method of claim 9, wherein

the demosaicing of each camera pixel further includes

estimating homogeneity of a given camera pixel with each neighboring camera pixel, and

further interpolating the given camera pixel with neighboring pixel data in response to the homogeneity to determine the first missing color of the given camera pixel.

12. The method of claim 11, wherein

each neighboring camera pixel includes adjacent on-axis camera pixels along vertical and horizontal axes and adjacent off-axis camera pixels along left and right diagonal axes.

13. The method of claim 8, wherein

in the array of camera pixels, no pair of color channels is substantially correlated such that the method further comprises

demosaicing each camera pixel in the array with on-axis adjacent camera pixels to generate full color pixel data for the given camera pixel of the plurality with lower resolution.

14. The method of claim 1, wherein

the first captured image is a medical image captured from a surgical site within a patient.

15. The method of claim 14, wherein

the surgical site includes tissue of the patient;

and

the first captured image includes a plurality of pixel data representing the tissue of the patient.

16. (canceled)

17. A method for a minimally invasive surgical system, the method comprising:

capturing an image from a surgical site within a patient, the image including a frame of pixel data captured by a plurality of camera pixels;

alternately assigning single color information to the pixel data of each of the plurality of camera pixels to form a color mosaic in the frame of initial pixel data;

in an array of a plurality of arrays of camera pixels in the frame,

estimating spatial correlation in intensity information for each color channel relative to the other color channels over all the camera pixels of the plurality in the array,

first demosaicing each camera pixel in the array with neighboring camera pixels the estimated spatial correlation to generate high-resolution full color pixel data for each camera pixel of the plurality in the array,

second demosaicing each camera pixel in the array with neighboring camera pixels to generate low-resolution full color pixel data for each camera pixel of the plurality in the array,

and

selectively combining the first demosaicing and the second demosaicing to generate full color pixel data for each camera pixel of the plurality in the array in response to the estimated spatial correlation.

18. The method of claim 17, wherein

the first demosaicing and the second demosaicing are selectively combined by weighting the low-resolution full color pixel data and the high-resolution full color pixel data to generate the full color pixel data for each camera pixel.

19. The method of claim 17, wherein

the first demosaicing and the second demosaicing are selectively combined by switching between the low-resolution full color pixel data and the high-resolution full color pixel data to generate the full color pixel data for each camera pixel.

20. The method of claim 17, further comprising:

prior to detecting the spatial correlation in intensity information of the initial pixel data, generating a correlation map of correlation information of at least one pair of

color channels for the plurality of camera pixels.

21. A method for a minimally invasive surgical system, the method comprising:

alternately assigning single color information to a plurality of first camera pixels;

capturing a first image from a surgical site within a patient, the first image including first pixel data captured by the plurality of first camera pixels;

detecting first spatial correlation of intensity information between a pair of colors in the first pixel data of a first array of camera pixels; and

variably demosaicing each camera pixel in the array of first camera pixels with its single color information and color information of neighboring camera pixels to generate first full color pixel data for each first camera pixel in response to the first spatial correlation.

22. The method of claim 21, further comprising:

alternately assigning single color information to a plurality of second camera pixels;

capturing a second image from the surgical site within the patient, the second image including second pixel data captured by the plurality of second camera pixels;

detecting second spatial correlation of intensity information between a pair of colors in the second pixel data of an array of second camera pixels; and

variably demosaicing each camera pixel in the array of second camera pixels with its single color information and color information of neighboring camera pixels to generate second full color pixel data for each second camera pixel in response to the second spatial correlation.

23. The method of claim 22, wherein

the first image is a left image of a stereo image and the first pixel data is left pixel data captured by left camera pixels, and

the second image is a right image of the stereo image and the second pixel data is right pixel data captured by right camera pixels.

24. The method of claim 23, further comprising:

outputting the left pixel data for display on a left display of a stereo display device, and

outputting the right pixel data for display on a right display of the stereo display device.

25-38. -38. (canceled)