Single-chip sensor multi-function imaging

• US 9,277,205 B2
• Filed: 05/14/2013
• Issued: 03/01/2016
• Est. Priority Date: 05/14/2012
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Zhao, Wenyi, DiCarlo, Jeffrey, McDowall, Ian E.
• Primary Examiner(s): KOSTAK, VICTOR R
• Application Number:
  US13/893,536
• Publication Number:
  US 20130300836A1
• Time in Patent Office: 1,022 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A61B 1/00006 of control signals
• CPC Class Codes: A61B 1/00009 of image signals during a use of endoscope
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: A61B 1/00193 adapted for stereoscopic vision
• CPC Class Codes: A61B 1/043 for fluorescence imaging
• CPC Class Codes: A61B 1/05 characterised by the image sensor, e.g. camera, being in the distal end portion
• CPC Class Codes: A61B 1/06 with illuminating arrangements
• CPC Class Codes: A61B 1/0638 providing two or more wavelengths
• CPC Class Codes: A61B 1/0646 with illumination filters
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/555 for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Mixed mode imaging is implemented using a single-chip image capture sensor with a color filter array. The single-chip image capture sensor captures a frame including a first set of pixel data and a second set of pixel data. The first set of pixel data includes a first combined scene, and the second set of pixel data includes a second combined scene. The first combined scene is a first weighted combination of a fluorescence scene component and a visible scene component due to the leakage of a color filter array. The second combined scene includes a second weighted combination of the fluorescence scene component and the visible scene component. Two display scene components are extracted from the captured pixel data in the frame and presented on a display unit.

Claims

1. A system comprising:

an image capture unit including a single-chip image capture sensor with a color filter array,

wherein the single-chip image capture sensor is configured to capture a frame of pixel data comprising a first set of pixel data and a second set of pixel data,

wherein the first set of pixel data comprises a first combined scene including a fluorescence scene component, and

wherein the second set of pixel data comprises a second combined scene including a combination of a visible color component scene and the fluorescence scene component;

a scene processing module configured to;

receive the first set of pixel data and the second set of pixel data;

extract a display visible scene component from the first and second combined scenes,

wherein the display visible scene component comprises a first linear weighted combination of a first plurality of image pixel data obtained from the first set of pixel data and a second plurality of image pixel data obtained from the second set of pixel data, and

wherein the first linear weighted combination is based on a first scale factor;

extract a display fluorescence scene component from the first and second combined scenes,

wherein the display fluorescence scene component comprises a second linear weighted combination of the first plurality of image pixel data and the second plurality of image pixel data,

wherein the second linear weighted combination is based on a second scale factor; and

generate a plurality of weighted combinations of the display fluorescence scene component and the display visible scene component; and

a display unit connected to the scene processing module to receive the plurality of weighted combinations, and configured to generate from the plurality of weighted combinations a displayed scene including a highlighted scene component and a reduced color scene component,

wherein the highlighted scene component represents the fluorescence scene component.

2. The system of claim 1:

wherein the first scale factor comprises a ratio including two color components in a first calibration image;

wherein the calibration image is captured by the image capture unit from light reflected from an illuminator; and

wherein the illuminator provides only one of the two color components.

3. The system of claim 1:

wherein the second scale factor comprises a ratio including two color components in a second calibration image; and

wherein the second calibration image is captured by the image capture unit from fluorescence.

4. The system of claim 2:

wherein the second scale factor comprises a ratio including two color components in a second calibration image; and

wherein the second calibration image is captured by the image capture unit from fluorescence.

5. The system of claim 1, further comprising:

an illuminator configured to provide at least two illumination components,

wherein one of the illumination components is a fluorescence excitation illumination component,

wherein other illumination components include less than all visible color components of white light, and

wherein the at least two illumination components are provided at the same time.

6. The system of claim 5, the illuminator further comprising:

a fluorescence excitation illumination source; and

a visible color component illumination source.

7. The system of claim 6, further comprising:

a power level and power supply controller connected to the illuminator; and

a mode changer, coupled to the power level and power supply controller, having a first state and a second state,

wherein, when the mode changer has the first state, the power level and power supply controller (a) provides power to the visible color component illumination source, and not to the fluorescence excitation illumination source, and (b) the visible color component illumination source has a first level of illumination; and

wherein, when the mode changer has the second state, the power level and power supply controller (a) provides power to the visible color component illumination source and to the fluorescence excitation illumination source, and (b) reduces the level of illumination of the first visible color component illumination source relative to a level of illumination of the visible color component illumination source in the first state.

8. The system of claim 1, the scene processing module further comprising:

a demosaic module coupled to the image capture unit to receive, in a first color channel, the first set of pixel data comprising the first combined scene, wherein the demosaic module is configured to demosaic the first set of pixel data to obtain the first plurality of image pixel data comprising the first combined scene.

9. The system of claim 8:

wherein the demosaic module is coupled to the image capture unit to receive, in a second color channel, the second set of pixel data comprising the second combined scene; and

wherein the demosaic module is configured to demosaic the second set of pixel data to obtain the second plurality of image pixel data comprising the second combined scene.

10. The system of claim 1, wherein the visible color component scene is captured as green color pixels in the single-chip image capture sensor.

11. A method comprising:

receiving a frame captured by a single-chip image capture sensor having a color filter array,

wherein the frame comprises a first set of pixel data and second set of pixel data,

wherein the first set of pixel data comprises a first combined scene including a fluorescence scene component, and

wherein the second set of pixel data comprises a second combined scene including a combination of a visible color component scene and the fluorescence scene component;

extracting a display visible scene component from the first and second combined scenes,

wherein the display visible scene component comprises a first linear weighted combination of a first plurality of image pixel data obtained from the first set of pixel data and a second plurality of image pixel data obtained from the second set of pixel data, and

wherein the first linear weighted combination is based on a first scale factor, and

extracting a display fluorescence scene component from the first and second combined scenes,

wherein the display fluorescence scene component comprises a second linear weighted combination of the first plurality of image pixel data and the second plurality of image pixel data,

wherein the second linear weighted combination is based on a second scale factor;

generating a plurality of weighted combinations of the display fluorescence scene component and the display visible scene component; and

generating from the plurality of weighted combinations a displayed scene including a highlighted scene component corresponding to the fluorescence scene component and a reduced color scene component.

12. The method of claim 11:

wherein the first scale factor comprises a ratio including two color components in a first calibration image;

wherein the calibration image is captured by the image capture unit from light reflected from an illuminator; and

wherein the illuminator provides only one of the two color components.

13. The method of claim 11:

wherein the scale factor comprises a ratio including two color components in a second calibration image; and

wherein the second calibration image is captured by the image capture unit from fluorescence.

14. The method of claim 12:

wherein the scale factor comprises a ratio including two color components in a second calibration image; and

wherein the second calibration image is captured by the image capture unit from fluorescence.

15. The method of claim 11 further comprising:

illuminating a surgical site with at least two illumination components;

wherein one of the illumination components is a fluorescence excitation illumination component,

wherein other illumination components include less than all visible color components of white light, and

wherein the at least two illumination components are provided at the same time.

16. The method of claim 11 further comprising:

demosaicing the first set of pixel data comprising the first combined scene to obtain the first plurality of image pixel data comprising the first combined scene, and,

demosaicing the second set of pixel data comprising the second combined scene to obtain the second plurality of image pixel data comprising the second combined scene.