Single-chip sensor multi-function imaging

• US 9,992,428 B2
• Filed: 01/27/2016
• Issued: 06/05/2018
• Est. Priority Date: 05/14/2012
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Zhao, Wenyi, DiCarlo, Jeffrey, McDowall, Ian E.
• Primary Examiner(s): Rao, Andy
• Application Number:
  US15/008,215
• Publication Number:
  US 20160227129A1
• Time in Patent Office: 860 Days
• Field of Search: 348 77
• CPC Class Codes: A61B 1/00006 of control signals
• CPC Class Codes: A61B 1/00009 of image signals during a use of endoscope
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: A61B 1/00193 adapted for stereoscopic vision
• CPC Class Codes: A61B 1/043 for fluorescence imaging
• CPC Class Codes: A61B 1/05 characterised by the image sensor, e.g. camera, being in the distal end portion
• CPC Class Codes: A61B 1/06 with illuminating arrangements
• CPC Class Codes: A61B 1/0638 providing two or more wavelengths
• CPC Class Codes: A61B 1/0646 with illumination filters
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/555 for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Mixed mode imaging is implemented using a single-chip image capture sensor with a color filter array. The single-chip image capture sensor captures a frame including a first set of pixel data and a second set of pixel data. The first set of pixel data includes a first combined scene, and the second set of pixel data includes a second combined scene. The first combined scene is a first weighted combination of a fluorescence scene component and a visible scene component due to the leakage of a color filter array. The second combined scene includes a second weighted combination of the fluorescence scene component and the visible scene component. Two display scene components are extracted from the captured pixel data in the frame and presented on a display unit.

Claims

1. A surgical system comprising a display image controller connected to a display unit;

the display image controller comprising a demosaic module, a scene component generator connected to the demosaic module, and a display color component generator connected to the scene component generator;

the demosaic module having a first input, a second input, a first output and a second output,

the first input configured to receive a first set of pixel data, and

the second input configured to receive a second set of pixel data,

the first set of pixel data and the second set of pixel data being based on a frame of pixel data captured by a single-chip image capture sensor with a color filter array, the first set of pixel data being a first combined scene including a fluorescence scene component, and the second set of pixel data comprises a second combined scene being a combination of a visible color component scene and the fluorescence scene component;

the first output configured to output a third set of pixel data of the first combined scene;

the second output configured to output a fourth set of pixel data of the second combined scene;

a scene component generator having a first input, a second input, a first output and a second output;

the first input of the scene component generator being connected to the first output of the demosaic module to receive the third set of pixel data of the first combined scene;

the second input of the scene component generator being connected to the second output of the demosaic module to receive a fourth set of pixel data of the second combined scene;

the first output being configure to output a display visible scene component extracted using the third set of pixel data of the first combined scene and the fourth set of pixel data of the second combined scene;

the second output being configured to output a display fluorescence scene component extracted using the third set of pixel data of the first combined scene and the fourth set of pixel data of the second combined scene;

the display color component generator having a first input, a second input, and a plurality of outputs;

the first input of the display color component generator being connected to the first output of the scene component generator to receive the display visible scene component;

the second input of the display color component generator being connected to the second output of the scene component generator to receive the display fluorescence scene component;

each of the plurality of outputs of the display color component generator being configured to output a different weighted combination of the display fluorescence scene component and the display visible scene component; and

the display unit having a plurality of inputs, each of the plurality of inputs being connected to a different one of the plurality of outputs of the display color component generator to receive the weighted combination of the display fluorescence scene component and the display visible scene component from the one of the plurality of outputs of the display component generator, the display unit being configured to generate from the received weighted combinations a displayed scene including a highlighted scene component and a reduced color scene component, wherein the highlighted scene component represents the fluorescence scene component.

2. The surgical system of claim 1, wherein the display visible scene component comprises a linear weighted combination of a first plurality of image pixel data obtained from the third set of pixel data and a second plurality of image pixel data obtained from the fourth set of pixel data.

3. The surgical system of claim 1, wherein the display fluorescence scene component comprises a linear weighted combination of a first plurality of image pixel data obtained from the third set of pixel data and a second plurality of image pixel data obtained from the fourth set of pixel data.

4. The surgical system of claim 1, further comprising:

an illuminator configured to provide at least two illumination components;

wherein one of the illumination components is a fluorescence excitation illumination component;

wherein other illumination components include less than all visible color components of white light; and

wherein the at least two illumination components are provided at the same time.

5. The surgical system of claim 1, wherein the visible color component scene is captured as green color pixels in the single-chip image capture sensor.

6. The surgical system of claim 1, further comprising an image capture unit including the single-chip image capture sensor with the color filter array.

7. The surgical system of claim 6, wherein the color filter array is a Bayer pattern color filter array.