Depth based modification of captured images

• US 9,860,510 B2
• Filed: 03/14/2014
• Issued: 01/02/2018
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Kerdok, Amy E., Zhao, Wenyi
• Primary Examiner(s): Yenke, Brian
• Assistant Examiner(s): Haiem, Sean
• Application Number:
  US14/210,936
• Publication Number:
  US 20140267603A1
• Time in Patent Office: 1,390 Days
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

An imaging system processes images of a plurality of objects which have been captured by an image capture device for display. Normal processing of the images is modified as either a function of a depth corresponding to one or more of the plurality of objects appearing in the captured images relative to the image capture device or as a function of the depth and one or more image characteristics extracted from the captured images. A depth threshold may be used to avoid inadvertent modifications due to noise.

Claims

1. An imaging system comprising:

an image capture device, the image capture device providing a captured image of a plurality of objects;

a depth detector, the depth detector providing depths relative to the image capture device;

a display; and

an image processor programmed to;

receive the captured image from the image capture device;

receive information of a plurality of selected objects, the plurality of objects including the plurality of selected objects and at least one non-selected object, wherein the plurality of selected objects is a plurality of lymph nodes, each of the plurality of lymph nodes storing an amount of IndoCyanine Green;

receive a depth corresponding to the plurality of selected objects from the depth detector;

generate a modified processed image by;

modifying normal processing, without eliminating or replacing any part, of an entire image of each selected object of the plurality of selected objects in the captured image according to a function of at least the depth by modifying images of the plurality of lymph nodes so that fluorescences resulting from the amounts of IndoCyanine Green stored in the plurality of lymph nodes are scaled to a common depth for comparison purposes in order to identify a sentinel lymph node among the plurality of lymph nodes, and

not modifying normal processing of an image of the at least one non-selected object; and

provide the modified processed image to the display, so that the modified processed image is displayed on the display.

2. The imaging system of claim 1,

wherein the function of at least the depth comprises a function of the depth and an image characteristic;

and

wherein the image processor is programmed to generate the modified processed image by modifying the, or another, image characteristic of the plurality of selected objects in the captured image according to the function of the depth and the image characteristic.

3. The imaging system of claim 1, wherein the depth detector determines the depth corresponding to the plurality of selected objects by using a structured light technique in which a known light pattern is projected onto a scene of the plurality of objects and the relative light intensities on the scene tracked to derive a depth map for the scene.

4. The imaging system of claim 1,

wherein the image capture device captures stereo images of the plurality of objects;

and

wherein the depth detector determines the depth corresponding to the plurality of selected objects by;

determining corresponding points in the stereo images using a robust sparse image matching algorithm, determining disparities between the corresponding points, converting the disparities to depth values using a predetermined disparity to depth mapping, and processing the depth values to determine the depth corresponding to the plurality of selected objects.

5. The imaging system of claim 4, wherein the depth detector comprises a laser range finder adapted to determine the depth of the plurality of selected objects in a three-dimensional scene including the plurality of objects.

6. The imaging system of claim 1, wherein the image processor is programmed to generate the modified processed image by modifying normal processing of the entire image of each selected object of the plurality of selected objects in the captured image according to the function of at least the depth, only when the depth is less than or equal to a depth threshold.

7. The imaging system of claim 1,

wherein the normal processing of the captured image comprises:

filtering the captured image to enhance one or more image characteristics of the plurality of objects; and

wherein the image processor is programmed to generate the modified processed image by adjusting the filtering for the entire image of each selected object of the plurality of selected objects according to the function of at least the depth when the depth is less than or equal to the depth threshold.

8. The imaging system of claim 1, wherein the plurality of selected objects comprises a user selected object.

9. The imaging system of claim 1, further comprising:

an image characteristics extractor for extracting image characteristics of the captured image, the image characteristics comprising a first image characteristic and a second image characteristic;

wherein the function of at least the depth comprises a function of the depth and the first image characteristic; and

wherein the image processor is programmed to generate the modified processed image by modifying at least one of the first image characteristic and the second image characteristic of the entire image of each selected object of the plurality of selected objects in the captured image according to the function of the depth and the first image characteristic.

10. The imaging system of claim 9, wherein the image characteristics comprises one or more of:

sharpness of edges of the plurality of objects in the captured image, colors of the plurality of objects in the captured image, and a brightness of each of the plurality of objects in the captured image.

11. The imaging system of claim 1, further comprising:

an object identifier, the object identifier identifying the plurality of selected objects in the captured image and providing information of the plurality of selected objects to the image processor.

12. The imaging system of claim 11, further comprising:

an image characteristics extractor, the image characteristics extractor extracting image characteristics of the captured image;

wherein the object identifier identifies the plurality of selected objects by using information of one or more of;

the image characteristics extracted by the image characteristics extractor of images of the plurality of objects in the captured image, shapes and relative sizes of the plurality of objects in the captured image, and spatial relationships between the plurality of objects in the captured image.

13. The imaging system of claim 11, further comprising:

a memory storing information of the plurality of objects;

wherein the object identifier identifies the plurality of selected objects by using the information of the plurality of objects that has been stored in the memory.

14. The imaging system of claim 13, wherein the stored information of the plurality of objects includes information of images of the plurality of objects captured by another image capture device.

15. The imaging system of claim 11, wherein the object identifier identifies the plurality of selected objects by using information provided by a user interface device.

16. The imaging system of claim 1, wherein one or more of the plurality of objects comprises an anatomical structure.

17. The imaging system of claim 1, wherein the image capture device comprises an endoscope configured with one or more of a stereo camera element and an ultrasound transducer.

18. The imaging system of claim 1, wherein the depth corresponding to the plurality of selected objects is one of:

a minimum depth value of the plurality of selected objects, a maximum depth value of the plurality of selected objects, and an intermediary depth value between minimum and maximum depth values of the plurality of selected objects.