Quantitative three-dimensional imaging of surgical scenes

• US 10,368,054 B2
• Filed: 03/28/2015
• Issued: 07/30/2019
• Est. Priority Date: 03/28/2014
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Panescu, Dorin, Jones, Daniel H
• Primary Examiner(s): Adams, Eileen M
• Application Number:
  US15/300,255
• Publication Number:
  US 20170188011A1
• Time in Patent Office: 1,585 Days
• Field of Search: None
• CPC Class Codes: A61B 1/00009 of image signals during a use of endoscope
• CPC Class Codes: A61B 1/051 Details of CCD assembly
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/10068 Endoscopic image
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06T 7/0014 using an image reference approach
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/302 for viewing without the aid of special glasses, i.e. using autostereoscopic displays
• CPC Class Codes: H04N 2213/001 Constructional or mechanical details

Abstract

A device is provided that includes an endoscope; an image sensor array is disposed to image a field of view adjacent to the endoscope, each sensor includes a pixel array that is separate from the pixel arrays of other sensors; and a light source is disposed to illuminate the field of view.

Claims

1. A device comprising:

an endoscope;

an image sensor array comprising at least three imaging sensors having coplanar overlapping fields of view, disposed to image fields of view adjacent to the endoscope, wherein each imaging endoscope,

wherein each imaging sensor including a pixel array that is separate from the pixel arrays of other imaging sensors; and

a light source disposed to illuminate the field of view;

wherein the endoscope includes an elongated portion having a first end portion and a second end portion opposite the first end portion;

wherein the image sensor array is disposed displaced from the first end portion of the endoscope, closer to the second end portion of the endoscope;

the device further including;

a light pipe disposed to transmit an image from a field of view adjacent the first end portion of the endoscope to the image sensor array displaced from the first end portion, closer to the second end portion of the endoscope.

2. The device of claim 1,

wherein the light source produces only non-structured light.

3. The device of claim 1,

wherein the light source produces white light.

4. The device of claim 1,

wherein the light source produces white light.

5. The device of claim 1,

wherein an end of the endoscope opposite the first end portion of the elongated portion is configured for mechanical coupling with a mechanical surgical arm.

6. A device comprising:

an endoscope;

an image sensor array disposed to image a field of view adjacent to the endoscope, each sensor including a pixel array that is separate from the pixel arrays of other sensors;

and a light source disposed to illuminate the field of view; and

a controller configured to;

determine a three-dimensional location of a target object based upon image information captured by the image sensor array;

wherein the controller is configured to identify pixels in each of multiple image sensors that are wherein identifying pixels includes using a correlation process to match image projections of the same target object in each of multiple image sensors;

wherein determining the three-dimensional location includes determining an estimated location of the target object in at least one dimension based at least in part upon relative positions of at least three identified pixels;

and in response to the determined estimated location differing by more than an acceptable tolerance, repeating the correlation process to refine the matching of image projections.

7. The device of claim 6,

wherein the three-dimensional location of the target object is determined based at least in part upon a pixel-distance relationship between identified pixels and the physical characteristics of the image sensor array.

8. The device of claim 6,

wherein the controller is configured to identify pixels in each of multiple image sensors of the image sensor array that are illuminated by projections from the same target object.

9. The device of claim 8,

wherein identifying pixels includes matching image projections of a same target object in each of multiple arrays of the image sensor array.

10. A device comprising:

an endoscope;

an image sensor array disposed to image a field of view adjacent to the endoscope, each sensor including a pixel array that is separate from the pixel arrays of other sensors;

and a light source disposed to illuminate the field of view; and

a controller configured to, determine a three-dimensional location of a target object based upon image information captured by the image sensor array;

wherein the controller is configured to identify pixels in each of multiple image sensors that are illuminated by projections from the same target object; and

wherein identifying pixels includes using a correlation process to match image projections of the same target object in each of multiple image sensors wherein determining the three-dimensional location includes determining an estimated location of the target object in at least one physical dimension generally parallel to a plane of the sensors based upon relative positions of at least three identified pixels;

and in response to the determined estimated location matching to within an acceptable tolerance, determining an estimated physical location of the target object in at least one other physical dimension generally perpendicular to the plane of the sensors based at least in part upon the determined physical location of the target object in the at least one physical dimension.