Efficient 3-D telestration for local and remote robotic proctoring

• US 9,402,690 B2
• Filed: 08/03/2014
• Issued: 08/02/2016
• Est. Priority Date: 12/31/2008
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Zhao, Wenyi, Wu, Chenyu, Hirvonen, David, Hasser, Christopher J., Miller, Brian E., Mohr, Catherine J., Zhao, Tao, Di Maio, Simon, Hoffman, Brian D.
• Primary Examiner(s): HE, YINGCHUN
• Application Number:
  US14/450,251
• Publication Number:
  US 20150025392A1
• Time in Patent Office: 730 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A61B 2034/2057 Details of tracking cameras
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 2090/373 using light, e.g. by using optical scanners
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 2207/10068 Endoscopic image
• CPC Class Codes: G06T 2207/20096 Interactive definition of curve of interest
• CPC Class Codes: G06T 2207/20101 Interactive definition of point of interest, landmark or seed
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/10 Processing, recording or transmission of stereoscopic or multi-view image signals
• CPC Class Codes: H04N 13/20 Image signal generators
• CPC Class Codes: H04N 13/30 Image reproducers optical systems for producing stereoscopic or other three-dimensional effects G02B30/00
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals

Abstract

An apparatus is configured to show telestration in 3-D to a surgeon in real time. A proctor is shown one side of a stereo image pair, such that the proctor can draw a telestration line on the one side with an input device. Points of interest are identified for matching to the other side of the stereo image pair. In response to the identified points of interest, regions and features are identified and used to match the points of interest to the other side. Regions can be used to match the points of interest. Features of the first image can be matched to the second image and used to match the points of interest to the second image, for example when the confidence scores for the regions are below a threshold value. Constraints can be used to evaluate the matched points of interest, for example by excluding bad points.

Claims

1. A medical system comprising:

an image capture device for capturing stereo images of a work site, each stereo image including a first two-dimensional image and a second two-dimensional image, wherein corresponding points in the first and second two-dimensional images depict a common point in the work site and exhibit a disparity between the corresponding points so as to indicate a depth of the common point relative to the image capture device;

a stereo display displaying the first and second two-dimensional images so as to provide a three-dimensional viewing experience; and

a processor that is programmed to;

receive information of a first telestration indicative of user selected points of interest of the first two-dimensional image;

selectively match the selected points of interest of the first two-dimensional image to corresponding points of the second two-dimensional image to determine matched points of interest of the second two-dimensional image by computing a plurality of image offsets wherein each of the plurality of image offsets is computed using a different image matching method than all others of the plurality of image offsets, and by using one of the computed plurality of image offsets which is selected according at least partially to one or more confidence scores of the plurality of image offsets, wherein the matched points of interest correspond to a second telestration mark; and

cause the first and second telestration marks to be displayed respectively over the first and second two-dimensional images on the stereo display such that the first telestration mark and the second telestration mark appear as a single three-dimensional telestration mark on the stereo display.

2. The medical system of claim 1, further comprising:

a user interface that is usable by a first user to define the first telestration by indicating the user selected points of interest of the first two-dimensional image.

3. The medical system of claim 2, wherein the user interface includes a touchscreen display for displaying the first two-dimensional image.

4. The medical system of claim 2, wherein the user interface includes a monovision display and an input device, wherein the input device is usable by the first user to define the first telestration by indicating the user selected points of interest on the first two-dimensional image being displayed on the monovision display.

5. The medical system of claim 1, wherein the plurality of image offsets includes at least two of:

a global offset which the processor is programmed to compute by using a coarse-to-fine global offset image matching method, a region offset which the processor is programmed to compute by using a coarse-to-fine region image matching method, and a feature offset which the processor is programmed to compute by using a point matching method based upon feature detection and matching.

6. The medical system of claim 5, wherein the processor is programmed to compute the global offset by using normalized cross correlations to compare information of the first and second two-dimensional images.

7. The medical system of claim 5, wherein the processor is programmed to compute the region offset by matching a region of the first two-dimensional image to a region of the second two-dimensional image by using at least one of cross-correlation, two-way matching, least squares regression, and non-linear regression.

8. The medical system of claim 5, wherein the processor is programmed to compute the feature offset by matching features of the first two-dimensional image to features of the second two-dimensional image by using at least one of Harris corner detection, scale-space extrema detection, local extrema detection, and scale invariant feature transform.

9. The medical system of claim 1, wherein the processor is programmed to perform the selectively matching by at least one of selectively matching regions, selectively matching features, selectively interpolating features, and selectively interpolating previously matched points of interest.

10. The medical system of claim 9, wherein the processor is programmed to determine region match confidence scores when the regions are selectively matched, determine feature match scores when the features are selectively matched, selectively match the features in response to the region match confidence scores, selectively interpolate the features in response to the feature match scores, and selectively interpolate the previously matched points of interest in response to the feature match scores;

and wherein locations of the matched points of interest are determined by using the region match confidence scores and the feature match scores.