Real-time depth estimation from monocular endoscope images

• US 9,750,399 B2
• Filed: 03/25/2010
• Issued: 09/05/2017
• Est. Priority Date: 04/29/2009
• Status: Active Grant

• Current Assignee: Koninklijke Philips N.V.
• Original Assignee: Koninklijke Philips N.V.
• Inventors: Popovic, Aleksandra
• Primary Examiner(s): HUANG, FRANK F
• Application Number:
  US13/266,465
• Publication Number:
  US 20120056986A1
• Time in Patent Office: 2,721 Days
• Field of Search: 348 45
• CPC Class Codes: A61B 1/000094 extracting biological structures
• CPC Class Codes: A61B 1/04 combined with photographic or television appliances
• CPC Class Codes: A61B 1/313 for introducing through surgical openings, e.g. laparoscopes
• CPC Class Codes: A61B 2090/061 for measuring dimensions, e.g. length
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 5/065 Determining position of the probe employing exclusively positioning means located on or in the probe, e.g. using position sensors arranged on the probe
• CPC Class Codes: A61B 8/461 Displaying means of special interest

Abstract

An endoscopic surgical method involves an advancement of an endoscope (20) to a target location within an anatomical region of a body and a generation of a plurality of endoscopic video frames (22) as the endoscope (20) is advanced to the target location with the endoscopic video frames (2) illustrating monocular endoscopic images of the anatomical region. For real-time estimation of a depth of an object within monocular endoscopic images (e.g., depth of a bronchial wall within an monocular endoscopic images of a bronchial tube), the method further involves (S41) a determination of an optical flow of one or more image points within a frame time series of the monocular endoscopic images of the anatomical region, and (S42) an estimation of a depth field indicative of a depth of the object within the monocular endoscopic images as a function of the optical flow of the image point(s).

Claims

1. A minimally invasive surgical system, comprising:

a monocular endoscope, having an optical axis, for generating a plurality of endoscopic video frames as the monocular endoscope is advanced along the optical axis of the monocular endoscope to a target location within an anatomical region of a body, the endoscopic video frames illustrating monocular endoscopic images of the anatomical region; and

an endoscopic surgical control unit in communication with the endoscope to receive the endoscopic video frames as the endoscope is advanced to the target location,

wherein the endoscope surgical control unit is operable to track positions of the endoscope within the anatomical region as the endoscope is advanced to the target location within the anatomical region,

wherein the endoscopic surgical control unit is operable to estimate a depth field indicative of a depth of an object within the monocular endoscopic images of the anatomical region as a function of an optical flow of at least one image point within a frame time series of the monocular endoscopic image of the anatomical region and the tracked positions of the endoscope within the anatomical region, and

wherein the endoscopic surgical control unit is operable to determine the optical flow by generating a vector field including a plurality of vectors, each vector indicative of a motion of one of the image points within the frame time series.

2. The minimally invasive surgical system of claim 1, wherein the endoscopic surgical control unit is further operable to generate a depth map display representative of a depth field estimation.

3. The minimally invasive surgical system of claim 1, wherein the endoscopic surgical control unit is further operable to register the monocular endoscopic images with a pre-operative image of the anatomical region of the body as a function of the depth field estimation.

4. The minimally invasive surgical system of claim 1, wherein the endoscopic surgical control unit is further operable to pre-operatively plan a kinematic path for the endoscope to reach the target location within the anatomical region.

5. The minimally invasive surgical system of claim 1, wherein the endoscope is one of a group including a bronchoscope and a nested cannula.

6. An endoscopic surgical method, comprising:

advancing a monocular endoscope to a target location within an anatomical region of a body;

generating a plurality of endoscopic video frames as the monocular endoscope is advanced along a focal axis of the monocular endoscope to the target location, the endoscopic video frames illustrating monocular endoscopic images of the anatomical region;

tracking positions of the endoscope within the anatomical region as the monocular endoscope is advanced to the target location within the anatomical region;

determining an optical flow of at least one image point within a frame time series of the monocular endoscopic images of the anatomical region; and

estimating a depth field indicative of a depth of an object within the monocular endoscopic images as a function of the optical flow and the tracked positions of the endoscope within the anatomical region;

wherein the step of determining the optical flow includes generating a vector field including a plurality of vectors, each vector indicative of a motion of one of the image points within the frame time series.

7. The endoscopic surgical method of claim 6, wherein the estimation of the depth field includes:

identifying a focus of expansion within the vector field; and

computing a depth point for each image point as a function of a distance of each image point from the focus of expansion.

8. The endoscopic surgical method of claim 6, wherein the estimation of the depth field includes:

computing a depth point for each image point as a function of an amplitude of each vector in the vector field.

9. The endoscopic surgical method of claim 6, wherein the estimation of the depth field includes:

computing a depth point for an image point as a function of a velocity of each vector in the vector field.

10. The endoscopic surgical method of claim 6, wherein the estimation of the depth field includes:

computing projection matrices as a function of the vector field.

12. The endoscopic surgical method of claim 6, further comprising:

displaying a depth map representative of the depth field estimation.

11. An endoscopic surgical method, comprising;

advancing a monocular endoscope to a target location within an anatomical region of a body;

generating a plurality of endoscope video frames as the monocular endoscope is advanced along a focal axis of the monocular endoscope to the target location, the endoscopic video frames illustrating monocular endoscopic images of the anatomical region;

determining an optical flow of at least one image point within frame time series of the monocular endoscopic images of the anatomical region; and

estimating a depth field indicative of a depth of an object within the monocular endoscope images as a function of the optical flow,

wherein the determination of the optical flow includes generating a vector field including a plurality of vectors, each vector indicative of a motion of one of the image points within the frame time series, and

wherein the estimation of the depth field includes computing projecting matrices as a function of the vector field, and computing a depth point for each image point as a function of a geometric triangulation of projection elements of the projection matrices.