Method and system for assisting an operator in endoscopic navigation

• US 10,004,387 B2
• Filed: 03/26/2009
• Issued: 06/26/2018
• Est. Priority Date: 03/26/2009
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Prisco, Giuseppe
• Primary Examiner(s): Newton, Alexandra L
• Application Number:
  US12/411,501
• Publication Number:
  US 20100249506A1
• Time in Patent Office: 3,379 Days
• Field of Search: 600117, 600424, 600114, 600109, 600115, 600182, 600214, 600196, 600103, 606196, 606108, 606 5, 348 65, 348 45, 356 2
• CPC Class Codes: A61B 1/000094 extracting biological structures
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: A61B 1/00059 provided with identification means for the endoscope
• CPC Class Codes: A61B 1/0051 with controlled bending of insertion part
• CPC Class Codes: A61B 1/009 with bending or curvature detection of the insertion part
• CPC Class Codes: A61B 1/04 combined with photographic or television appliances
• CPC Class Codes: A61B 1/05 characterised by the image sensor, e.g. camera, being in the distal end portion
• CPC Class Codes: A61B 2017/00278 Transorgan operations, e.g. transgastric
• CPC Class Codes: A61B 2034/102 Modelling of surgical devices, implants or prosthesis
• CPC Class Codes: A61B 2034/105 Modelling of the patient, e.g. for ligaments or bones
• CPC Class Codes: A61B 2034/2059 Mechanical position encoders
• CPC Class Codes: A61B 2034/2068 using pointers, e.g. pointers having reference marks for determining coordinates of body points
• CPC Class Codes: A61B 2034/256 having a database of accessory information, e.g. including context sensitive help or scientific articles
• CPC Class Codes: A61B 2034/301 for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes
• CPC Class Codes: A61B 2090/3614 using optical fibre
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 5/065 Determining position of the probe employing exclusively positioning means located on or in the probe, e.g. using position sensors arranged on the probe
• CPC Class Codes: A61B 5/066 Superposing sensor position on an image of the patient, e.g. obtained by ultrasound or x-ray imaging
• CPC Class Codes: A61B 6/12 Arrangements for detecting or locating foreign bodies

Abstract

Navigation guidance is provided to an operator of an endoscope by determining a current position and shape of the endoscope relative to a reference frame, generating an endoscope computer model according to the determined position and shape, and displaying the endoscope computer model along with a patient computer model referenced to the reference frame so as to be viewable by the operator while steering the endoscope within the patient.

Claims

1. A computer-implemented method for assisting an operator in endoscopic navigation, the method comprising:

referencing a patient to a fixed reference frame wherein the patient is movable relative to the fixed reference frame;

retrieving from a computer memory a three-dimensional patient computer model generated from a tomography scan and stored prior to the endoscopic navigation without reference to the fixed reference frame;

subsequent to retrieving the stored three-dimensional patient computer model, referencing the three-dimensional patient computer model to the fixed reference frame;

determining a current position and shape of an endoscopic device relative to the fixed reference frame, wherein the endoscopic device has a distal tip with an image capturing device positioned to capture images from a perspective of the distal tip;

generating a three-dimensional computer model of the endoscopic device according to the determined current position and shape;

displaying the three-dimensional computer model of the endoscopic device registered to the three-dimensional patient computer model on a display screen so as to provide the endoscopic navigation assistance to the operator;

simultaneously displaying a first captured image from a set of captured images from the endoscopic device adjacent to a distal tip of the model of the endoscopic device, the first captured image being displayed in a position based on the registration of the three-dimensional model of the endoscopic device to the three-dimensional patient computer model;

computing a panoramic image from the set of captured images stored in a memory device by mosaicking a plurality of the captured images together;

displaying the panoramic image on the display screen; and

displaying the three-dimensional computer model of the endoscopic device and the three-dimensional patient computer model as a picture-in-picture over the panoramic image on the display screen.

2. The method according to claim 1, wherein the referencing of the patient to the fixed reference frame comprises:

collecting a set of reference points by contacting multiple locations on the patient with a pointer device;

referencing the set of reference points on the patient to the fixed reference frame; and

wherein referencing the three-dimensional patient computer model to the fixed reference frame includes associating a corresponding point of the three-dimensional patient computer model with each of the referenced set of reference points on the patient.

3. The method according to claim 2, wherein referencing of the three-dimensional patient computer model to the fixed reference frame comprises:

aligning the three-dimensional patient computer model with the patient in the fixed reference frame using the associated reference points of the patient and the three-dimensional patient computer model.

4. The method according to claim 2, wherein referencing of the three-dimensional patient computer model to the fixed reference frame comprises:

conforming the three-dimensional patient computer model with the patient by overlaying the associated reference points of the three-dimensional patient computer model and the patient.

5. The method according to claim 2, wherein the referencing of the set of reference points on the patient to the fixed reference frame comprises:

determining a position in the fixed reference frame of a point on the endoscopic device when the point is touched to a corresponding one of the set of reference points on the patient.

6. The method according to claim 1, wherein the three-dimensional patient computer model includes a head and a torso of the patient.

7. The method according to claim 1, wherein displaying the three-dimensional computer model of the endoscopic device comprises:

displaying an indication of a landmark along a path of the endoscopic device that leads to an identified target at an end of the path.

8. The method according to claim 7, further comprising:

displaying a first pattern on a portion of the endoscopic device proximal to the landmark; and

displaying a second pattern on a portion of the endoscopic device distal to the landmark.

9. The method according to claim 8, further comprising:

periodically capturing images with the image capturing device and storing the captured images in the memory device;

determining positions of the image capturing device at times of such image capturing and storing the positions in the memory device; and

storing timestamps corresponding to the times of such image capturing in the memory device.

10. The method according to claim 1, further comprising:

displaying the panoramic image in a first area on the display screen; and

displaying the three-dimensional computer model of the endoscopic device and the three-dimensional patient computer model in a second area on the display screen.

11. The method according to claim 1, wherein simultaneously displaying the first captured image further comprises:

displaying the panoramic image so as to appear to project from the modeled distal tip of the three-dimensional computer model of the endoscopic device being displayed on the display screen.

12. The method according to claim 1, further comprising:

determining positions in the fixed reference frame for at least two points on an anatomic structure within the patient; and

displaying a computer model of the anatomic structure along with the computer models of the patient and endoscopic device on the display screen according to the determined positions for the at least two points on the anatomic structure.

13. The method according to claim 1, wherein referencing the patient to the fixed reference frame includes referencing a first set of points on a patient to the fixed reference frame and wherein referencing the three-dimensional patient computer model to the fixed reference frame includes referencing a second set of points on the three-dimensional patient computer model to the fixed reference frame, the method further comprising conforming the second set of points on the three-dimensional patient computer model to the first set of points on the patient to create a conformed three-dimensional patient computer model, wherein displaying includes displaying the three-dimensional computer model of the endoscopic device and the conformed three-dimensional patient computer model on the display screen.

14. The method of claim 13, wherein creating the conformed three-dimensional patient computer model includes modifying the three-dimensional patient computer model size.

15. A system comprising:

a display screen;

an endoscopic device including an image capturing device positioned at a distal tip of the endoscopic device to capture images from a perspective of the distal tip; and

a processor adapted to;

reference a patient to a fixed reference frame wherein the patient is movable relative to the fixed reference frame;

retrieve from a computer memory a three-dimensional patient computer model generated from a tomography scan and stored prior to the endoscopic navigation without reference to the fixed reference frame;

subsequent to the retrieval of the stored three-dimensional patient computer model, reference the three-dimensional patient computer model to the fixed reference frame;

determine a current position and shape of the endoscopic device relative to the fixed reference frame;

generate a three-dimensional computer model of the endoscopic device according to the determined current position and shape;

display the three-dimensional computer model of the endoscopic device registered to the three-dimensional patient computer model on the display screen so as to provide endoscopic navigation assistance to an operator of the system;

simultaneously display a first captured image, from a set of images captured by the endoscopic device, adjacent to a distal tip of the model of the endoscopic device on the display screen, the first captured image being displayed in a position based on the registration of the three-dimensional model of the endoscopic device to the three-dimensional patient computer model;

compute a panoramic image from the set of captured images stored in a memory device by mosaicking a plurality of the captured images together;

display the panoramic image on the display screen;

display the three-dimensional computer model of the endoscopic device and the three-dimensional patient computer model as a picture-in-picture over the panoramic image on the display screen;

display an indication of a landmark along a path of the endoscopic device; and

display a first pattern on a portion of the endoscopic device proximal to the landmark and display a second pattern on a portion of the endoscopic device distal to the landmark.

16. The system according to claim 15, wherein the processor is adapted to reference the three-dimensional patient computer model to the fixed reference frame by:

receiving information, collected with the endoscopic device, for a set of points on the patient and processing the received information so as to reference the set of points to the fixed reference frame; and

wherein the processor is adapted to reference the three-dimensional patient computer model to the fixed reference frame by associating a corresponding point of the three-dimensional patient computer model with each of the referenced points in the set of points on the patient.

17. The system according to claim 16, wherein the three-dimensional patient computer model is a three-dimensional computer model and the processor is adapted to reference the three-dimensional patient computer model to the fixed reference frame by:

aligning the three-dimensional patient computer model with the patient in the fixed reference frame using the associated reference points of the patient and the three-dimensional patient computer model.

18. The system according to claim 16, wherein the three-dimensional patient computer model is a three-dimensional computer model and the processor is adapted to reference the three-dimensional patient computer model to the fixed reference frame by:

conforming the three-dimensional patient computer model with the patient by overlaying the associated reference points of the three-dimensional patient computer model and the patient.

19. The system according to claim 16, wherein the processor is adapted to reference the set of points on the patient to the fixed reference frame by:

determining a position in the fixed reference frame of a point on the endoscopic device when the point is touched to a corresponding one of the set of points on the patient.

20. The system according to claim 15, further comprising:

one or more optical fibers configured with strain sensors extending through the endoscopic device so as to provide shape information of the endoscopic device.

21. The system according to claim 15, further comprising a memory device;

and wherein the processor is adapted to;

periodically capture images with the image capturing device and store the captured images in the memory device,

determine positions of the image capturing device at times of such image capturing and store the positions in the memory device, and

store timestamps corresponding to the times of such image capturing in the memory device.

22. The system according to claim 15, wherein the processor is adapted to display the panoramic image in a first area on the display screen and the three-dimensional computer model of the endoscopic device and the three-dimensional patient computer model in a second area on the display screen.

23. The system according to claim 15, wherein the processor is adapted to display the captured image adjacent to a distal tip of the model of the endoscopic device on the display screen by displaying the panoramic image so as to appear to project from the distal tip of the three-dimensional computer model of the endoscopic device on the display screen.

24. The system according to claim 15, wherein the processor is adapted to determine positions in the fixed reference frame for at least two points on an anatomic structure within the patient and display a computer model of the anatomic structure along with the computer models of the patient and endoscopic device on the display screen according to the determined positions for the at least two points on the anatomic structure.