Systems and methods for interactive user interfaces for robotic minimally invasive surgical systems

• US 9,795,446 B2
• Filed: 02/25/2013
• Issued: 10/24/2017
• Est. Priority Date: 06/06/2005
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC), Johns Hopkins University
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: DiMaio, Simon P., Hasser, Christopher J., Taylor, Russell H., Larkin, David Q., Kazanzides, Peter, Deguet, Anton, Vagvolgyi, Balazs Peter, Leven, Joshua
• Primary Examiner(s): NEWTON, ALEXANDRA L
• Application Number:
  US13/775,574
• Publication Number:
  US 20130245375A1
• Time in Patent Office: 1,702 Days
• CPC Class Codes: A61B 1/00193 adapted for stereoscopic vision
• CPC Class Codes: A61B 2017/00203 with speech control or speech recognition
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/378 using ultrasound
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 34/70 Manipulators specially adapted for use in surgery
• CPC Class Codes: A61B 34/71 Manipulators operated by drive cable mechanisms
• CPC Class Codes: A61B 34/76 Manipulators having means for providing feel, e.g. force or tactile feedback
• CPC Class Codes: A61B 8/00 Diagnosis using ultrasonic, sonic or infrasonic waves imaging of objects using sonar G01S15/00
• CPC Class Codes: A61B 8/12 in body cavities or body tracts, e.g. by using catheters catheters per se A61M25/00
• CPC Class Codes: A61B 8/4245 involving determining the position of the probe, e.g. with respect to an external reference frame or to the patient
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: G16H 20/40 relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 50/50 for simulation or modelling of medical disorders

Abstract

In one embodiment of the invention, a method for a minimally invasive surgical system is disclosed. The method includes capturing and displaying camera images of a surgical site on at least one display device at a surgeon console; switching out of a following mode and into a masters-as-mice (MaM) mode; overlaying a graphical user interface (GUI) including an interactive graphical object onto the camera images; and rendering a pointer within the camera images for user interactive control. In the following mode, the input devices of the surgeon console may couple motion into surgical instruments. In the MaM mode, the input devices interact with the GUI and interactive graphical objects. The pointer is manipulated in three dimensions by input devices having at least three degrees of freedom. Interactive graphical objects are related to physical objects in the surgical site or a function thereof and are manipulatable by the input devices.

Claims

1. A method for a minimally invasive teleoperated surgical system, the method comprising:

displaying one or more captured camera images of a surgical site on at least one display device at a first surgeon console, the one or more captured camera images including an image of a first tool;

overlaying a first menu system, a pointer, and a first interactive graphical tool icon onto the one or more captured camera images of the surgical site, the first interactive graphical tool icon associated with the image of the first tool in the one or more captured camera images of the surgical site;

if the pointer is moved to be over a first menu button in the first menu system in response to movement of at least one input device and the first menu button is selected by the at least one input device, overlaying a first pull down menu list of a plurality of menu options each associated with a medical image data set for selection;

if the pointer is moved to be over a second menu button in the first menu system in response to movement of at least one input device and the second menu button is selected by the at least one input device, overlaying a second menu system onto the one or more captured camera images, associated with defining and adjusting virtual fixtures to guide the movement of the first tool;

if the pointer is moved to be over the first interactive graphical tool icon in response to movement of at least one input device and the first interactive graphical tool icon is selected by the at least one input device, overlaying a second pull down menu list of a plurality of menu options each associated with the first tool;

controlling the first tool using the at least one input device to perform a teleoperated surgical procedure; and

implementing virtual fixtures to guide the movement of the first tool if virtual fixtures are selected from the second menu system.

2. The method of claim 1, wherein

the second pull down menu list of the plurality of menu options each associated with the first tool include

a first menu option to select to capture intra-operative images with the first tool and overlay them onto the one or more captured camera images of the surgical site in a flashlight image window near the first tool, and

a second menu option to select to capture intra-operative images with the first tool and overlay them onto the one or more captured camera images of the surgical site in a first inset window.

3. The method of claim 2, wherein

the one or more captured camera images include an image of a second tool;

a second interactive graphical tool icon is overlaid onto the one or more captured camera images of the surgical site, the second interactive graphical tool icon is associated with the image of the second tool in the one or more captured camera images of the surgical site; and

if the pointer is moved to be over the second interactive graphical tool icon in response to movement of at least one input device and the second interactive graphical tool icon is selected by the at least one input device, overlaying a third pull down menu list of a plurality of menu options each associated with the second tool.

4. The method of claim 3, wherein

the third pull down menu list of the plurality of menu options each associated with the second tool include

a first menu option to select to capture intra-operative images with the second tool and overlay them onto the one or more captured camera images of the surgical site in a flashlight image window near the second tool, and

a second menu option to select to capture intra-operative images with the second tool and overlay them onto the one or more captured camera images of the surgical site in a second inset window.

5. The method of claim 2, wherein

if the pointer is moved to be over a fourth menu button in the first menu system in response to movement of at least one input device and the fourth menu button is selected by the at least one input device, overlaying a fourth menu system indicating menu options associated with a plurality of slave robots controllable by the first surgeon console.

6. The method of claim 5, further comprising:

selecting a first menu option associated with a first slave robot controllable by the first surgeon console; and

adapting the first surgeon console to control the first slave robot.

7. The method of claim 6, further comprising:

selecting a second menu option associated with a second slave robot controllable by the first surgeon console; and

adapting the first surgeon console to control the second slave robot.

8. A minimally invasive teleoperated surgical system, the system comprising:

a display device at a first surgeon console to display one or more captured camera images of a surgical site, the one or more captured camera images including an image of a first tool;

a first menu system, a pointer, and a first graphical tool icon to overlay onto the one or more captured camera images of the surgical site, the first graphical tool icon associated with the image of the first tool in the one or more captured camera images of the surgical site;

a first menu button in the first menu system responsive to a movement of the pointer controlled by at least one input device and the first menu button is selected by the at least one input device to overlay a first pull down menu list of a plurality of menu options each associated with a medical image data set for selection;

a second menu button in the first menu system responsive to movement of the pointer controlled by the at least one input device, the second menu button is selected by the at least one input device to overlay, onto the one or more captured camera images of the surgical site, a second menu system associated with defining and adjusting virtual fixtures to guide the movement of the first tool;

controlling the first tool using the at least one input device to perform a teleoperated surgical procedure; and

implementing virtual fixtures to guide the movement of the first tool if virtual fixtures are selected from the second menu system;

wherein if the pointer is moved to be over the first graphical tool icon in response to movement of the at least one input device and the first graphical tool icon is selected by the at least one input device, overlaying a second pull down menu list of a plurality of menu options each associated with the first tool.

9. The system of claim 8, wherein

the second pull down menu list of the plurality of menu options each associated with the first tool include

a first menu option to select to capture intra-operative images with the first tool and overlay them onto the one or more captured camera images of the surgical site in a flashlight image window near the first tool, and

a second menu option to select to capture intra-operative images with the first tool and overlay them onto the one or more captured camera images of the surgical site in a first inset window.

10. The system of claim 9, wherein

the one or more captured camera images include an image of a second tool;

a second graphical tool icon is overlaid onto the one or more captured camera images of the surgical site, the second graphical tool icon is associated with the image of the second tool in the one or more captured camera images of the surgical site; and

if the pointer is moved to be over the second graphical tool icon in response to movement of at least one input device and the second graphical tool icon is selected by the at least one input device, overlaying a third pull down menu list of a plurality of menu options each associated with the second tool.

11. The system of claim 10, wherein

the third pull down menu list of the plurality of menu options each associated with the second tool include

a first menu option to select to capture intra-operative images with the second tool and overlay them onto the one or more captured camera images of the surgical site in a flashlight image window near the second tool, and

a second menu option to select to capture intra-operative images with the second tool and overlay them onto the one or more captured camera images of the surgical site in a second inset window.

12. The method of claim 9, wherein

if the pointer is moved to be over a fourth menu button in the first menu system in response to movement of at least one input device and the fourth menu button is selected by the at least one input device, overlaying a fourth menu system indicating menu options associated with a plurality of slave robots controllable by the first surgeon console.

13. The system of claim 12, further comprising:

a first menu option associated with a first slave robot controllable by the first surgeon console adapted to control the first slave robot.

14. The system of claim 13, further comprising:

a second menu option associated with a second slave robot controllable by the first surgeon console adapted to control the second slave robot.