INTERACTIVE USER INTERFACES FOR ROBOTIC MINIMALLY INVASIVE SURGICAL SYSTEMS

• US 20130245375A1
• Filed: 02/25/2013
• Published: 09/19/2013
• Est. Priority Date: 06/06/2005
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC), Johns Hopkins University
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC), Johns Hopkins University
• Inventors: DiMaio, Simon P., Hasser, Christopher J., Taylor, Russell H., Larkin, David Q., Kazanzides, Peter, Deguet, Anton, Vagvolgyi, Balazs Peter, Leven, Joshua
• Granted Patent: US 9,795,446 B2
• Time in Patent Office: Days
• US Class Current: 600/111
• CPC Class Codes: A61B 1/00193 adapted for stereoscopic vision
• CPC Class Codes: A61B 2017/00203 with speech control or speech recognition
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/378 using ultrasound
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 34/70 Manipulators specially adapted for use in surgery
• CPC Class Codes: A61B 34/71 Manipulators operated by drive cable mechanisms
• CPC Class Codes: A61B 34/76 Manipulators having means for providing feel, e.g. force or tactile feedback
• CPC Class Codes: A61B 8/00 Diagnosis using ultrasonic, sonic or infrasonic waves imaging of objects using sonar G01S15/00
• CPC Class Codes: A61B 8/12 in body cavities or body tracts, e.g. by using catheters catheters per se A61M25/00
• CPC Class Codes: A61B 8/4245 involving determining the position of the probe, e.g. with respect to an external reference frame or to the patient
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: G16H 20/40 relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 50/50 for simulation or modelling of medical disorders

Abstract

In one embodiment of the invention, a method for a minimally invasive surgical system is disclosed. The method includes capturing and displaying camera images of a surgical site on at least one display device at a surgeon console; switching out of a following mode and into a masters-as-mice (MaM) mode; overlaying a graphical user interface (GUI) including an interactive graphical object onto the camera images; and rendering a pointer within the camera images for user interactive control. In the following mode, the input devices of the surgeon console may couple motion into surgical instruments. In the MaM mode, the input devices interact with the GUI and interactive graphical objects. The pointer is manipulated in three dimensions by input devices having at least three degrees of freedom. Interactive graphical objects are related to physical objects in the surgical site or a function thereof and are manipulatable by the input devices.

Claims

1-22. -22. (canceled)

23. A method for a minimally invasive surgical system, the method comprising:

displaying one or more captured camera images of a surgical site on at least one display device at a first surgeon console, the one or more captured camera images including an image of a first tool;

overlaying a first menu system, a pointer, and a first graphical tool icon onto the one or more captured camera images of the surgical site, the first graphical tool icon associated with the image of the first tool in the one or more video images of the surgical site;

if the pointer is moved to be over a first menu button in the first menu system in response to movement of at least one input device and the first menu button is selected by the at least one input device, overlaying a first pull down menu list of a plurality of menu options each associated with a medical image data set for selection;

if the pointer is moved to be over a second menu button in the first menu system in response to movement of at least one input device and the second menu button is selected by the at least one input device, overlaying a second menu system associated with defining and adjusting virtual fixtures; and

if the pointer is moved to be over the first graphical tool icon in response to movement of at least one input device and the first graphical tool icon is selected by the at least one input device, overlaying a second pull down menu list of a plurality of menu options each associated with the first tool.

24. The method of claim 23, wherein

the second pull down menu list of the plurality of menu options each associated with the first tool include

a first menu option to select to capture intra-operative images with the first tool and overlay them onto the one or more captured camera images of the surgical site in a flashlight image window near the first tool, and

a second menu option to select to capture intra-operative images with the first tool and overlay them onto the one or more captured camera images of the surgical site in a first inset window.

25. The method of claim 24, wherein

the one or more captured camera images include an image of a second tool;

a second graphical tool icon is overlaid onto the one or more captured camera images of the surgical site, the second graphical tool icon is associated with the image of the second tool in the one or more video images of the surgical site; and

if the pointer is moved to be over the second graphical tool icon in response to movement of at least one input device and the second graphical tool icon is selected by the at least one input device, overlaying a third pull down menu list of a plurality of menu options each associated with the second tool.

26. The method of claim 25, wherein

the third pull down menu list of the plurality of menu options each associated with the second tool include

a first menu option to select to capture intra-operative images with the second tool and overlay them onto the one or more captured camera images of the surgical site in a flashlight image window near the second tool, and

a second menu option to select to capture intra-operative images with the second tool and overlay them onto the one or more captured camera images of the surgical site in a second inset window.

27. The method of claim 24, wherein

if the pointer is moved to be over a fourth menu button in the first menu system in response to movement of at least one input device and the fourth menu button is selected by the at least one input device, overlaying a fourth menu system indicating menu options associated with a plurality of slave robots controllable by the first surgeon console.

28. The method of claim 27, further comprising:

selecting a first menu option associated with a first slave robot controllable by the first surgeon console; and

adapting the first surgeon console to control the first slave robot.

29. The method of claim 28, further comprising:

selecting a second menu option associated with a second slave robot controllable by the first surgeon console; and

adapting the first surgeon console to control the second slave robot.

30. A method for a minimally invasive surgical system, the method comprising:

displaying one or more captured camera images of a surgical site on at least one display device at a first surgeon console, the one or more captured camera images including an image of a first tool;

overlaying a first menu system and a pointer onto the one or more captured camera images of the surgical site,

receiving a signal moving the pointer over the first menu system and overlaying a plurality of first menu buttons onto the one or more captured camera images of the surgical site;

receiving a signal selecting a first menu button in the first menu system;

in response to the selection of the first menu button, overlaying a second menu system onto the one or more captured camera images of the surgical site, the second menu system including a plurality of second menu buttons associated with a context of the first menu button to further control the minimally invasive surgical system or provide further information regarding the surgical procedure.

31. The method of claim 30, further comprising:

receiving a signal selecting a second menu button in the first menu system; and

in response to the selection of the second menu button, overlaying a third menu system onto the one or more captured camera images of the surgical site, the third menu system including a plurality of third menu buttons associated with a context of the second menu button to further control the minimally invasive surgical system or to display further information regarding the surgical procedure.

32. The method of claim 30, further comprising:

receiving a signal selecting a first menu button in the second menu system; and

in response to the selection of the first menu button in the second menu system, controlling the minimally invasive surgical system in accordance with the context of the first menu button.

33. The method of claim 30, further comprising:

receiving a signal selecting a first menu button in the second menu system; and

in response to the selection of the first menu button in the second menu system, displaying information regarding the surgical procedure overlaid onto the one or more captured camera images of the surgical site in accordance with the context of the first menu button.

34. A method for a minimally invasive surgical system, the method comprising:

displaying one or more captured camera images of a surgical site onto a stereoscopic display device at a first surgeon console, the one or more captured camera images appearing to be three dimensional (3D) to a user and including an image of a first tool;

overlaying a first tool icon and a pointer onto the one or more captured camera images of the surgical site to appear to be 3D to a user, the first tool icon located near an image of the first tool in the one or more captured camera images, wherein the pointer is manipulated in three dimensions within the one or more captured camera images by at least one input device of a surgeon console having at least three degrees of freedom;

receiving a signal moving the pointer over the first tool icon;

receiving a signal selecting the first tool icon;

in response to the selection of the first tool icon, overlaying a first menu system onto the one or more captured camera images of the surgical site, the first menu system including a plurality of first menu buttons associated with a context of the first tool to further control the first tool or provide information associated with the first tool.

35. The method of claim 34, further comprising:

tracking a position of the first tool in the surgical site, and

wherein in response to the tracking of the position of the first tool, the first tool icon is overlaid near images of the first tool in the one or more captured camera images.

36. The method of claim 35, further comprising:

adjusting a size of the first tool icon in response to a change in depth of the first tool in the surgical site.

37. The method of claim 34, further comprising:

receiving a signal selecting a first menu button in the first menu system; and

in response to the selection of the first menu button, overlaying intra-operative images captured by the first tool within an inset window onto the one or more captured camera images of the surgical site.

38. The method of claim 34, further comprising:

receiving a signal selecting a second menu button in the first menu system; and

in response to the selection of the second menu button, overlaying intra-operative images captured by the first tool within a flashlight image window onto the one or more captured camera images of the surgical site.

39. The method of claim 34, wherein

the one or more captured camera images further include an image of a second tool;

and

the method further includes

overlaying a second tool icon onto the one or more captured camera images of the surgical site, the second tool icon located near the second tool,

receiving a signal moving the pointer over the second tool icon;

receiving a signal selecting the second tool icon;

in response to the selection of the second tool icon, overlaying a second menu system onto the one or more captured camera images of the surgical site, the second menu system including a plurality of second menu buttons associated with a context of the second tool to further control the second tool or provide information associated with the second tool.

40. A minimally invasive surgical system comprising:

a stereoscopic endoscope to capture camera images of a surgical site and a first tool;

a stereoscopic video display device of a surgeon console to display stereo video images to a user to view images with a three dimensional (3D) appearance;

a processor coupled to the stereoscopic endoscope and the stereoscopic video display device, in response to stored program instructions the processor configured to

overlay a first tool icon and a pointer onto the one or more captured camera images of the surgical site, the first tool icon located near the first tool,

receive a signal moving the pointer over the first tool icon;

receive a signal selecting the first tool icon; and

in response to the selection of the first tool icon, overlay a first menu system onto the one or more captured camera images of the surgical site, the first menu system including a plurality of first menu buttons associated with the first tool to further control the first tool or provide information associated with the first tool.

87. The minimally invasive surgical system of claim 40, wherein in response to further stored program instructions the processor is further configured to

track a position of the first tool in the surgical site,

overlay the first tool icon near images of the first tool in the one or more captured camera images in response to the tracking of the position of the first tool, and

adjust a size of the first tool icon in response to a change in depth of the first tool in the surgical site.

88. The minimally invasive surgical system of claim 87, wherein in response to further stored program instructions the processor is further configured to

adjust a size of the first tool icon in response to a change in depth of the first tool in the surgical site.

41-86. -86. (canceled)