Tool position and identification indicator displayed in a boundary area of a computer display screen

• US 10,730,187 B2
• Filed: 06/29/2017
• Issued: 08/04/2020
• Est. Priority Date: 06/29/2006
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Larkin, David Q., Nixon, Thomas R., Mintz, David S.
• Primary Examiner(s): Downey, John R
• Application Number:
  US15/638,172
• Publication Number:
  US 20170305016A1
• Time in Patent Office: 1,132 Days
• Field of Search: 600342, 600473-478, 606 1
• CPC Class Codes: A61B 1/00009 of image signals during a use of endoscope
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: A61B 1/0051 with controlled bending of insertion part
• CPC Class Codes: A61B 1/3132 for laparoscopy
• CPC Class Codes: A61B 17/00234 for minimally invasive surgery A61B17/0218, A61B17/0469, A61B17/12013, A61B17/1285, A61B17/29, A61B17/320016 take precedence
• CPC Class Codes: A61B 2017/00199 with a console, e.g. a control panel with a display
• CPC Class Codes: A61B 2017/00292 mounted on or guided by flexible, e.g. catheter-like, means
• CPC Class Codes: A61B 2017/00296 mounted on an endoscope
• CPC Class Codes: A61B 2034/102 Modelling of surgical devices, implants or prosthesis
• CPC Class Codes: A61B 2034/107 Visualisation of planned trajectories or target regions
• CPC Class Codes: A61B 2034/2057 Details of tracking cameras
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2034/301 for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes
• CPC Class Codes: A61B 2034/741 Glove like input devices, e.g. "data gloves"
• CPC Class Codes: A61B 2034/742 Joysticks
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/367 creating a 3D dataset from 2D images using position information
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 2090/373 using light, e.g. by using optical scanners
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/20 : Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 : User interfaces for surgical systems
• CPC Class Codes: A61B 34/30 : Surgical robots
• CPC Class Codes: A61B 34/37 : Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 34/74 : Manipulators with manual electric input means
• CPC Class Codes: A61B 90/36 : Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/361 : Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/37 : Surgical systems with images on a monitor during operation
• CPC Class Codes: B25J 9/1692 : Calibration of manipulator
• CPC Class Codes: B25J 9/1694 : characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: B25J 9/1697 : Vision controlled systems
• CPC Class Codes: G05B 2219/39449 : Pendant, pda displaying camera images overlayed with graphics, augmented reality
• CPC Class Codes: G05B 2219/40607 : Fixed camera to observe workspace, object, workpiece, global
• CPC Class Codes: G05B 2219/45123 : Electrogoniometer, neuronavigator, medical robot used by surgeon to operate

Abstract

An endoscope captures images of a surgical site for display in a viewing area of a monitor. When a tool is outside the viewing area, a GUI indicates the position of the tool by positioning a symbol in a boundary area around the viewing area so as to indicate the tool position. The distance of the out-of-view tool from the viewing area may be indicated by the size, color, brightness, or blinking or oscillation frequency of the symbol. A distance number may also be displayed on the symbol. The orientation of the shaft or end effector of the tool may be indicated by an orientation indicator superimposed over the symbol, or by the orientation of the symbol itself. When the tool is inside the viewing area, but occluded by an object, the GUI superimposes a ghost tool at its current position and orientation over the occluding object.

Claims

1. A method for indicating a position of a tool at a work site when the tool is entirely out of a field of view of an imaging device, the method comprising:

a processor causing images captured by the imaging device to be displayed in a viewing area on a display screen;

the processor determining a position of the tool in a reference frame of the imaging device, wherein the tool has a shaft and an end effector including opposing members coupled to the shaft;

the processor determining a position to display a non-depictive symbol for the tool in a boundary area circumscribing the viewing area on the display screen so as to indicate a direction of the determined position of the tool relative to the field of view of the imaging device, by determining a trajectory of the tool from current and past positions of the tool, and determining where an extrapolation of the trajectory enters the field of view of the imaging device;

wherein the non-depictive symbol for the tool does not resemble the end effector of the tool in appearance; and

wherein the position of the non-depictive symbol for the tool in the boundary area does not indicate a distance of the tool from the field of view of the imaging device; and

the processor causing the non-depictive symbol to be displayed at the determined position in the boundary area while images that were captured by the imaging device are restricted to being displayed in the viewing area on the display screen.

2. The method according to claim 1, wherein the non-depictive symbol for the tool provides information identifying the tool.

3. The method according to claim 1, wherein the non-depictive symbol for the tool provides information identifying a manipulator coupled to the tool.

4. The method according to claim 3, wherein the non-depictive symbol for the tool is a color that is associated with and indicated on the manipulator.

5. The method according to claim 3, wherein the non-depictive symbol for the tool is marked with a number that is associated with and indicated on the manipulator.

6. The method according to claim 1, wherein the processor determines the position of the tool by using kinematics for a manipulator coupled to the tool.

7. The method according to claim 1, wherein the processor causing the non-depictive symbol for the tool to be displayed in the boundary area comprises:

determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the imaging device, and causing the non-depictive symbol for the tool to be displayed so that a size of the non-depictive symbol for the tool indicates the distance.

8. The method according to claim 1, wherein the processor causing the non-depictive symbol for the tool to be displayed in the boundary area comprises:

determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the imaging device, and causing the non-depictive symbol for the tool to be displayed so that a color of the non-depictive symbol for the tool indicates the distance.

9. The method according to claim 1, wherein the processor causing the non-depictive symbol for the tool to be displayed in the boundary area comprises:

determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the imaging device, and causing the non-depictive symbol for the tool to be displayed so that an intensity of a color of the non-depictive symbol for the tool indicates the distance.

10. The method according to claim 1, wherein the processor causing the non-depictive symbol for the tool to be displayed in the boundary area comprises:

determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the imaging device, and causing the non-depictive symbol for the tool to be displayed so that a color of the non-depictive symbol for the tool indicates the distance by a position of the color in a color spectrum.

11. The method according to claim 1, wherein the processor causing the non-depictive symbol for the tool to be displayed in the boundary area comprises:

determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the imaging device, and causing the non-depictive symbol for the tool to be displayed so that a frequency of blinking of the non-depictive symbol for the tool indicates the distance.

12. The method according to claim 1, wherein the processor causing the non-depictive symbol for the tool to be displayed in the boundary area comprises:

determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the imaging device, and causing the non-depictive symbol for the tool to be displayed so that a frequency of oscillation of the non-depictive symbol for the tool about the determined position in the boundary area indicates the distance.

13. The method according to claim 1, wherein the processor causing the non-depictive symbol for the tool to be displayed in the boundary area comprises:

determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the imaging device, and causing the non-depictive symbol for the tool to be displayed so that the distance is indicated by overlaying a distance number over the non-depictive symbol for the tool.

14. The method according to claim 1,

wherein the processor determining the position of the tool further comprises determining an orientation of the end effector in the reference frame of the imaging device, and

wherein the processor causing the non-depictive symbol for the tool to be displayed in the boundary area comprises:

causing an orientation indicator to be displayed over the non-depictive symbol for the tool such that the orientation indicator is oriented so as to indicate the orientation of the end effector.

15. The method according to claim 1,

wherein the tool includes a shaft having an axis extending along a length of the shaft,

wherein the processor determining the position of the tool further comprises determining an orientation of the axis in the reference frame of the imaging device, and

wherein the processor causing the non-depictive symbol for the tool to be displayed in the boundary area comprises:

causing an orientation indicator to be displayed over the non-depictive symbol for the tool such that the orientation indicator is oriented so as to indicate the orientation of the axis.