TOOL POSITION AND IDENTIFICATION INDICATOR DISPLAYED IN A BOUNDARY AREA OF A COMPUTER DISPLAY SCREEN

• US 20170305016A1
• Filed: 06/29/2017
• Published: 10/26/2017
• Est. Priority Date: 06/29/2006
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Larkin, David Q., Nixon, Thomas R., Mintz, David S.
• Granted Patent: US 10,730,187 B2
• Time in Patent Office: Days
• CPC Class Codes: A61B 1/00009 of image signals during a use of endoscope
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: A61B 1/0051 with controlled bending of insertion part
• CPC Class Codes: A61B 1/3132 for laparoscopy
• CPC Class Codes: A61B 17/00234 for minimally invasive surgery A61B17/0218, A61B17/0469, A61B17/12013, A61B17/1285, A61B17/29, A61B17/320016 take precedence
• CPC Class Codes: A61B 2017/00199 with a console, e.g. a control panel with a display
• CPC Class Codes: A61B 2017/00292 mounted on or guided by flexible, e.g. catheter-like, means
• CPC Class Codes: A61B 2017/00296 mounted on an endoscope
• CPC Class Codes: A61B 2034/102 Modelling of surgical devices, implants or prosthesis
• CPC Class Codes: A61B 2034/107 Visualisation of planned trajectories or target regions
• CPC Class Codes: A61B 2034/2057 Details of tracking cameras
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2034/301 for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes
• CPC Class Codes: A61B 2034/741 Glove like input devices, e.g. "data gloves"
• CPC Class Codes: A61B 2034/742 Joysticks
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/367 creating a 3D dataset from 2D images using position information
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 2090/373 using light, e.g. by using optical scanners
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/20 : Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 : User interfaces for surgical systems
• CPC Class Codes: A61B 34/30 : Surgical robots
• CPC Class Codes: A61B 34/37 : Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 34/74 : Manipulators with manual electric input means
• CPC Class Codes: A61B 90/36 : Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/361 : Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/37 : Surgical systems with images on a monitor during operation
• CPC Class Codes: B25J 9/1692 : Calibration of manipulator
• CPC Class Codes: B25J 9/1694 : characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: B25J 9/1697 : Vision controlled systems
• CPC Class Codes: G05B 2219/39449 : Pendant, pda displaying camera images overlayed with graphics, augmented reality
• CPC Class Codes: G05B 2219/40607 : Fixed camera to observe workspace, object, workpiece, global
• CPC Class Codes: G05B 2219/45123 : Electrogoniometer, neuronavigator, medical robot used by surgeon to operate

Abstract

An endoscope captures images of a surgical site for display in a viewing area of a monitor. When a tool is outside the viewing area, a GUI indicates the position of the tool by positioning a symbol in a boundary area around the viewing area so as to indicate the tool position. The distance of the out-of-view tool from the viewing area may be indicated by the size, color, brightness, or blinking or oscillation frequency of the symbol. A distance number may also be displayed on the symbol. The orientation of the shaft or end effector of the tool may be indicated by an orientation indicator superimposed over the symbol, or by the orientation of the symbol itself. When the tool is inside the viewing area, but occluded by an object, the GUI superimposes a ghost tool at its current position and orientation over the occluding object.

Claims

1-76. -76. (canceled)

77. A computer implemented method for indicating a position of a tool at a work site when the tool is out of a field of view of a camera, the method comprising:

capturing images of the work site using the camera;

displaying the captured images in a viewing area of a computer display screen;

determining a position of the tool in a reference frame of the camera;

determining a position to display a non-depictive symbol, that is non-depictive of the tool, in a boundary area circumscribing the viewing area on the computer display screen so as to indicate the determined position of the tool out of the field of view of the camera; and

displaying the non-depictive symbol at the determined position in the boundary area.

78. The computer implemented method according to claim 77, wherein the non-depictive symbol provides information identifying the tool.

79. The computer implemented method according to claim 77, wherein the non-depictive symbol provides information identifying a patient-side manipulator associated with the tool.

80. The computer implemented method according to claim 79, wherein the non-depictive symbol is a color that is associated with and indicated on the patient-side manipulator.

81. The computer implemented method according to claim 79, wherein the non-depictive symbol is marked with a number that is associated with and indicated on the patient-side manipulator.

82. The computer implemented method according to claim 77, wherein the position of the tool is determined using kinematics for a patient-side manipulator moving the tool.

83. The computer implemented method according to claim 77, wherein the tool includes a shaft having an axis extending along a length of the shaft, and the determination of the position of the non-depictive symbol in the boundary area comprises:

determining where a line passing through the axis of the shaft enters the field of view of the camera.

84. The computer implemented method according to claim 77, wherein the determination of the position of the non-depictive symbol in the boundary area comprises:

determining where a line extending from the position of the tool to a reference point in the field of view of the camera enters the field of view.

85. The computer implemented method according to claim 77, wherein the determination of the position of the non-depictive symbol in the boundary area comprises:

determining a trajectory of the tool from current and past positions of the tool, and determining where an extrapolation of the trajectory enters the field of view of the camera.

86. The computer implemented method according to claim 77, wherein the display of the non-depictive symbol in the boundary area comprises:

determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the camera, and sizing the non-depictive symbol so as to indicate the distance.

87. The computer implemented method according to claim 77, wherein the display of the non-depictive symbol in the boundary area comprises:

determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the camera, and displaying the non-depictive symbol so that its color indicates the distance.

88. The computer implemented method according to claim 87, wherein the intensity of the color of the non-depictive symbol indicates the distance.

89. The computer implemented method according to claim 87, wherein the position of the non-depictive symbol color in a color spectrum indicates the distance.

90. The computer implemented method according to claim 77, wherein the display of the non-depictive symbol in the boundary area comprises:

determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the camera, and displaying the non-depictive symbol so that a frequency of blinking of the non-depictive symbol indicates the distance.

91. The computer implemented method according to claim 77, wherein the display of the non-depictive symbol in the boundary area comprises:

determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the camera, and displaying the non-depictive symbol so that a frequency of oscillation of the non-depictive symbol about the determined position in the boundary area indicates the distance.

92. The computer implemented method according to claim 77, wherein the display of the non-depictive symbol in the boundary area comprises:

determining a distance of a position of a reference point on the tool from a position of a reference point in the field of view of the camera, and displaying the non-depictive symbol so that the distance is indicated by overlaying a distance number over the non-depictive symbol.

93. The computer implemented method according to claim 77,

wherein the tool includes an end effector,

wherein the determination of the position of the tool includes determining an orientation of the end effector in the reference frame of the camera, and

wherein the display of the non-depictive symbol in the boundary area comprises:

displaying an orientation indicator over the non-depictive symbol such that the orientation indicator is oriented so as to indicate the orientation of the end effector.

94. The computer implemented method according to claim 77,

wherein the tool includes a shaft having an axis extending along a length of the shaft,

wherein the determination of the position of the tool includes determining an orientation of the axis in the reference frame of the camera, and

wherein the display of the non-depictive symbol in the boundary area comprises:

displaying an orientation indicator over the non-depictive symbol such that the orientation indicator is oriented so as to indicate the orientation of the axis.