Tool position and identification indicator displayed in a boundary area of a computer display screen

• US 20080004603A1
• Filed: 06/29/2006
• Published: 01/03/2008
• Est. Priority Date: 06/29/2006
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Incorporated (Intuitive Surgical Holdings, LLC)
• Inventors: Nixon, Thomas Robert, Larkin, David Q., Mintz, David Stephen
• Granted Patent: US 9,718,190 B2
• Time in Patent Office: Days
• US Class Current: 606/1
• CPC Class Codes: A61B 1/00009 of image signals during a use of endoscope
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: A61B 1/0051 with controlled bending of insertion part
• CPC Class Codes: A61B 1/3132 for laparoscopy
• CPC Class Codes: A61B 17/00234 for minimally invasive surgery A61B17/0218, A61B17/0469, A61B17/12013, A61B17/1285, A61B17/29, A61B17/320016 take precedence
• CPC Class Codes: A61B 2017/00199 with a console, e.g. a control panel with a display
• CPC Class Codes: A61B 2017/00292 mounted on or guided by flexible, e.g. catheter-like, means
• CPC Class Codes: A61B 2017/00296 mounted on an endoscope
• CPC Class Codes: A61B 2034/102 Modelling of surgical devices, implants or prosthesis
• CPC Class Codes: A61B 2034/107 Visualisation of planned trajectories or target regions
• CPC Class Codes: A61B 2034/2057 Details of tracking cameras
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2034/301 for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes
• CPC Class Codes: A61B 2034/741 Glove like input devices, e.g. "data gloves"
• CPC Class Codes: A61B 2034/742 Joysticks
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/367 creating a 3D dataset from 2D images using position information
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 2090/373 using light, e.g. by using optical scanners
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/20 : Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 : User interfaces for surgical systems
• CPC Class Codes: A61B 34/30 : Surgical robots
• CPC Class Codes: A61B 34/37 : Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 34/74 : Manipulators with manual electric input means
• CPC Class Codes: A61B 90/36 : Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/361 : Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/37 : Surgical systems with images on a monitor during operation
• CPC Class Codes: B25J 9/1692 : Calibration of manipulator
• CPC Class Codes: B25J 9/1694 : characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: B25J 9/1697 : Vision controlled systems
• CPC Class Codes: G05B 2219/39449 : Pendant, pda displaying camera images overlayed with graphics, augmented reality
• CPC Class Codes: G05B 2219/40607 : Fixed camera to observe workspace, object, workpiece, global
• CPC Class Codes: G05B 2219/45123 : Electrogoniometer, neuronavigator, medical robot used by surgeon to operate

Abstract

An endoscope captures images of a surgical site for display in a viewing area of a monitor. When a tool is outside the viewing area, a GUI indicates the position of the tool by positioning a symbol in a boundary area around the viewing area so as to indicate the tool position. The distance of the out-of-view tool from the viewing area may be indicated by the size, color, brightness, or blinking or oscillation frequency of the symbol. A distance number may also be displayed on the symbol. The orientation of the shaft or end effector of the tool may be indicated by an orientation indicator superimposed over the symbol, or by the orientation of the symbol itself. When the tool is inside the viewing area, but occluded by an object, the GUI superimposes a ghost tool at its current position and orientation over the occluding object.

Claims

1. A computer implemented method for indicating a position of a tool on a computer display screen, comprising:

determining a position of the tool;

determining a position of a symbol in a boundary area around an image displayed on the computer display screen so as to indicate the position of the tool; and

displaying the symbol at the determined position in the boundary area.

2. The computer implemented method according to claim 1, wherein the symbol provides information identifying the tool.

3. The computer implemented method according to claim 1, wherein the symbol provides information identifying a patient-side manipulator associated with the tool.

4. The computer implemented method according to claim 3, wherein the symbol is a color that is associated with and indicated on the patient-side manipulator.

5. The computer implemented method according to claim 3, wherein the symbol is marked with a number that is associated with and indicated on the patient-side manipulator.

6. The computer implemented method according to claim 1, wherein the position of the tool is determined in a reference frame of an imaging device providing the image being displayed on the computer display screen.

7. The computer implemented method according to claim 6, wherein the position of the tool is determined using kinematics for a patient-side manipulator moving the tool.

8. The computer implemented method according to claim 6, wherein the position of the tool is determined using an image identification technique.

9. The computer implemented method according to claim 1, wherein the tool includes a shaft having an axis extending along a length of the shaft, and the determination of the position of the symbol in the boundary area comprises:

determining where a line passing through the axis of the shaft intersects the boundary area.

10. The computer implemented method according to claim 1, further comprising:

determining whether the position of the tool is within a viewing area including the image being displayed on the computer display screen.

11. The computer implemented method according to claim 10, wherein if the position of the tool is outside the viewing area, then the determination of the position of the symbol in the boundary area comprises:

determining where a line extending from the position of the tool to a reference point in the viewing area intersects the boundary area.

12. The computer implemented method according to claim 10, wherein if the position of the tool is outside the viewing area, then the determination of the position of the symbol in the boundary area comprises:

determining a trajectory of the tool from current and past positions of the tool, and determining where an extrapolation of the trajectory intersects the boundary area.

13. The computer implemented method according to claim 10, wherein if the position of the tool is outside the viewing area, then the display of the symbol in the boundary area comprises:

determining a distance of a position of a reference point on the tool from a position of a reference point in the viewing area, and sizing the symbol so as to indicate the distance.

14. The computer implemented method according to claim 10, wherein if the position of the tool is outside the viewing area, then the display of the symbol in the boundary area comprises:

determining a distance of a position of a reference point on the tool from a position of a reference point in the viewing area, and displaying the symbol so that its color indicates the distance.

15. The computer implemented method according to claim 14, wherein the intensity of the color of the symbol indicates the distance.

16. The computer implemented method according to claim 14, wherein the position of the symbol color in a color spectrum indicates the distance.

17. The computer implemented method according to claim 10, wherein if the position of the tool is outside the viewing area, then the display of the symbol in the boundary area comprises:

determining a distance of a position of a reference point on the tool from a position of a reference point in the viewing area, and displaying the symbol so that a frequency of blinking of the symbol indicates the distance.

18. The computer implemented method according to claim 10, wherein if the position of the tool is outside the viewing area, then the display of the symbol in the boundary area comprises:

determining a distance of a position of a reference point on the tool from a position of a reference point in the viewing area, and displaying the symbol so that a frequency of oscillation of the symbol about the determined position in the boundary area indicates the distance.

19. The computer implemented method according to claim 10, wherein if the position of the tool is outside the viewing area, then the display of the symbol in the boundary area comprises:

determining a distance of a position of a reference point on the tool from a position of a reference point in the viewing area, and displaying the symbol so that the distance is indicated by overlaying a distance number over the symbol.

20. The computer implemented method according to claim 10, wherein if the position of the tool is outside the viewing area, then the determination of the position of the tool includes determining an orientation of an end effector of the tool, and the display of the symbol in the boundary area comprises:

displaying an orientation indicator over the symbol such that the orientation indicator is oriented so as to indicate the orientation of the end effector.

21. The computer implemented method according to claim 10, wherein the tool includes a shaft having an axis extending along a length of the shaft, and if the position of the tool is outside the viewing area, then the determination of the position of the tool includes determining an orientation of the axis, and the display of the symbol in the boundary area comprises:

displaying an orientation indicator over the symbol such that the orientation indicator is oriented so as to indicate the orientation of the axis.

22. The computer implemented method according to claim 10, wherein if the position of the tool is outside the viewing area, then the determination of the position of the tool includes determining an orientation of an end effector of the tool, and the display of the symbol in the boundary area comprises:

displaying the symbol so that the symbol is oriented so as to indicate the orientation of the end effector.

23. The computer implemented method according to claim 10, further comprising:

if the tool is in the viewing area, then identifying an image of the tool in the viewing area.

24. The computer implemented method according to claim 23, wherein if the image of the tool cannot be identified in the viewing area, then displaying a ghost tool in the viewing area where the tool should be positioned.

25. The computer implemented method according to claim 23, wherein the viewing area includes stereoscopic right and left two-dimensional views, and the identification of the image of the tool comprises:

generating a three-dimensional computer model of the tool;

positioning and orientating the computer model to coincide with the current position and orientation of the tool in the reference frame of the computer display screen;

generating a two-dimensional outline of the computer model that projects onto a selected one of the right and left two-dimensional views; and

cross-correlating the two-dimensional outline of the computer model with the selected one of the right and left two-dimensional views.

26. The computer implemented method according to claim 24, further comprising:

if the image of the tool is not identified in the selected one of the right and left two-dimensional views, then displaying at least the two-dimensional outline of the computer model of the tool so as to be displayed as an overlay on the selected one of the right and left two-dimensional views as it is being displayed on the computer screen.

27. The computer implemented method according to claim 24, further comprising:

if the image of the tool is not identified in the selected one of the right and left two-dimensional views, then overlaying right and left two-dimensional ghost images of the tool in corresponding of the right and left two-dimensional views, so as to appear as a three-dimensional ghost image of the tool at the position and orientation of the tool in the reference frame of the computer display screen.

28. An apparatus for indicating a position of a tool on a computer display screen, comprising a processor coupled to the computer display screen and configured to:

determine a current position of the tool;

determine a position of a symbol in a boundary area around an image displayed on the computer display screen so as to indicate the position of the tool; and

cause the symbol to be displayed at the determined position in the boundary area.

29. The apparatus according to claim 28, wherein the symbol provides information identifying the tool.

30. The apparatus according to claim 28, wherein the symbol provides information identifying a patient-side manipulator associated with the tool.

31. The apparatus according to claim 30, wherein the symbol is a color that is associated with and indicated on the patient-side manipulator.

32. The apparatus according to claim 30, wherein the symbol is marked with a number that is associated with and indicated on the patient-side manipulator.

33. The apparatus according to claim 28, wherein the position of the tool is determined in a reference frame of an imaging device providing the image being displayed on the computer display screen.

34. The apparatus according to claim 33, wherein the position of the tool is determined using kinematics for a patient-side manipulator moving the tool.

35. The apparatus according to claim 33, wherein the position of the tool is determined using an image identification technique.

36. The apparatus according to claim 28, wherein the tool includes a shaft having an axis extending along a length of the shaft, and the processor determines the position of the symbol in the boundary area by determining where a line passing through the axis of the shaft intersects the boundary area.

37. The apparatus according to claim 28, wherein the processor is further configured to determine whether the position of the tool is within a viewing area including the image being displayed on the computer display screen.

38. The apparatus according to claim 37, wherein if the position of the tool is outside the viewing area, then the processor is configured to determine the position of the symbol in the boundary area by determining where a line extending from the position of the tool to a reference point in the viewing area intersects the boundary area.

39. The apparatus according to claim 37, wherein if the position of the tool is outside the viewing area, then the processor is configured to determine the position of the symbol in the boundary area by determining a trajectory of the tool from current and past positions of the tool, and determining where an extrapolation of the trajectory intersects the boundary area.

40. The apparatus according to claim 37, wherein if the position of the tool is outside the viewing area, then the processor is configured to display the symbol in the boundary area by determining a distance of a position of a reference point on the tool from a position of a reference point in the viewing area, and sizing the symbol so as to indicate the distance.

41. The apparatus according to claim 37, wherein if the position of the tool is outside the viewing area, then the processor is configured to display the symbol in the boundary area by determining a distance of a position of a reference point on the tool from a position of a reference point in the viewing area, and displaying the symbol so that its color indicates the distance.

42. The apparatus according to claim 41, wherein the intensity of the color of the symbol indicates the distance.

43. The apparatus according to claim 41, wherein the position of the symbol color in a color spectrum indicates the distance.

44. The apparatus according to claim 37, wherein if the position of the tool is outside the viewing area, then the processor is configured to display the symbol in the boundary area by determining a distance of a position of a reference point on the tool from a position of a reference point in the viewing area, and a frequency of blinking of the symbol indicates the distance.

45. The apparatus according to claim 37, wherein if the position of the tool is outside the viewing area, then the processor is configured to display the symbol in the boundary area by determining a distance of a position of a reference point on the tool from a position of a reference point in the viewing area, and a frequency of oscillation of the symbol about the determined position in the boundary area indicates the distance.

46. The apparatus according to claim 37, wherein if the position of the tool is outside the viewing area, then the processor is configured to display the symbol in the boundary area by determining a distance of a position of a reference point on the tool from a position of a reference point in the viewing area, and the distance is indicated by overlaying a distance number over the symbol.

47. The apparatus according to claim 37, wherein if the position of the tool is outside the viewing area, then the processor is configured to determine the position and orientation of an end effector of the tool, and display an orientation indicator over the symbol such that the orientation indicator is oriented so as to indicate the orientation of the end effector.

48. The apparatus according to claim 37, wherein the tool includes a shaft having an axis extending along a length of the shaft, and if the position of the tool is outside the viewing area, then the processor is configured to determine an orientation of the axis, and display an orientation indicator over the symbol such that the orientation indicator is oriented so as to indicate the orientation of the axis.

49. The apparatus according to claim 37, wherein if the position of the tool is outside the viewing area, then the processor is configured to determine the position and orientation of an end effector of the tool, and display the symbol such that the symbol is oriented so as to indicate the orientation of the end effector.

50. The apparatus according to claim 37, wherein the processor is further configured to identify an image of the tool in the viewing area, if the tool is within the viewing area being displayed on the computer display screen.

51. The apparatus according to claim 50, wherein if the image of the tool cannot be identified in the viewing area, then the processor is further configured to display a ghost tool in the viewing area where the tool should be positioned.

52. The apparatus according to claim 50, wherein the viewing area includes stereoscopic right and left two-dimensional views, and the processor is configured to identify the image of the tool by:

generating a three-dimensional computer model of the tool;

positioning and orientating the computer model to coincide with the current position and orientation of the tool in the reference frame of the computer display screen;

generating a two-dimensional outline of the computer model that projects onto a selected one of the right and left two-dimensional views; and

cross-correlating the two-dimensional outline of the computer model with the selected one of the right and left two-dimensional views.

53. The processor according to claim 52, the processor is further configured to cause at least the two-dimensional outline of the computer model of the tool to be displayed as an overlay on the selected one of the right and left two-dimensional views as it is being displayed on the computer screen, if the image of the tool is not identified in the selected one of the right and left two-dimensional views.

54. The apparatus according to claim 52, wherein the processor is further configured to cause right and left two-dimensional ghost images of the tool to be overlayed in corresponding of the right and left two-dimensional views, so as to appear as a three-dimensional ghost image of the tool at the position and orientation of the tool in the reference frame of the computer display screen, if the image of the tool is not identified in the selected one of the right and left two-dimensional views.

55. A medical robotic system comprising:

a surgical tool;

a robotic arm mechanically coupled to the surgical tool so as to be capable of positioning and orienting the surgical tool;

an imaging device;

a computer display screen; and

a processor coupled to the robotic arm, imaging device, and computer display screen, wherein the processor is configured to cause images captured by the imaging device to be displayed in a viewing area of the computer display screen and a symbol to be displayed in a boundary area around the viewing area so as to indicate a location of the tool.

56. The system according to claim 55, wherein the symbol provides information identifying the tool.

57. The system according to claim 55, wherein the symbol provides information identifying a patient-side manipulator associated with the tool.

58. The system according to claim 57, wherein the symbol is a color that is associated with and indicated on the patient-side manipulator.

59. The system according to claim 57, wherein the symbol is marked with a number that is associated with and indicated on the patient-side manipulator.

60. The system according to claim 55, wherein the position of the tool is determined in a reference frame of the imaging device.

61. The system according to claim 60, wherein the position of the tool is determined using kinematics for a patient-side manipulator moving the tool.

62. The system according to claim 60, wherein the position of the tool is determined using an image identification technique.

63. The system according to claim 55, wherein the tool includes a shaft having an axis extending along a length of the shaft, and the processor is configured to determine the position of the symbol in the boundary area by determining where a line passing through the axis of the shaft intersects the boundary area.

64. The system according to claim 55, wherein the processor is further to determine whether the position of the tool is within the viewing area being displayed on the computer display screen.

65. The system according to claim 55, wherein the processor is further configured to cause the symbol to be displayed in the boundary of the computer display screen so as to indicate an orientation of an end effector of the tool when the tool is outside of the viewing area.

66. The system according to claim 55, wherein the processor is further configured to cause the symbol to be displayed in the boundary of the computer display screen so as to indicate a distance from a position of a reference point on the tool to a reference point in the viewing area when the tool is outside of the viewing area.

67. The system according to claim 66, wherein the distance is indicated by a size of the displayed symbol.

68. The system according to claim 66, wherein the distance is indicated by a color of the displayed symbol.

69. The system according to claim 66, wherein the distance is indicated by brightness of the displayed symbol.

70. The system according to claim 66, wherein the distance is indicated by a frequency of blinking of the displayed symbol.

71. The system according to claim 70, wherein the frequency of blinking increases as the distance decreases.

72. The system according to claim 66, wherein a frequency of oscillation of the symbol about the determined position in the boundary area indicates the distance.

73. The system according to claim 66, wherein the distance is indicated by overlaying a distance number over the symbol.

74. The system according to claim 55, wherein the imaging device includes at least one camera.

75. The system according to claim 55, wherein the imaging device captures MRI images.

76. The system according to claim 55, wherein the imaging device captures ultrasound images.