Method and apparatus for transforming coordinate systems in a telemanipulation system

• US 6,259,806 B1
• Filed: 10/15/1998
• Issued: 07/10/2001
• Est. Priority Date: 01/21/1992
• Status: Expired due to Term

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Green, Philip S.
• Primary Examiner(s): Bella, Matthew C.
• Application Number:
  US09/174,051
• Time in Patent Office: 999 Days
• Field of Search: 382/128, 382/311, 600/101, 600/103, 600/104, 600/109, 600/114, 600/118, 600/160
• US Class Current: 382/128
• CPC Class Codes: A61B 17/00234 for minimally invasive surgery A61B17/0218, A61B17/0469, A61B17/12013, A61B17/1285, A61B17/29, A61B17/320016 take precedence
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/35 for telesurgery
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 34/70 Manipulators specially adapted for use in surgery
• CPC Class Codes: A61B 34/76 Manipulators having means for providing feel, e.g. force or tactile feedback
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 3/04 involving servo mechanisms servo-actuated heads B25J15/02
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G03C 1/0053 with high content of silver chloride
• CPC Class Codes: G05B 2219/35506 Camera images overlayed with graphics, model
• CPC Class Codes: G05B 2219/39389 Laparoscopic surgery, camera on center of operated part, view around, scale
• CPC Class Codes: G05B 2219/40158 Correlate actual image at angle with image presented to operator without angle
• CPC Class Codes: G05B 2219/40161 Visual display of machining, operation, remote viewing
• CPC Class Codes: G05B 2219/40169 Display of actual situation at the remote site
• CPC Class Codes: G05B 2219/45123 Electrogoniometer, neuronavigator, medical robot used by surgeon to operate
• CPC Class Codes: G06T 1/0014 Image feed-back for automatic industrial control, e.g. robot with camera robots B25J19/023
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 : Transmission of image signals
• CPC Class Codes: H04N 13/239 : using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 : Calibration of cameras
• CPC Class Codes: H04N 13/296 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/337 : using polarisation multiplexing
• CPC Class Codes: H04N 13/341 : using temporal multiplexing
• CPC Class Codes: H04N 13/398 : Synchronisation thereof; Control thereof

Abstract

In a telemanipulation system for manipulating objects located in a workspace at a remote worksite by an operator from an operator'"'"'s station, such as in a remote surgical system, the remote worksite having a manipulator with an end effector for manipulating an object at the workspace, such as a body cavity, a controller including a hand control at the control operator'"'"'s station for remote control of the manipulator, an image capture device, such as a camera, and image output device for reproducing a viewable real-time image, the improvement wherein a position sensor associated with the image capture device senses position relative to the end effector and a processor transforms the viewable real-time image into a perspective image with correlated manipulation of the end effector by the hand controller such that the operator can manipulate the end effector and the manipulator as if viewing the workspace in true presence. Image transformation according to the invention includes translation, rotation and perspective correction.

Claims

1. A telesurgical system including,

a work station having an endoscope for capturing an image of a surgical site and at least one manipulator arranged to perform at least part of a surgical procedure at the surgical site;

a remote operator station having an image display device operatively associated with the endoscope to display the image captured by the endoscope and at least one hand control operatively associated with the manipulator to cause the manipulator to perform movements in response to inputs on the hand control, the hand control being positioned relative to the image display device to permit an operator to manipulate the hand control while viewing responsive movements of the manipulator in the image displayed on the image display device; and

a control system which determines the operative association between the hand control and the manipulator so as to cause the manipulator as viewed in the image displayed on the display device to appear as if it extends from the hand control so that in use, an operating environment is induced in which the viewed image of the manipulator and the hand control appear to be integral portions of a single hand-held instrument.

2. A telesurgical system as claimed in claim 1, which includes at least two remote controllable manipulators at the work station and at least two operatively associated hand controls at the operator station, each hand control being operatively associated with one of the manipulators to cause that manipulator to perform movements in response to inputs on its associated hand control, the control system determining the operative association between each hand control and its associated manipulator so as to cause the manipulators as viewed in the image displayed on the display device to appear as if they extend from the hand controls so that, in use, the viewed image of each manipulator and its associated hand control appears to be integral portions of a single hand-held instruments.

3. A telesurgical system as claimed in claim 2, wherein each hand control is mounted for movement about a pivot point and each manipulator is mounted for movement about a pivot point.

4. A telesurgical system as claimed in claim 2, wherein the endoscope is a stereo endoscope and the image display device is arranged to provide a corresponding stereo image.

5. A telesurgical system as claimed in claim 2, wherein the operative association between the hand control and the manipulator includes servo-control.

6. A telesurgical system as claimed in claim 3, which includes position sensors provided on the respective hand controls and manipulators, the position sensors being operatively connected in the control system to enable the control system to determine the positions of the hand controls and the manipulators, respectively, within at least one reference coordinate system.

7. A telesurgical system as claimed in claim 6, wherein the at least one reference coordinate system is a Cartesian reference coordinate system.

8. A telesurgical system as claimed in claim 2, which includes a magnification arrangement for magnifying the image captured by the endoscope.

9. A telesurgical system as claimed in claim 8, wherein the control system is arranged to cause the appearance that the displayed images of the manipulators appear to extend from their associated hand controls so that the operating environment in which the viewed image of each manipulator and its associated hand control appears to be integral portions of two separate hand-held instruments is again induced after the image is magnified.

10. A telesurgical system as claimed in claim 9, wherein the control system includes a processor arranged to compute a new position of each hand control relative to the image display device at which new positions the hand controls correspond to new positions of the manipulators as viewed in the magnified image displayed on the image display device.

11. A telesurgical system as claimed in claim 10, which includes a displacement arrangement for displacing the hand controls into their new positions.

12. A telesurgical system as claimed in claim 11, wherein the displacement arrangement is operatively connected in the control system to enable the control system to instruct the displacement arrangement to displace the hand controls into their new positions.

13. A telesurgical system as claimed in claim 12, wherein the control system is arranged to instruct the manipulators to be locked in their positions while the hand controls are caused to be displaced into their new positions.

14. A telesurgical system as claimed in claim 13, in which the processor computes offsets relating to the new positions of the hand controls relative to the positions of the manipulators and uses these offsets in determining the operative association between the hand controls and the manipulators thereafter.

15. A telesurgical system including

a work station having an endoscope for capturing an image of a surgical site and at least two manipulators arranged to perform at least part of a surgical procedure at the surgical site, each manipulator movable about a pivot point;

a remote operator station having an image display device operatively associated with the endoscope to display the image captured by the endoscope and at least two hand controls operatively associated with the manipulators, each hand control being mounted for movement about a pivot point and operatively associated with one of the manipulators to cause that manipulator to perform movements in response to inputs on its associated hand control, the hand control being positioned relative to the image display device to permit an operator to manipulate the hand control while viewing responsive movements of the manipulator in the image displayed on the image display device; and

a control system which determines the operative association between each hand control and its associated manipulator so as to cause the manipulators as viewed in the image displayed on the display device to appear as if they extend from the hand controls so that, in use, an operating environment is induced in which the viewed image of each manipulator and its associated hand control appear to be integral portions of two integrally connected hand-held instruments, position sensors being provided on the respective hand controls and manipulators, the position sensors being operatively connected in the control system to enable the control system to determine the positions of the hand controls and the manipulators, respectively, within at least one reference coordinate system, wherein the endoscope is movably mounted at the work station to enable the endoscope to be moved from one position into a new position relative to the surgical site.

16. A telesurgical system as claimed in claim 15, wherein the control system is arranged to cause the appearance that the viewed image of each manipulator and its associated hand control appears to be integral portions of two separate integrally connected hand-held instruments to be reestablished after the endoscope is moved into a new position.

17. A telesurgical system as claimed in claim 16, wherein the control system and manipulator effect movement of the endoscope about a pivot point.

18. A telesurgical system as claimed in claim 16, wherein the control system is arranged to cause the endoscope to displace angularly about a viewing axis thereof in reestablishing the appearance that the displayed images of the manipulators appear to extend from their associated hand controls so that the operating environment in which the viewed image of each manipulator and its associated hand control appears to be integral portions of a single hand-held instruments is again induced.

19. A telesurgical system as claimed in claim 15, wherein position sensors are provided at appropriate positions relative to the endoscope, the position sensors being operatively connected in the control system to enable the control system to determine the position of the endoscope within at least one reference coordinate system.

20. A telesurgical system as claimed in claim 19, wherein the reference coordinate system is a Cartesian reference coordinate system.

21. A telesurgical system as claimed in claim 18, wherein the control system includes a processor arranged to compute a new position of each hand control relative to the image display device at which new positions the hand controls correspond to new positions of the manipulators as viewed in the image displayed on the image display device.

22. A telesurgical system as claimed in claim 21, which includes a displacement arrangement for displacing the hand controls into their new positions.

23. A telesurgical system as claimed in claim 22, wherein the displacement arrangement is operatively connected in the control system to enable the control system to instruct the displacement arrangement to displace the hand controls into their new positions.

24. A telesurgical system as claimed in claim 22, in which the displacement arrangement is arranged to provide tactile feedback during the performance of a surgical procedure.

25. A telesurgical system as claimed in claim 23, wherein the control system is arranged to instruct the manipulators to be locked in position while the hand controls are caused to be displaced into their new positions.

26. A telesurgical system as claimed in claim 25, wherein the processor computes offsets relating to the new positions of the hand controls relative to the positions of the manipulators and uses these offsets in determining the operative association between the hand controls and the manipulators thereafter.

27. A telesurgical method comprising:

positioning a surgical end effector within an internal surgical site in a patient body, the surgical end effector operatively associated with a hand controller via a processor so that the processor effects movement of the end effector in response to movement of the hand controller;

capturing an image of the surgical end effector within the internal surgical site with an endoscope;

displaying the image of the end effector so that the end effector defines a displayed position relative to the hand controller;

moving the endoscope within the internal surgical site;

statically realigning the displayed position of the image of the end effector relative to the hand controller so as to compensate for the endoscope moving step by transforming the displayed position of the image of the end effector so that the end effector is superimposed on the hand controller in the view of a surgeon operating the hand controller.

28. A telesurgical system including,

a work station having an endoscope for capturing an image of a surgical site and at least one manipulator arranged to perform at least part of a surgical procedure at the surgical site;

a remote operator station having an image display device operatively associated with the endoscope to display the image captured by the endoscope and at least one hand control operatively associated with the manipulator to cause the manipulator to perform movements in response to inputs on the hand control, the hand control being positioned relative to the image display device to permit an operator to manipulate the hand control while viewing responsive movements of the manipulator in the image displayed on the image display device; and

a control system which determines the operative association between the hand control and the manipulator so as to cause the manipulator as viewed in the image displayed on the display device to appear as if it extends from the hand control so that in use, an operating environment is induced in which the viewed image of the manipulator and the hand control appear to be integral portions of a single integrally connected hand-held instrument, the endoscope movably mounted at the work station to enable the endoscope to be moved from one position into a new position relative to the surgical site, the control system arranged to reestablish the integral instrument appearance of the manipulator and the hand control after the endoscope is moved into a new position.