Method and apparatus for transforming coordinate systems in a telemanipulation system

• US 6,574,355 B2
• Filed: 03/21/2001
• Issued: 06/03/2003
• Est. Priority Date: 01/21/1992
• Status: Expired due to Fees

• Current Assignee: Intuitive Surgical Incorporated (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Incorporated (Intuitive Surgical Holdings, LLC)
• Inventors: Green, Philip S.
• Primary Examiner(s): Patel, Jayanti K.
• Assistant Examiner(s): Tabatabai, Abolfazl
• Application Number:
  US09/813,506
• Publication Number:
  US 20010046313A1
• Time in Patent Office: 804 Days
• Field of Search: 382/128, 382/129, 382/130, 382/131, 382/132, 382/133, 600/25, 600/160, 600/179, 600/191, 600/200, 600/224, 600/249, 600/101, 600/102, 600/103, 600/104, 600/105, 606/600, 607/46
• US Class Current: 382/128
• CPC Class Codes: A61B 17/00234 for minimally invasive surgery A61B17/0218, A61B17/0469, A61B17/12013, A61B17/1285, A61B17/29, A61B17/320016 take precedence
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/35 for telesurgery
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 34/70 Manipulators specially adapted for use in surgery
• CPC Class Codes: A61B 34/76 Manipulators having means for providing feel, e.g. force or tactile feedback
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 3/04 involving servo mechanisms servo-actuated heads B25J15/02
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G03C 1/0053 with high content of silver chloride
• CPC Class Codes: G05B 2219/35506 Camera images overlayed with graphics, model
• CPC Class Codes: G05B 2219/39389 Laparoscopic surgery, camera on center of operated part, view around, scale
• CPC Class Codes: G05B 2219/40158 Correlate actual image at angle with image presented to operator without angle
• CPC Class Codes: G05B 2219/40161 Visual display of machining, operation, remote viewing
• CPC Class Codes: G05B 2219/40169 Display of actual situation at the remote site
• CPC Class Codes: G05B 2219/45123 Electrogoniometer, neuronavigator, medical robot used by surgeon to operate
• CPC Class Codes: G06T 1/0014 Image feed-back for automatic industrial control, e.g. robot with camera robots B25J19/023
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 : Transmission of image signals
• CPC Class Codes: H04N 13/239 : using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 : Calibration of cameras
• CPC Class Codes: H04N 13/296 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/337 : using polarisation multiplexing
• CPC Class Codes: H04N 13/341 : using temporal multiplexing
• CPC Class Codes: H04N 13/398 : Synchronisation thereof; Control thereof

Abstract

In a telemanipulation system for manipulating objects located in a workspace at a remote worksite by an operator from an operator'"'"'s station, such as in a remote surgical system, the remote worksite having a manipulator with an end effector for manipulating an object at the workspace, such as a body cavity, a controller including a hand control at the control operator'"'"'s station for remote control of the manipulator, an image capture device, such as a camera, and image output device for reproducing a viewable real-time image, the improvement wherein a position sensor associated with the image capture device senses position relative to the end effector and a processor transforms the viewable real-time image into a perspective image with correlated manipulation of the end effector by the hand controller such that the operator can manipulate the end effector and the manipulator as if viewing the workspace in true presence. Image transformation according to the invention includes translation, rotation and perspective correction.

Claims

1. A telesurgical method comprising:

positioning a surgical end effector within an internal surgical site in a patient body, the surgical end effector operatively associated with a hand controller via a processor so that the processor effects a movement of the end effector in response to a movement of the hand controller;

capturing an image of the surgical end effector within the internal surgical site with an endoscope;

displaying the image of the end effector;

moving the endoscope within the internal surgical site;

dynamically realigning the movement of the image of the end effector relative to the movement of the hand controller by altering a coordinate transformation of the processor.

2. The telesurgical method of claim 1, wherein the dynamic realigning step is performed after the endoscope moving step.

3. The telesurgical method of claim 2, wherein the dynamically realigning step is performed so as to compensate for the endoscope moving step.

4. The telesurgical method of claim 1, wherein the dynamically realigning step aligns a direction of movement of the displayed image of the end effector with a direction of movement of the hand controller.

5. The telesurgical method of claim 1, further comprising statically realigning a displayed position of the image of the end effector relative to the hand controller.

6. A surgical robotic system comprising:

a master controller having an hand input device movable in an operator controller workspace;

a manipulator including a slave arm having a surgical end effector;

at least one driving servo operatively coupled to the end effector, the driving servo moving the end effector in a surgical workspace in response to slave control signals;

an imaging system including an image capture device with a field of view movable in the surgical workspace, the imaging system generating position signals indicating the field of view; and

a processor coupling the master controller to the slave arm, the processor generating the slave control signals by mapping the input device in the operator workspace with the end effector in the surgical workspace according to a transformation, the processor deriving the transformation in response to the position signals of the imaging system.

7. The surgical robotic system of claim 6, wherein the image capture device moves independently of the manipulator slave.

8. The surgical robotic system of claim 7, wherein the processor derives the transformation so that an image of the end effector in the display appears substantially connected to the input device in the controller workspace.

9. The surgical robotic system of claim 6, wherein the manipulator slave generates signals responsive to a position of the end effector in the surgical workspace, the processor deriving the transformation using the signals of the slave.

10. The surgical robotic system of claim 9,

wherein the processor determines a position and orientation of the input device in the master operator space from position signals of the master controller,

wherein the processor determines a position and orientation of the end effector in the surgical workspace from the position signals of the slave, and

wherein the processor generates the slave control signals by comparing the position and orientation of the input device and the end effector in the mapped space.

11. The surgical robotic system of claim 6, wherein the processor derives the transformation in real time.

12. A surgical robotic system comprising:

a master controller having an hand control input device movable in a operator controller workspace;

a slave manipulator having a surgical end effector and at least one driving servo operatively coupled to the end effector, the driving servo moving the end effector in a surgical workspace in response to slave control signals;

an imaging system including an image capture device with a field of view movable in the surgical workspace, the imaging system transmitting an image to a display; and

a processor coupling the master controller to the slave arm,

the processor generating the slave control signals by mapping the input device in the controller workspace with the end effector in the surgical workspace according to a transformation,

the processor deriving the transformation so that an image of the end effector in the display appears substantially connected to the input device in the workspace.

13. The surgical robotic system of claim 12, wherein points of articulation of the master controller and points of articulation of the slave manipulator define different locations in the mapped space.

14. The surgical robotic system of claim 12, wherein:

the slave comprises an articulated manipulator including multiple points of motion supporting the end effector, the driving servos coupled to the points of motion so as to pivot a rigid shaft of the slave about a first arbitrary entry port for surgical access to a body cavity and move the end effector in response to slave control signals; and

the imaging system comprises driving servos and an articulated assembly including multiple points of motion supporting the image capture device, the driving servos coupled to the articulated assembly to move the field of view within the surgical workspace when an endoscope of the image capture device extends through a second arbitrary entry port for surgical access to the body cavity.

15. The surgical robotic system of claim 12, wherein the processor calculates the transformation in response to a signal indicating at least one member of the group consisting of:

a movement of the camera,

a decoupling and repositioning of one of the master and the slave relative to the other,

a change in scale of the mapping, and

manual movement of the master or slave.

16. The surgical robotic system of claim 12, wherein the slave senses non-visual sensory information at the surgical workspace, the master controller presenting the non-visual information to a surgeon manipulating the input device.

17. The surgical robotic system of claim 16, wherein the non-visual sensory information includes tactile feedback from at least one manipulator to at least one hand control.

18. The surgical robotic system of claim 12, wherein the master controller presents non-visual sensory information to a surgeon manipulating the input device.

19. The surgical robotic system of claim 18, wherein the master controller provides tactile feedback to the surgeon manipulating the input device.

20. A surgical robotic method comprising:

moving a master input device in a controller workspace by moving a plurality of points of articulation of the master;

moving a surgical end effector in a surgical workspace by moving a plurality of points of articulation of the slave in response to slave control signals;

displaying, on a display, an image of an arbitrary field of view within the surgical workspace adjacent the master controller; and

automatically generating the slave control signals in response to moving the master so that an image of the end effector shown in the display appears substantially connected with the input device in the master controller space.

21. A Surgical robotic system comprising:

a hand controller having a hand control input device in a operator controller workspace;

a surgical end effector;

an image capturing device with a field of view that captures an image of the end effector;

a display displaying the image of the end effector in a surgical workspace;

a processor generating a control signal coupling the hand controller and the end effector, the control signal related to a coordinate transformation;

the processor deriving the coordinate transformation so that the image of the end effector in the display appears substantially related to the input device in the workspace; and

the processor realigning the image of the end effector relative to the hand controller by altering the coordinate transformation.