Method and apparatus for transforming coordinate systems in a telemanipulation system

• US 5,696,837 A
• Filed: 04/20/1995
• Issued: 12/09/1997
• Est. Priority Date: 05/05/1994
• Status: Expired due to Term

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Green, Philip S.
• Primary Examiner(s): Boudreau, Leo
• Assistant Examiner(s): KELLEY, CHRISTOPHER S
• Application Number:
  US08/425,965
• Time in Patent Office: 964 Days
• Field of Search: 382/128, 382/154, 348/42, 348/45, 348/46, 348/51, 348/69, 395/86, 395/89, 395/97, 395/119, 395/127, 364/474.36, 364/474, 901/47
• US Class Current: 382/128
• CPC Class Codes: A61B 17/00234 for minimally invasive surgery A61B17/0218, A61B17/0469, A61B17/12013, A61B17/1285, A61B17/29, A61B17/320016 take precedence
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/35 for telesurgery
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 34/70 Manipulators specially adapted for use in surgery
• CPC Class Codes: A61B 34/76 Manipulators having means for providing feel, e.g. force or tactile feedback
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 3/04 involving servo mechanisms servo-actuated heads B25J15/02
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G03C 1/0053 with high content of silver chloride
• CPC Class Codes: G05B 2219/35506 Camera images overlayed with graphics, model
• CPC Class Codes: G05B 2219/39389 Laparoscopic surgery, camera on center of operated part, view around, scale
• CPC Class Codes: G05B 2219/40158 Correlate actual image at angle with image presented to operator without angle
• CPC Class Codes: G05B 2219/40161 Visual display of machining, operation, remote viewing
• CPC Class Codes: G05B 2219/40169 Display of actual situation at the remote site
• CPC Class Codes: G05B 2219/45123 Electrogoniometer, neuronavigator, medical robot used by surgeon to operate
• CPC Class Codes: G06T 1/0014 Image feed-back for automatic industrial control, e.g. robot with camera robots B25J19/023
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 : Transmission of image signals
• CPC Class Codes: H04N 13/239 : using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 : Calibration of cameras
• CPC Class Codes: H04N 13/296 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/337 : using polarisation multiplexing
• CPC Class Codes: H04N 13/341 : using temporal multiplexing
• CPC Class Codes: H04N 13/398 : Synchronisation thereof; Control thereof

Abstract

In a telemanipulation system for manipulating objects located in a workspace at a remote worksite by an operator from an operator'"'"'s station, such as in a remote surgical system, the remote worksite having a manipulator with an end effector for manipulating an object at the workspace, such as a body cavity, a controller including a hand control at the control operator'"'"'s station for remote control of the manipulator, an image capture device, such as a camera, and image output device for reproducing a viewable real-time image, the improvement wherein a position sensor associated with the image capture device senses position relative to the end effector and a processor transforms the viewable real-time image into a perspective image with correlated manipulation of the end effector by the hand controller such that the operator can manipulate the end effector and the manipulator as if viewing the workspace in true presence. Image transformation according to the invention includes translation, rotation and perspective correction.

Claims

1. For use with a telemanipulation system for manipulating objects located in a workspace at a remote worksite by a control operator from a control operator station, said telemanipulation system including a manipulator means at the worksite with end effector means for manipulation of an object in the workspace, controller means including hand control means at the control operator station for remote control by the control operator of the manipulator means, image capture means for taking a real-time image of the workspace, and image producing means for reproducing a viewable real-time image on a monitor for eyes of the control operator, a method for dynamic calibration of the manipulator means through actual movement using a control loop wherein movements of the manipulator means directly follow movements of the hand control means, the method comprising the steps of:

a) superimposing a representation of a four-point coordinate graphic element in three-dimensional space on the viewable real-time image;

b) causing the hand control means to move to control motion of the end effector means;

thereupon

c) positioning the end effector means so that the end effector means appear to be touching a first reference point of said graphic element in the viewable real-time image;

d) indicating to the system that superposition of the end effector means and said first reference point has been achieved;

e) determining an offset by the relation σ

₃ =L'"'"'-L to adjust the extension L of the hand control means to match the extension L'"'"' of the manipulator means;

f) causing the hand control means to move about its pivot point to an angular orientation (Ψ

, Ω

), where the manipulator means in the viewable real-time image appears to emerge from the hand control means;

g) determining offsets according to the relationships σ

₁ =Ψ

-Ψ

'"'"' and σ

₂ =Ω

-Ω

'"'"' for adjusting the angular orientation (Ψ

, Ω

) of the hand control means to match the angular orientation (Ψ

'"'"', Ω

'"'"') of the manipulator means;

h) recording a control position (Ψ

, Ω

, L) in three-dimensional space of said hand control means;

i) recording a controlled position (Ψ

'"'"', Ω

'"'"', L'"'"') in three-dimensional space of said manipulator means corresponding to the control position; and

j) repeating steps a) through i) for each remaining reference point, thereby establishing calibration.

2. The method of claim 1 further comprising the step of synchronizing incremental changes in position of the hand control means to incremental changes in position of the manipulator means according to the following set of equations:

space="preserve" listing-type="equation">.increment.Ω

'"'"'≈

k.sub.11 .increment.Ω

+k.sub.12 .increment.Ψ

+k.sub.13 .increment.L

space="preserve" listing-type="equation">.increment.Ψ

'"'"'≈

k.sub.21 .increment.Ω

+k.sub.22 .increment.Ψ

+k.sub.23 .increment.L

space="preserve" listing-type="equation">.increment.L'"'"'≈

k.sub.31 .increment.Ω

+k.sub.32 .increment.Ψ

+k.sub.33 .increment.L

whose solution is as follows;

##EQU5##

3. For use with a telemanipulation system for manipulating objects located in a workspace at a remote worksite by a control operator from a control operator station, said telemanipulation system including a manipulator means at the worksite with end effector means for manipulation of an object in the workspace, controller means including hand control means at the control operator station for remote control by the control operator of the manipulator means, image capture means for taking a real-time image of the workspace, and image producing means for reproducing a viewable real-time image on a monitor for eyes of the control operator, a method for dynamic calibration of the manipulator means through virtual movement using a control loop where movements of the manipulator means directly follow movements of the hand control means, the method comprising the steps of:

a) causing the end effector means to move to a first reference point of a selection of four points distributed in three-dimensional space on the viewable real-time image;

b) manually moving the hand control means so that an overlaid graphic dot appears superimposed on the viewable image of the end effector means and said first reference point;

c) indicating to the system that superposition of the end effector means and said first reference point has been achieved;

d) calculating an offset σ

₃ =L'"'"'-L to adjust the extension L of the hand control means to match the extension L'"'"' of the manipulator means;

e) moving the hand control means about its pivot point to an angular orientation (Ψ

, Ω

), where the manipulator means in the viewable real-time image appears to emerge from the hand control means;

f) calculating offsets σ

₁ =Ψ

-Ψ

'"'"' and σ

₂ =Ω

-Ω

'"'"' to adjust the angular orientation (Ψ

, Ω

) of the hand control means to match the angular orientation (Ψ

'"'"', Ω

'"'"') of the manipulator means;

g) recording a position (Ψ

, Ω

, L) in three-dimensional space of said hand control means;

h) recording a position (Ψ

'"'"', Ω

'"'"', L'"'"') in three-dimensional space of said manipulator means; and

i) repeating steps a) through h) for each remaining reference point, thereby establishing calibration.

4. The method of claim 3 further comprising the step of relating incremental changes in position of the hand control means to incremental changes in position of the manipulator means according to the following set of equations:

space="preserve" listing-type="equation">.increment.Ω

'"'"'≈

k.sub.11 .increment.Ω

+k.sub.12 .increment.Ψ

+k.sub.13 .increment.L

space="preserve" listing-type="equation">.increment.Ψ

'"'"'≈

k.sub.21 .increment.Ω

+k.sub.22 .increment.Ψ

+k.sub.23 .increment.L

space="preserve" listing-type="equation">.increment.L'"'"'≈

k.sub.31 .increment.Ω

+k.sub.32 .increment.Ψ

+k.sub.33 .increment.L

whose solution is as follows;

##EQU6##