Method for telemanipulation with telepresence

• US 5,631,973 A
• Filed: 05/05/1994
• Issued: 05/20/1997
• Est. Priority Date: 05/05/1994
• Status: Expired due to Term

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Green, Philip S.
• Primary Examiner(s): Boudreau, Leo
• Assistant Examiner(s): KELLEY, CHRISTOPHER S
• Application Number:
  US08/239,086
• Time in Patent Office: 1,111 Days
• Field of Search: 382/128, 382/293, 382/296, 382/276, 348/65-78, 128/4, 128/6, 128/10, 395/95, 395/924, 395/137, 395/138
• US Class Current: 382/128
• CPC Class Codes: A61B 17/00234 for minimally invasive surgery A61B17/0218, A61B17/0469, A61B17/12013, A61B17/1285, A61B17/29, A61B17/320016 take precedence
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/35 for telesurgery
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 34/70 Manipulators specially adapted for use in surgery
• CPC Class Codes: A61B 34/76 Manipulators having means for providing feel, e.g. force or tactile feedback
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 3/04 involving servo mechanisms servo-actuated heads B25J15/02
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G03C 1/0053 with high content of silver chloride
• CPC Class Codes: G05B 2219/35506 Camera images overlayed with graphics, model
• CPC Class Codes: G05B 2219/39389 Laparoscopic surgery, camera on center of operated part, view around, scale
• CPC Class Codes: G05B 2219/40158 Correlate actual image at angle with image presented to operator without angle
• CPC Class Codes: G05B 2219/40161 Visual display of machining, operation, remote viewing
• CPC Class Codes: G05B 2219/40169 Display of actual situation at the remote site
• CPC Class Codes: G05B 2219/45123 Electrogoniometer, neuronavigator, medical robot used by surgeon to operate
• CPC Class Codes: G06T 1/0014 Image feed-back for automatic industrial control, e.g. robot with camera robots B25J19/023
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 : Transmission of image signals
• CPC Class Codes: H04N 13/239 : using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 : Calibration of cameras
• CPC Class Codes: H04N 13/296 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/337 : using polarisation multiplexing
• CPC Class Codes: H04N 13/341 : using temporal multiplexing
• CPC Class Codes: H04N 13/398 : Synchronisation thereof; Control thereof

Abstract

In a telemanipulation system for manipulating objects located in a workspace at a remote worksite by an operator from an operator'"'"'s station, such as in a remote surgical system, the remote worksite having a manipulator with an end effector for manipulating an object at the workspace, such as a body cavity, a controller including a hand control at the control operator'"'"'s station for remote control of the manipulator, an image capture device, such as a camera, and image output device for reproducing a viewable real-time image, the improvement wherein a position sensor associated with the image capture device senses position relative to the end effector and a processor transforms the viewable real-time image into a perspective image with correlated manipulation of the end effector by the hand controller such that the operator can manipulate the end effector and the manipulator as if viewing the workspace in true presence. Image transformation according to the invention includes translation, rotation and perspective correction.

Claims

1. In a telemanipulation system for manipulating objects located in a workspace at a remote worksite by a control operator from a control operator station, said telemanipulation system including a manipulator at the worksite with an end effector for manipulation of an object in the workspace, a controller for remote control by the control operator of the manipulator, an image capture device for taking real-time image of the workspace, and an image generator for producing a display for eyes of the control operator, wherein the image capture device is at an angle and a position relative to the end effector which differs from a relative position of eyes of the control operator and the controller, the system further comprising:

an image processing device coupled to the image generator and the image capture device, the image processing device processing the real-time image into a processed image for display to the control operator, wherein the processed image is substantially the same as an image taken by an image capture device located at an angle and position relative to the end effector which is substantially equivalent to a relative angle and position of eyes of the control operator and the controller;

a servo control for controlling position of said manipulator;

a first position sensor disposed to sense a position of said manipulator in said workspace;

a second position sensor disposed to sense a position of said hand control at the operator station; and

a coordinate transformation device coupled to said image processing device, to said first position sensor and to said second position sensor for transforming information about manipulator position, hand control position and processed image position and into control signals applied to said servo control.

2. The system according to claim 1 wherein said image processing device further comprises:

means for converting an actual position of the end effector in the workspace to a perceived position of the end effector in the viewable real-time image, the image capture device being monoscopic and being positioned at an angle θ

on a reference plane, said image capture device being above the reference plane in a reference space, and wherein the eyes of the control operator are positioned at an angle 0°

relative to a base line on said reference plane, said eyes being above said reference plane in said reference space, the viewable real-time image being rotated pixel-by-pixel by an angle θ

'"'"', which is approximately equal to -θ

.

3. The system of claim 2, wherein coordinates of said perceived position of the end effector are dynamically related to coordinates of said actual position of the end effector by the following relationship:

##EQU5## where [p,q,r] are coordinates of said perceived position,

[a,b,c] are coordinates of said actual position,

angle θ

is the rotation of the image capture means in a horizontal plane about a vertical axis,

angle θ

'"'"' being substantially equal to -θ

, and

angle Φ

is the angle of declination below said vertical axis of an axis normal to a surface of the image capture means.

4. The system according to claim 1 wherein said image procesing device further comprises:

means for static realignment of the manipulator effected by rotating the image capture device about an axis normal to a surface of the image capture device to a position where the manipulator appears in a viewable real-time image as presented to the control operator, where the controller are below a viewpoint of the control operator.

5. The system according to claim 1 wherein said image processing device further comprises:

means for static realignment of the manipulator means effected by rotating the image capture device about an axis normal to a surface of the image capture device to a position where the manipulator means appear in the viewable real-time image as presented to the control operator, where the controller are above a viewpoint of the control operator.

6. The system according to claim 4 wherein the image capture device comprises:

a monoscopic camera, wherein rotation of the image capture means is about a visual axis of said monoscopic camera through a center of a lens of said monoscopic camera.

7. The system according to claim 4 wherein the image capture device comprises:

a stereoscopic dual aperture camera with a normal axis bisecting a line between each aperture of said stereoscopic camera, wherein rotation of the image capture device is about a visual axis of said stereoscopic camera through the center of said normal axis.

8. The system according to claim 4 wherein said image processing device further comprises:

means for translating an actual position of the end effector in the workspace to a perceived position of the end effector in the viewable real-time image such that motion of the end effector means appears rigidly connected to the controller.

9. The system according to claim 8 wherein the translating of said actual position to said perceived position is according to the following relationship:

##EQU6## where [p,q,r] are coordinates of said perceived position,

[a,b,c] are coordinates of said actual position,

angle Γ

is the rotation of the image capture device about an axis normal to a surface of the image capture device,

angle Φ

is the angle of declination of said normal axis below a vertical axis, and

angle θ

is the rotation of the image capture device in a horizontal plane about said vertical axis.

10. The system of claim 1 wherein the image processing device further comprising means for transforming coordinates of the image capture device to coordinates near the end effectors such that the processed image provides the operator a sense that the controller is substantially integral with the end effectors.

11. The system of claim 1 further comprising a third position sensor disposed to sense a position of the image capture device, wherein the coordinate transformation device is coupled to the third position sensor for transforming position information about the image capture device into control signals.

12. In a telemanipulation system for manipulating objects located in a workspace at a remote worksite by a control operator from a control operator station, said telemanipulation system including a manipulator means at the remote worksite with end effector means for manipulation of an object in the workspace, controller means including hand control means at the control operator station for remote control by the control operator of the manipulator means, image capture means for taking a real-time image of the workspace, and image producing means for producing a display on a monitor for eyes of the control operator, control means comprising:

means coupled to said image capture means for processing the real-time into a processed image for display to the control operator wherein said image capture means is at an angle and a position relative to the end effector means which differs from a relative position of eyes of the control operator and the hand control means, and which further differs in scale and in lateral position, said image processing means correlating manipulation of said end effector means with said hand control means, said image processing means further comprising;

means for performing a perspective correction in the viewable real-time image following a lateral shift and a scale change of the image capture means, wherein said perspective correction means comprises means for performing transformations according to the following relationships;

space="preserve" listing-type="equation">α

'"'"'=arctan [(x'"'"'-m)/(y'"'"'-n)],

space="preserve" listing-type="equation">L'"'"'=[(x'"'"'-m).sup.2 +(y'"'"'-n).sup.2 ].sup.1/2,

space="preserve" listing-type="equation">x'"'"'=M(x-Δ

x),

space="preserve" listing-type="equation">y'"'"'=M(y-Δ

y),

space="preserve" listing-type="equation">x=L(sin α

)+m,

space="preserve" listing-type="equation">y=L(cos α

)+n

where

coordinates (m,n) describe a position of a pivot point of the manipulator means before said lateral shift and said scale change occur,

coordinates (m'"'"',n'"'"') describe a position of said pivot point of the manipulator means after said lateral shift and said scale change occur,

coordinates (x,y) describe a position of the end effector means before said lateral shift and said scale change occur,

coordinates (x'"'"',y'"'"') describe a position of the end effector means after said lateral shift and said scale change occur,

coordinates (Δ

x,Δ

y) describe the change in position of the end effector means brought about by said lateral shift,

factor M describes the magnification factor of said scale change,

angle α

is the rotation of the manipulator means about a vertical axis before said lateral shift and said scale change occur,

angle α

'"'"' is the rotation of the manipulator means about said vertical axis after said lateral shift and said scale change occur,

length L is a distance between said pivot point and the end effector means before said lateral shift and said scale change occur, and

length L'"'"' is a distance between said pivot point and the end effector means after said lateral shift and said scale change occur.