Method and apparatus for transforming coordinate systems in a telemanipulation system

• US 8,526,737 B2
• Filed: 10/25/2011
• Issued: 09/03/2013
• Est. Priority Date: 05/05/1994
• Status: Expired due to Fees

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Green, Philip S.
• Primary Examiner(s): Tabatabai, Abolfazl
• Application Number:
  US13/281,342
• Publication Number:
  US 20130023899A1
• Time in Patent Office: 679 Days
• Field of Search: 382/128, 382/129, 382/130, 382/131, 382/132, 382/133, 382/134, 600/101, 600/102, 600/103, 600/104, 600/105, 600/118, 600/126, 600/132, 600/137, 600/407, 600/410, 600/425, 600/427, 128/920, 128/922, 606/130, 348/45, 901/2, 901/47
• US Class Current: 382/180
• CPC Class Codes: A61B 17/00234 for minimally invasive surgery A61B17/0218, A61B17/0469, A61B17/12013, A61B17/1285, A61B17/29, A61B17/320016 take precedence
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/35 for telesurgery
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 34/70 Manipulators specially adapted for use in surgery
• CPC Class Codes: A61B 34/76 Manipulators having means for providing feel, e.g. force or tactile feedback
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 3/04 involving servo mechanisms servo-actuated heads B25J15/02
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G03C 1/0053 with high content of silver chloride
• CPC Class Codes: G05B 2219/35506 Camera images overlayed with graphics, model
• CPC Class Codes: G05B 2219/39389 Laparoscopic surgery, camera on center of operated part, view around, scale
• CPC Class Codes: G05B 2219/40158 Correlate actual image at angle with image presented to operator without angle
• CPC Class Codes: G05B 2219/40161 Visual display of machining, operation, remote viewing
• CPC Class Codes: G05B 2219/40169 Display of actual situation at the remote site
• CPC Class Codes: G05B 2219/45123 Electrogoniometer, neuronavigator, medical robot used by surgeon to operate
• CPC Class Codes: G06T 1/0014 Image feed-back for automatic industrial control, e.g. robot with camera robots B25J19/023
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 : Transmission of image signals
• CPC Class Codes: H04N 13/239 : using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 : Calibration of cameras
• CPC Class Codes: H04N 13/296 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/337 : using polarisation multiplexing
• CPC Class Codes: H04N 13/341 : using temporal multiplexing
• CPC Class Codes: H04N 13/398 : Synchronisation thereof; Control thereof

Abstract

In a telemanipulation system for manipulating objects located in a workspace at a remote worksite by an operator from an operator'"'"'s station, such as in a remote surgical system, the remote worksite having a manipulator with an end effector for manipulating an object at the workspace, such as a body cavity, a controller including a hand control at the control operator'"'"'s station for remote control of the manipulator, an image capture device, such as a camera, and image output device for reproducing a viewable real-time image, the improvement wherein a position sensor associated with the image capture device senses position relative to the end effector and a processor transforms the viewable real-time image into a perspective image with correlated manipulation of the end effector by the hand controller such that the operator can manipulate the end effector and the manipulator as if viewing the workspace in true presence.

Claims

1. A master-slave robotic system comprising:

a master input device movable in a master workspace;

a slave manipulator comprising a manipulator arm coupled to an end effector, the slave manipulator movable in a slave workspace;

an image capture device having a field of view movable in the slave workspace for generating a captured image, the image capture device generating position signals indicating the field of view; and

a processor coupling the master input device to the slave manipulator, the processor configured to map movement of the master input device in the master workspace to movement of the slave manipulator in the slave workspace, the processor configured to transform the captured image into a displayed image in the master workspace based on the position signals of the image capture device.

2. The system of claim 1, wherein the processor is configured to transform said captured image such that an image of the end effector in the display is substantially positioned at a perceived location of the master input device in the master workspace.

3. The system of claim 1, wherein the slave manipulator generates signals responsive to a position of the end effector in the slave workspace, the processor configured to derive the transformation using the signals of the slave.

4. The system of claim 3, the master input device configured to generate position signals indicating a position of the master input device, the processor configured to determine a position and orientation of the master input device in the master workspace based on said position signals of the master input device, the processor further configured to determine a position and orientation of the end effector in the slave workspace from the position signals of the end effector, the processor further configured to move the slave by comparing the position and orientation of the master input device with the position and orientation of the end effector.

5. The system of claim 1, the processor configured to derive the transformation in real time.

6. The system of claim 1, the processor further configured to transform said captured image such that a first vector aligns with a second vector, the first vector defined as originating from the master input device pointing toward a distal portion of the slave manipulator in the displayed image, the second vector defined as originating from a proximal portion of the slave manipulator in the displayed image pointing toward the distal portion of the slave manipulator in the displayed image.

7. The system of claim 1, the processor further configured to transform said captured image by rotating the captured image.

8. The system of claim 1, further comprising a second image capture device for capturing a stereo image of the slave manipulator, the processor further configured to transform a stereo image of the display.

9. The system of claim 1, the master input device comprising first and second input handles, the slave manipulator further comprising a second manipulator arm coupled to a second end effector, the processor configured to transform said captured image such that an image of the two end effectors in the display is substantially positioned at a perceived position of the first and second input handles of the master input device in the master workspace.

10. The system of claim 1, the processor further configured to effect dynamic synchronization of the position of the distal portion of the slave manipulator in the displayed image with the position of the master input device.

11. A method for operating a master-slave robotic system having a master input device, a slave manipulator comprising a manipulator arm coupled to an end effector, an image capture device, a display, and a processor, the method comprising:

moving the master input device in a master workspace;

moving the slave manipulator in a slave workspace in response to the moving the master input device in the master workspace;

capturing an image of the slave workspace using the image capture device;

generating signals indicating the position of the image capture device; and

using the processor, transforming the captured image into an image for the display based on the signals indicating the position of the image capture device.

12. The method of claim 11, the transforming the captured image comprising transforming said captured image such that an image of the end effector in the display is substantially positioned at a perceived location of the master input device in the master workspace.

13. The method of claim 11, further comprising generating signals indicating a position of the end effector in the surgical workspace, the processor deriving the transformation using said generated signals indicating the position of the end effector.

14. The method of claim 13, further comprising:

generating signals indicating the position of the master input device;

using the processor, determining a position and orientation of the master input device in the master workspace based on said position signals of the master input device;

using the processor, determining a position and orientation of the end effector in the slave workspace from the position signals of the end effector; and

moving the slave by comparing the position and orientation of the master input device and the end effector in the mapped space.

15. The method of claim 11, further comprising performing the transforming in real time.

16. The method of claim 11, further comprising transforming said captured image such that a first vector aligns with a second vector, the first vector defined as originating from the master input device pointing toward a distal portion of the slave manipulator in the displayed image, the second vector defined as originating from a proximal portion of the slave manipulator in the displayed image pointing toward the distal portion of the slave manipulator in the displayed image.

17. The method of claim 11, further comprising transforming said captured image by rotating the captured image.

18. The method of claim 11, further comprising:

capturing a stereo image of the slave workspace using a second image capture device in combination with the image capture device; and

transforming said stereo image of the display based on signals indicating the positions of the image capture device and the second image capture device.

19. The method of claim 11, the master input device comprising first and second input handles, the slave manipulator further comprising a second manipulator arm coupled to a second end effector, the method further comprising:

transforming said captured image such that an image of the two end effectors in the display is substantially positioned at a perceived location of the first and second input handles of the master input device in the master workspace.

20. The method of claim 11, further comprising effecting dynamic synchronization of the position of the distal portion of the slave manipulator in the displayed image with the position of the master input device.