Efficient Vision and Kinematic Data Fusion For Robotic Surgical Instruments and Other Applications

• US 20100331855A1
• Filed: 06/30/2009
• Published: 12/30/2010
• Est. Priority Date: 05/16/2005
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Incorporated (Intuitive Surgical Holdings, LLC)
• Inventors: Hui, Hua, Hoffman, Brian David, Nowlin, William C., Zhao, Tao, Zhao, Wenyi
• Granted Patent: US 8,971,597 B2
• Time in Patent Office: Days
• US Class Current: 606/130
• CPC Class Codes: A61B 2034/2059 Mechanical position encoders
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/3937 Visible markers
• CPC Class Codes: A61B 34/30 Surgical robots

Abstract

Robotic devices, systems, and methods for use in telesurgical therapies through minimally invasive apertures make use of joint-based data throughout much of the robotic kinematic chain, but selectively rely on information from an image capture device to determine location and orientation along the linkage adjacent a pivotal center at which a shaft of the robotic surgical tool enters the patient. A bias offset may be applied to a pose (including both an orientation and a location) at the pivotal center to enhance accuracy. The bias offset may be applied as a simple rigid transformation from the image-based pivotal center pose to a joint-based pivotal center pose.

Claims

1. A robotic method for performing surgery on a patient through a minimally invasive aperture, the method comprising:

positioning an elongate shaft of a tool through the aperture so that the shaft pivots at an actual pivotal center adjacent the aperture;

acquiring an image of the tool within the patient;

determining image-based pivotal center correction information using the image;

determining a position and/or orientation of the tool by imposing the image-based pivotal center correction information onto joint-based data, the joint-based data being used by a robotic command module to calculate movements of a tool manipulator supporting the tool.

2. The robotic method of claim 1, wherein determining the image-based pivotal center correction information comprises determining a bias offset between a joint-based location of the pivotal center per the joint data from the manipulator, and an image-based location of the pivotal center per the image, and wherein the position and/or orientation of the tool are determined by adjusting the joint-based pivotal center location per the bias offset such that the image-based pivotal center location substantially replaces the joint-based pivotal center location.

3. The robotic method of claim 2, wherein determining the bias offset further comprises determining a bias offset between a joint-based pivotal center pose defined by the joint-based location of the pivotal center and an orientation of the shaft per joint data from the manipulator, and an image-based pivotal center pose defined by the image-base pivotal center location and an orientation of the shaft per the image, and wherein position and orientation of the tool are determined by adjusting the joint-based pivotal center pose per the bias offset such that the image-based pivotal center pose substantially replaces the joint-based pivotal center pose, and such that other poses along a linkage of the tool and tool manipulator are calculated using joint-based data.

4. The robotic method of claim 3, wherein determining the offset further comprises determining a rigid transformation between the joint-based pivotal center pose and the image-based pivotal center pose, and wherein the command module applies the rigid transformation to the joint-based pivotal center pose.

5. The robotic method of claim 3, wherein the image is acquired by an image capture device movably supported by a camera manipulator, wherein the image based pivotal center pose is determined using a coordinate system of the image capture device, wherein the command module generates the joint-based pivotal center pose in response to joint states of the tool and the tool manipulator and also determines a joint-based camera coordinate system, and wherein the determined bias offset is determined between the joint-based pivotal center in the joint-based camera coordinate system and the image-based pivotal center location in the camera coordinate system.

6. The robotic method of claim 2, wherein the camera manipulator is supported by a movable camera support, wherein the tool manipulator is supported by a movable tool support, wherein the tool manipulator pivots the shaft during the movements at the pivotal center about a pitch axis and about a yaw axis, and wherein the bias offset primarily comprises a combination of:

tool support data error;

camera support data error;

pitch data error; and

yaw data error.

7. The robotic method of claim 6, wherein the tool support comprises a tool set-up linkage and the tool support data error comprises joint state data error and structural inaccuracy of the tool set-up linkage, wherein the camera support comprises a set-up linkage and the tool support data error comprises joint state data error and structural inaccuracy of the camera set-up linkage.

8. The robotic method of claim 1, wherein a plurality of images are obtained by a plurality of angularly offset image capture devices, further comprising determining a three-dimensional (3D) image-based pivotal center pose using the image, the image-based pivotal center pose comprising an image-based 3D orientation along the shaft and an image-based pivotal point in three dimensions.

9. The robotic method of claim 8, wherein the plurality of images comprise a time-series of images, and further comprising obtaining, for each image, associated joint-based data synchronized with the image, and computing the image based pivotal center pose using the time series of images and the associated joint-based data.

10. The robotic method of claim 9, further comprising determining a time series of tool locations from the images, detecting at least one of the tool locations as being an outlier, and removing the at least one outlier before computer the image based pivotal center pose.

11. The robotic method of claim 9, further comprising determining a region of interest of a first image of the time-series based on a location of the tool in at least one prior second image of the series, wherein the location of the tool in the first image is determined by processing the portion of the first image within the region of interest.

12. The robotic method of claim 8, the robotic command module determining a joint-based pivotal center pose using joint data from the tool manipulator, wherein the image-based pivotal center location is determined using image pairs from the image capture devices, and wherein the tool position and/or orientation is determined by solving for a rigid transformation between the image-based pivotal center pose and the joint-based pivotal center pose using a recursive filter.

13. The robotic method of claim 12, wherein the image-based pivotal center pose is determined by identifying a plurality of marker points along the tool from the images.

14. The robotic method of claim 12, wherein the image-base pivotal center pose is determined by identifying at least one structural location line along the tool from the images.

15. A robotic method for performing surgery on a patient through a minimally invasive aperture with a robotic system, the robotic system including a tool having an elongate shaft and a command module that determines movements of a tool manipulator supporting the tool by determining a joint-based pivotal center pose of the tool, the method comprising:

positioning the elongate shaft of the tool through the aperture so that the shaft pivots at an actual pivotal center adjacent the aperture, the actual pivotal center and the shaft defining an actual pivotal center pose;

acquiring an image of the tool within the patient;

determining an image-based pivotal center pose of the tool using the image;

calculating a rigid transformation between the image-based pivotal center pose and the joint-based pivotal center pose using a recursive filter; and

determining a location and/or orientation of the tool by using the rigid transformation to mitigate an error between the joint-based pivotal center pose and the actual pivotal center pose.

16. A robotic method for use with a robotic system including a manipulator movably supporting a tool and a command module generating movement commands of the manipulator so as to generate desired movements of the tool based on joint data from the manipulator, wherein environmental constraints induce an error in a joint-based pose calculated by the command manipulator along a linkage of the manipulator or tool, the method comprising:

acquiring an image of the tool and/or manipulator;

selectively determining an image-based pose from the image, the pose reflecting the environmental constraints;

calculating a rigid transformation between the image-based pose and the joint-based calculated pose; and

determining a pose of the tool by using the rigid transformation so as to mitigate the error.

17. The robotic method of claim 16, wherein the rigid transformation is calculated using a recursive filter with a series of image-based poses and an associated series of calculated poses.

18. The robotic method of claim 17, wherein updating the rigid transformation takes a first computation time, wherein a general solution for poses of the manipulator linkage, when unconstrained by the environmental constraint, would involve a second computation time at least 1000 times greater than the first computational time.

19. A robotic system for performing surgery on a patient through a minimally invasive aperture, the system comprising:

a tool having an elongate shaft with a proximal end and a distal end, the distal end of the tool insertable through the aperture so that the shaft pivots at a pivotal center adjacent the aperture;

an image capture device for acquiring an image of the tool within the patient;

a tool manipulator supporting the proximal end of the tool; and

a processor system coupling the image capture device to the tool manipulator, the processor system including a tool correction module and a robotic command module coupled to the tool manipulator so as to transmit tool movement commands thereto, and so as to calculate a joint-based pivotal center in response to joint signals therefrom, the tool correction module generating a corrected tool position and/or orientation by determining image-based pivotal center correction information from the image and by correcting the joint-based pivotal center with the image-based pivotal center correction information.

20. A correction system for use with a robotic surgical system, the robotic surgical system including a tool having an elongate shaft with a distal end insertable through a minimally invasive aperture so that the shaft pivots at an actual pivotal center adjacent the aperture, an image capture device for acquiring an image of the tool within the patient, a tool manipulator supporting the tool, and a robotic command module transmitting movement commands to the tool manipulator, and calculates a joint-based pivotal center, the correction system comprising:

tool correction module coupling the image capture device to the command module so as to determine a position and/or orientation of the tool by revising the joint-based pivotal center with image-based pivotal center correction information derived in response to the image of the tool within the patient such that an error between the joint-based pivotal center and the actual pivotal center is mitigated.

21. The correction system of claim 20, wherein the tool correction module is configured to determine a bias offset between a joint-based location of the pivotal center per the joint signals from the manipulator, and an image-based location of the pivotal center per the image.

22. The correction system of claim 21, wherein the tool correction module is configured to determine the bias offset by determining a bias offset between a joint-based pivotal center pose defined by the joint-based location of the pivotal center and an orientation of the shaft per the joint signals from the manipulator, and an image-based pivotal center pose defined by the image-base pivotal center location and an orientation of the shaft per the image.

23. The correction system of claim 22, wherein the module is configured to determine the offset by determining a rigid transformation between the joint-based pivotal center pose and the image-based pivotal center pose.

24. The correction system of claim 23, the robotic system including a camera manipulator movably supporting the image capture device, the command module of the robotic system generating the joint-based pivotal center pose in response to joint states of the tool and the tool manipulator and also determining a joint-based camera coordinate system, wherein the tool correction module is configured to determine the image-based pivotal center pose using a coordinate system of the image capture device, and wherein the tool correction module is configured to determine the bias offset between the joint-based pivotal center in the joint-based camera coordinate system and the image-based pivotal center location in the camera coordinate system.

25. The correction system of claim 20, the robotic system comprising a plurality of angularly offset image capture devices for acquiring a plurality of images, wherein the module is configured for determining a three-dimensional (3D) image-based pivotal center pose using the images, the image-based pivotal center pose comprising an image-based 3D orientation along the shaft and an image-based pivotal point in three dimensions.

26. The correction system of claim 25, the robotic command module determining a joint-based pivotal center pose using joint data from the tool manipulator, wherein the module is configured for determining the pivotal center location from of image pairs transmitted from the image capture devices, and wherein the tool correction module comprises a recursive filter that determines the pivotal center pose by solving for a rigid transformation between the image-based pivotal center pose and the joint-based pivotal center pose.

27. The correction system of claim 26, wherein the module determines the image-based pivotal center pose is by identifying a plurality of marker points along the tool from the images.

28. The correction system of claim 26, wherein the module determines the image-base pivotal center pose by identifying at least one structural location line along the tool from the images.

29. A robotic system comprising:

a manipulator movably supporting a tool;

a command module generating movement commands of the manipulator so as to generate desired movements of the tool based on joint data from the manipulator, wherein environmental constraints induce an error in a joint-based pose calculated by the command manipulator along a linkage of the manipulator or tool;

an image capture device for acquiring an image of the tool and/or manipulator;

means for image-based correction coupling the image capture device to the command module, the correction means comprising means for determining an image-based pose of the manipulator or tool from the image, the pose reflecting the environmental constraints, means for calculating a rigid transformation between the image-base pose and the joint-based calculated pose, and means for using the rigid transformation to mitigate the error for determining a position and/or orientation of the tool.

30. The robotic system of claim 29, wherein the means for correcting comprises a recursive filter.

31. The robotic system of claim 29, wherein the command module does not apply the image-based pose for calculating the movement commands.

32. The robotic system of claim 29, wherein the command module comprises means for applying the rigid transformation to calculate the movement commands.