Efficient vision and kinematic data fusion for robotic surgical instruments and other applications

• US 8,971,597 B2
• Filed: 06/30/2009
• Issued: 03/03/2015
• Est. Priority Date: 05/16/2005
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Zhao, Tao, Zhao, Wenyi, Hoffman, Brian D., Nowlin, William C., Hui, Hua
• Primary Examiner(s): Thomson, William
• Assistant Examiner(s): Downey, John R
• Application Number:
  US12/495,304
• Publication Number:
  US 20100331855A1
• Time in Patent Office: 2,072 Days
• Field of Search: 606/1, 382/128
• US Class Current: 382/128
• CPC Class Codes: A61B 2034/2059 Mechanical position encoders
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/3937 Visible markers
• CPC Class Codes: A61B 34/30 Surgical robots

Abstract

Robotic devices, systems, and methods for use in telesurgical therapies through minimally invasive apertures make use of joint-based data throughout much of the robotic kinematic chain, but selectively rely on information from an image capture device to determine location and orientation along the linkage adjacent a pivotal center at which a shaft of the robotic surgical tool enters the patient. A bias offset may be applied to a pose (including both an orientation and a location) at the pivotal center to enhance accuracy. The bias offset may be applied as a simple rigid transformation from the image-based pivotal center pose to a joint-based pivotal center pose.

Claims

1. A robotic-method for performing surgery on a patient through a minimally invasive aperture, the method comprising:

positioning, by a robotic surgical system, an elongate shaft of a tool through the minimally invasive aperture so that the shaft pivots at an actual pivotal center adjacent the minimally invasive aperture;

acquiring, by an image capture device, an image of a surgical site within the patient, the surgical site image including an image of a distal portion of the tool within the field of view of the image capture device, the image of the distal portion including an image of markers, the image capture device being inserted into the patient;

determining, by the robotic surgical system using the image of the markers, an image-based location of the actual pivotal center;

determining image-based pivotal center correction information using the image-based location of the actual pivotal center; and

determining, by the robotic surgical system, a position and/or orientation of the tool by imposing the image-based pivotal center correction information onto joint-based data, the joint-based data being used by a command module to calculate movements of a tool manipulator supporting the tool.

2. The robotic method of claim 1:

wherein the determining the image-based pivotal center correction information comprises determining a bias offset between a joint-based location of the actual pivotal center per the joint data from the manipulator and the image-based location of the actual pivotal center, and

wherein the determining the position and/or orientation of the tool comprises adjusting the joint-based pivotal center location per the bias offset such that the image-based location of the actual pivotal center substantially replaces the joint-based location of the actual pivotal center.

3. The robotic method of claim 2:

wherein the determining the bias offset further comprises determining a bias offset between a joint-based pivotal center pose defined by the joint-based location of the actual pivotal center and an orientation of the shaft per joint data from the manipulator, and an image-based pivotal center pose defined by the image-based location of the actual pivotal center and an orientation of the shaft per the image, and

wherein the determining the position and/or orientation of the tool further comprises adjusting the joint-based pivotal center pose per the bias offset such that the image-based pivotal center pose substantially replaces the joint-based pivotal center pose, and such that other poses along a linkage of the tool and the tool manipulator are calculated using joint-based data.

4. The robotic method of claim 3:

wherein the determining the bias offset further comprises determining a rigid transformation between the joint-based pivotal center pose and the image-based pivotal center pose, and

wherein the robotic method further comprises applying, by the command module, the rigid transformation to the joint-based pivotal center pose.

5. The robotic method of claim 3:

wherein the image capture device is movably supported by a camera manipulator of the robotic surgical system,

wherein the determining the bias offset further comprises;

determining the image-based pivotal center pose using a camera coordinate system of the image capture device,

generating, by the command module, the joint-based pivotal center pose in response to joint states of the tool and the tool manipulator,

determining, by the command module, a joint-based camera coordinate system, and

wherein the determined bias offset is determined between the joint-based location of the actual pivotal center in the joint-based camera coordinate system and the image-based location of the actual pivotal center in the camera coordinate system.

6. The robotic method of claim 5:

wherein the camera manipulator is supported by a movable camera support of the robotic surgical system,

wherein the tool manipulator is supported by a movable tool support of the robotic surgical system,

wherein the robotic method further comprises pivoting, by the tool manipulator, the shaft during the movements at the actual pivotal center about a pitch axis and about a yaw axis, and

wherein the bias offset comprises a combination of;

tool support data error;

camera support data error;

pitch data error; and

yaw data error.

7. The robotic method of claim 6:

wherein the tool support comprises a tool set-up linkage and the tool support data error comprises joint state data error and structural inaccuracy of the tool set-up linkage, and

wherein the camera support comprises a set-up linkage and the tool support data error comprises joint state data error and structural inaccuracy of the camera set-up linkage.

8. The robotic method of claim 1:

wherein a plurality of images is obtained by a plurality of angularly offset image capture devices of the robotic surgical system, the plurality of offset image capture devices including the image capture device,

wherein the robotic method further comprises determining a three-dimensional (3D) image-based pivotal center pose using the plurality of images, and

wherein the image-based pivotal center pose comprises an image-based 3D orientation along the shaft and an image-based location of the actual pivotal center in three dimensions.

9. The robotic method of claim 8:

wherein the plurality of images comprises a time-series of images, and

wherein the robotic method further comprises obtaining, for each image in the time-series of images, associated joint-based data synchronized with the image, and computing the image based pivotal center pose using the time series of images and the associated joint-based data.

10. The robotic method of claim 9, further comprising determining a time series of tool locations from the time-series of images, detecting any outlier tool location, and removing any detected outlier before computing the image based pivotal center pose.

11. The robotic method of claim 9, further comprising determining a region of interest of a first image of the time-series based on a location of the tool in at least one prior second image of the time-series, wherein the location of the tool in the first image is determined by processing the portion of the first image within the region of interest.

12. The robotic method of claim 8, further comprising:

determining, by the command module, a joint-based pivotal center pose using joint data from the tool manipulator,

wherein the determining the image-based pivotal center pose comprises determining an image-based location of the actual pivotal center using image pairs of the plurality of images from the plurality of angularly offset image capture devices, and

wherein the determining the position and/or orientation of the tool comprises solving for a rigid transformation between the image-based pivotal center pose and the joint-based pivotal center pose using a recursive filter.

13. The robotic method of claim 12, wherein the determining the image-based pivotal center pose comprises identifying a plurality of marker points along the tool from the plurality of images.

14. The robotic method of claim 12, wherein the determining the image-based pivotal center pose comprises identifying at least one structural location line along the tool from the plurality of images.

15. A robotic method for performing surgery on a patient through a minimally invasive aperture with a robotic system, the robotic system including a tool having an elongate shaft and a command module that determines movements of a tool manipulator supporting the tool by determining a joint-based pivotal center pose of the tool, the method comprising:

positioning, by the robotic system, the elongate shaft of the tool through the minimally invasive aperture so that the shaft pivots at an actual pivotal center adjacent the minimally invasive aperture, the actual pivotal center and the shaft defining an actual pivotal center pose;

acquiring, by an image capture device of the robotic system, an image of a surgical site within the patient, the surgical site image including an image of a distal portion of the tool within the field of view of the image capture device, the image of the distal portion of the tool including an image of markers, the image capture device being inserted into the patient;

determining, by the robotic system using the image of the markers, an image-based pose of the actual pivotal center pose;

calculating, by the robotic system, a rigid transformation between the image-based pose of the actual pivotal center pose and the joint-based pivotal center pose using a recursive filter; and

determining, by the robotic system, a location and/or orientation of the tool by using the rigid transformation to mitigate an error between the joint-based pivotal center pose and the actual pivotal center pose.

16. A robotic method for use with a robotic system including a manipulator movably supporting a tool and a command module generating movement commands of the manipulator so as to generate desired movements of the tool based on joint data from the manipulator, wherein environmental constraints induce an error in a joint-based pivotal center pose of the tool, the joint-based pivotal center pose being calculated by the command module along a linkage of the manipulator or tool, the joint-based pivotal center pose being defined by a joint-based location of an actual pivotal center and an orientation of a shaft of the tool per joint data, the shaft of the tool entering the patient through an incision in a patient, the shaft pivoting at the actual pivotal center, the actual pivotal center being adjacent the incision, the method comprising:

acquiring, by the robotic system, an image of a surgical site within the patient, the surgical site image including an image of a distal portion of the tool within the field of view of the image capture device, the image of the distal portion of the tool including an image of markers, the image capture device being inserted into the patient

selectively determining, by the robotic system using the image of the markers, an image-based pivotal center pose of the tool reflecting the environmental constraints, the image-based pivotal center pose of the tool being defined by an image-based location of the actual pivotal center and an orientation of the shaft per the image of the distal portion of the tool;

calculating, by the robotic system, a rigid transformation between the image-based pivotal center pose and the joint-based pivotal center pose; and

determining, by the robotic system, a location of the actual pivotal center and orientation of the shaft of the tool by using the rigid transformation so as to mitigate the error in the joint-based pivotal center pose of the tool induced by the environmental constraints.

17. The robotic method of claim 16, wherein calculating the rigid transformation comprises using a recursive filter with a series of image-based pivotal center poses and an associated series of calculated joint-based pivotal center poses.

18. The robotic method of claim 17, further comprising updating the rigid transformation.

19. A robotic system for performing surgery on a patient through a minimally invasive aperture, the system comprising:

a tool having a proximal end, a distal end, and an elongate shaft, the distal end of the tool being configured to be insertable through the minimally invasive aperture, the distal end of the tool including markers, and the shaft being configured to pivot at a pivotal center adjacent the minimally invasive aperture;

an image capture device, the image capture device being configured to be insertable in the patent, the image capture device being configured to acquire an image of a surgical site within the patient, the surgical site image including an image of a distal end of the tool within the field of view of the image capture device, the image of the distal end of the tool including an image of markers;

a tool manipulator supporting the proximal end of the tool; and

a processor system coupling the image capture device to the tool manipulator, the processor system including a tool correction module and a command module, the command module being coupled to the tool manipulator, the command module being configured to transmit tool movement commands to the tool manipulator, the command module being configured to calculate a joint-based pivotal center in response to joint signals from the tool manipulator, the tool correction module being configured to generate a corrected tool position and/or orientation by determining an image-based location of the pivotal center using the image of the markers, by determining image-based pivotal center correction information using the image-based location of the pivotal center, and by correcting the joint-based pivotal center with the image-based pivotal center correction information.

20. A correction system comprising:

a tool correction module coupling an image capture device of a robotic surgical system to a command module of the robotic surgical system,

the robotic surgical system comprising a tool having a proximal end, a distal end, and an elongate shaft, the distal end of the tool being configured to be insertable through a minimally invasive aperture in a patient, the distal end of the tool including markers, and the shaft being configured to pivot at a pivotal center adjacent the minimally invasive aperture;

the image capture device being configured to be insertable in the patent and the image capture device being configured to acquire an image of a surgical site within the patient, the surgical site image including an image of a distal end of the tool within the field of view of the image capture device, the image of the distal end of the tool including an image of markers;

a tool manipulator supporting the proximal end of the tool; and

a processor system including the command module coupled to the tool manipulator, the command module being configured to transmit tool movement commands to the tool manipulator, and the command module being configured to calculate a joint-based location of the pivotal center in response to joint signals; and

the tool correction module being configured to determine a position and/or orientation of the tool by determining an image-based location of the pivotal center using the image of the markers, by determining image-based pivotal center correction information using the image-based location of the pivotal center, and by correcting the joint-based location of the pivotal center with the image-based pivotal center correction information such that an error between the joint-based pivotal center and the actual pivotal center is mitigated.

21. The correction system of claim 20, wherein the tool correction module is configured to determine a bias offset between a joint-based location of the pivotal center per the joint signals from the tool manipulator, and the image-based location of the pivotal center per the image.

22. The correction system of claim 21, wherein the tool correction module is configured to determine the bias offset by determining a offset between a joint-based pivotal center pose defined by the location of the joint-based location of the pivotal center and an orientation of the shaft per the joint signals from the tool manipulator, and an image-based pivotal center pose defined by the image-based location of the pivotal center and an orientation of the shaft per the image.

23. The correction system of claim 22, wherein the tool correction module is configured to determine the bias offset by determining a rigid transformation between the joint-based pivotal center pose and the image-based pivotal center pose.

24. The correction system of claim 23, the robotic system including a camera manipulator movably supporting the image capture device, the command module of the robotic system generating the joint-based pivotal center pose in response to joint states of the tool and the tool manipulator and also determining a joint-based camera coordinate system, wherein the tool correction module is configured to determine the image-based pivotal center pose using a camera coordinate system of the image capture device, and wherein the tool correction module is configured to determine the bias offset between the joint-based location of the pivotal center in the joint-based camera coordinate system and the image-based location of the pivotal center in the camera coordinate system.

25. The correction system of claim 20, the robotic system comprising a plurality of angularly offset image capture devices configured to acquire a plurality of images, the plurality of angularly offset image capture devices including the image capture device, wherein the tool correction module is configured for determining a three-dimensional (3D) image-based pivotal center pose using the plurality of images, the image-based pivotal center pose comprising an image-based 3D orientation along the shaft and an image-based location of the pivotal center in three dimensions.

26. The correction system of claim 25, the command module determining a joint-based pivotal center pose using joint data from the tool manipulator, wherein the tool correction command module is configured for determining the image-based location of the pivotal center from of image pairs of the plurality of images transmitted from the plurality of angularly offset image capture devices, and wherein the tool correction module comprises a recursive filter that determines the pivotal center pose by solving for a rigid transformation between the image-based pivotal center pose and the joint-based pivotal center pose.

27. The correction system of claim 26, wherein the tool correction module determines the image-based pivotal center pose is by identifying a plurality of marker points along the tool from the plurality of images.

28. The correction system of claim 26, wherein the tool correction module determines the image-base pivotal center pose by identifying at least one structural location line along the tool from the plurality of images.

29. A robotic system comprising:

a manipulator movably supporting a tool having an elongate shaft;

a command module generating movement commands of the manipulator so as to generate desired movements of the tool based on joint data from the manipulator, wherein environmental constraints induce an error in a joint-based pivotal center pose calculated by the command module along a linkage of the manipulator or tool, the joint-based pivotal center pose being defined by a joint-based location of an actual pivotal center and an orientation of a shaft of the tool per joint data, the shaft of the tool entering the patient through an incision in a patient, the shaft pivoting at the actual pivotal center, the actual pivotal center being adjacent the incision;

an image capture device for acquiring an image of a surgical site within the patient, the surgical site image including an image of a distal portion of the tool within the field of view of the image capture device, the image of the distal portion of the tool including an image of markers, if the image capture device is inserted into the patient;

means for image-based correction coupling the image capture device to the command module, the correction means comprising means for determining an image-based pivotal center pose of the tool from the image of the markers, the image-based pivotal center pose reflecting the environmental constraints, the image-based pivotal center pose of the tool being defined by the image-based pivotal center location and an orientation of the shaft per the image of the distal portion of the tool, means for calculating a rigid transformation between the image based pivotal center pose of the tool and the joint-based pivotal center pose, and means for using the rigid transformation to mitigate the error in the joint-based pivotal center pose of the tool induced by the environmental constraints.

30. The robotic system of claim 29, wherein the means for correcting comprises a recursive filter.

31. The robotic system of claim 29, wherein the command module does not apply the image-based pose for calculating the movement commands.

32. The robotic system of claim 29, wherein the command module comprises means for applying the rigid transformation to calculate the movement commands.