COLLISION AVOIDANCE DURING CONTROLLED MOVEMENT OF IMAGE CAPTURING DEVICE AND MANIPULATABLE DEVICE MOVABLE ARMS

• US 20140163736A1
• Filed: 12/10/2013
• Published: 06/12/2014
• Est. Priority Date: 12/10/2012
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC), Siemens Healthineers AG (Siemens AG)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Azizian, Mahdi, Sorger, Jonathan, Kunze, Holger, Blohm, Lutz, Niebler, Christine
• Granted Patent: US 9,259,282 B2
• Time in Patent Office: Days
• US Class Current: 700/259
• CPC Class Codes: A61B 1/00009 of image signals during a use of endoscope
• CPC Class Codes: A61B 1/00087 Tools A61B17/00234 takes precedence
• CPC Class Codes: A61B 1/00149 using articulated arms
• CPC Class Codes: A61B 1/00183 for variable viewing angles
• CPC Class Codes: A61B 1/05 characterised by the image sensor, e.g. camera, being in the distal end portion
• CPC Class Codes: A61B 2017/00216 with eye tracking or head position tracking control
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2034/2059 Mechanical position encoders
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/306 using optical fibres
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 2090/376 using X-rays, e.g. fluoroscopy
• CPC Class Codes: A61B 2090/378 using ultrasound
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 6/102 Protection against mechanical damage, e.g. anti-collision devices
• CPC Class Codes: A61B 6/4441 the rigid structure being a C-arm or U-arm
• CPC Class Codes: A61B 6/4458 the source unit or the detector unit being attached to robotic arms
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: B25J 9/1676 : Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1697 : Vision controlled systems
• CPC Class Codes: Y10S 901/02 : Arm motion controller
• CPC Class Codes: Y10S 901/47 : Optical

Abstract

A system and method for movement control includes a controller coupled to a computer-assisted surgical device having a first movable arm coupled to a manipulatable device having a working end and a second movable arm coupled to an image capturing device. The controller is configured to receive first configurations for the first movable arm; receive second configurations for the second movable arm; receive a plurality of images of the working end from the image capturing device; determine a position and an orientation of the working end; determine a first movable arm position and trajectory for the first movable arm; determine a second movable arm position and trajectory for the second movable arm; determine whether motion of the movable arms will result in an undesirable relationship between the movable arms; and send a movement command to the first or second movable arm to avoid the undesirable relationship.

Claims

1. A movement control system comprising:

a controller comprising;

one or more processors; and

memory coupled to the one or more processors;

wherein;

the controller is coupled to a computer-assisted surgical device having a first movable arm coupled to a manipulatable device having a working end;

the controller is further coupled to a second movable arm coupled to an image capturing device separate from the computer-assisted surgical device;

the controller is configured to;

receive one or more first configurations for the first movable arm;

receive one or more second configurations for the second movable arm;

receive a first plurality of images of the working end from the image capturing device;

determine, based on at least one of the first plurality of images, a position and an orientation of the working end in a common reference frame;

determine, based on the first configurations, a first movable arm position and a first movable arm trajectory for the first movable arm in the common reference frame;

determine, based on the second configurations, a second movable arm position and a second movable arm trajectory for the second movable arm in the common reference frame;

based on the first movable arm position, the first movable arm trajectory, the second movable arm position, and the second movable arm trajectory, determine whether motion of the first movable arm, motion of the second movable arm, or motions of the first and second movable arms together will result in an undesirable relationship between the first and second movable arms; and

send a first movement command to the first movable arm or the second movable arm to avoid the undesirable relationship.

2. The movement control system of claim 1 wherein:

the common reference frame is a reference frame of the image capturing device; and

the controller is further configured to determine the first movable arm position in the common reference frame further based on the position and the orientation of the working end in the common reference frame.

3. The movement control system of claim 1 wherein:

the common reference frame is a reference frame of the computer-assisted surgical device; and

the controller is further configured to;

determine the second movable arm position in a reference frame of the image capturing device;

determine the position and the orientation of the working end in the reference frame of the image capturing device; and

transform the second movable arm position and the position and orientation of the working end from the reference frame of the image capturing device to the common reference frame.

4. The movement control system of claim 1 wherein:

the first plurality of images are a plurality of two-dimensional images from a image capturing device; and

the controller is further configured to determine the position and the orientation of the working end from the two-dimensional images even when the working end of the manipulatable device is occluded by one or more objects disposed between the image capturing device and the working end of the manipulatable device.

5. The movement control system of claim 1, further comprising:

a viewer adapted to display a second plurality images of a work space of the working end of the manipulatable device; and

an input unit configured to receive information of a user specified region of interest within the images being displayed on the viewer;

wherein the controller is further configured to send second movement commands to the second movable arm so that the image capturing device captures images of the user specified region of interest.

6. The movement control system of claim 5 wherein:

the input unit comprises a gaze tracking unit configured to track a gaze of a user on a display screen of the viewer; and

the gaze tracking unit includes an indicator operable by the user to indicate the region of interest by using a current gaze point of the user on the display screen.

7. The movement control system of claim 5, wherein:

the input unit comprises a telestrator unit configured to receive one of the second plurality of images and display the received one of the second plurality of images on a display screen of the telestrator; and

the telestrator unit includes a marker unit operable by a user to indicate the region of interest on the displayed one of the second plurality of images.

8. The movement control system of claim 5, further comprising a camera, wherein the second plurality of images are captured by the camera.

9. The movement control system of claim 5, further comprising an ultrasound probe, wherein the second plurality of images are captured by the ultrasound probe.

10. The movement control system of claim 1 wherein:

the first movable arm has redundant degrees of freedom so that for each controllable position and orientation of the working end of the manipulatable device there are a first plurality of possible positions and orientations for the first movable arm;

the second movable arm has redundant degrees of freedom so that for each controllable position and orientation of the image capturing device there are a second plurality of possible positions and orientations for the second movable arm; and

the first movement command sent to the first movable arm or the second movable arm directs the first movable arm to move to one of the first plurality of possible positions and orientations or directs the second movable arm to move to one of the second plurality of possible positions and orientations that avoid the undesirable relationship.

11. The movement control system of claim 10 wherein the controller is further configured to:

determine which one of the first movable arm and the second movable arms is to be sent the first movement command based on differences between the first movable arm position and the first plurality of possible positions and orientations and differences between the second movable arm position and the second plurality of possible positions and orientations; and

the determination is made so as to minimize a cost function.

12. The movement control system of claim 11 wherein the first movement command directs the first movable arm to move from the first movable arm position to a first selected one of the first plurality of possible positions and orientations or directs the second movable arm to move from the second movable arm position to a second selected one of the second plurality of possible positions and orientations based on which of the first selected one of the first plurality of possible positions and orientations and the second selected one of the second plurality of possible positions and orientations minimizes the cost function.

13. The movement control system of claim 1 wherein the undesirable relationship is selected from a group consisting of a collision between the first movable arm and the second movable arm, too close a proximity between the first movable arm and the second movable arm, and obstruction of a region of interest of the image capturing device by the first movable arm.

14. A method of controlling movement in a medical system, the method comprising:

receiving one or more first configurations for a first movable arm of a computer-assisted surgical device, the first movable arm being coupled to a manipulatable device having a working end;

receiving one or more second configurations for a second movable arm coupled to an image capturing device separate from the computer-assisted surgical device;

receiving a first plurality of images of the working end from the image capturing device;

determining, based on at least one of the first plurality of images, a position and an orientation of the working end in a common reference frame;

determining, based on the first configurations, a first movable arm position and a first movable arm trajectory for the first movable arm in the common reference frame;

determining, based on the second configurations, a second movable arm position and a second movable arm trajectory for the second movable arm in the common reference frame;

based on the first movable arm position, the first movable arm trajectory, the second movable arm position, and the second movable arm trajectory, determining whether motion of the first movable arm, motion of the second movable arm, or motions of the first and second movable arms together will result in an undesirable relationship between the first and second movable arms; and

sending a first movement command to the first movable arm or the second movable arm to avoid the undesirable relationship.

15. The method of claim 14 wherein:

the common reference frame is a reference frame of the image capturing device; and

the method further comprises determining the first movable arm position in the common reference frame further based on the position and the orientation of the working end in the common reference frame.

16. The method of claim 14 wherein:

the common reference frame is a reference frame of the computer-assisted surgical device; and

the method further comprises;

determining the second movable arm position in a reference frame of the image capturing device;

determining the position and the orientation of the working end in the reference frame of the image capturing device; and

transforming the second movable arm position and the position and orientation of the working end from the reference frame of the image capturing device to the common reference frame.

17. The method of claim 14 wherein:

the first plurality of images are a plurality of two-dimensional images from a image capturing device; and

the method further comprises determining the position and the orientation of the working end from the two-dimensional images even when the working end of the manipulatable device is occluded by one or more objects disposed between the image capturing device and the working end of the manipulatable device.

18. The method of claim 14, further comprising determining the first movement command so as to minimize a cost function.

19. A movement control system comprising:

a tracking system; and

a controller coupled to the tracking system, the controller comprising;

one or more processors; and

memory coupled to the one or more processors;

wherein;

the controller is coupled to a computer-assisted surgical device having a first movable arm coupled to a manipulatable device having a working end;

the controller is further coupled to a second movable arm coupled to an image capturing device separate from the computer-assisted surgical device;

the controller is configured to;

receive first kinematic data for the first movable arm;

receive second kinematic data for the second movable arm;

receive first tracking data for the first movable arm from the tracking system;

receive second tracking data for the second movable arm from the tracking system;

determine, based on the first kinematic data and the first tracking data, a first location of the first movable arm;

determine, based on the second kinematic data and the second tracking data, a second location of the second movable arm;

send a first movement command to the second movable arm that directs the second movable arm to move into a first set-up position to capture images of at least a portion of the manipulatable device while maintaining a first safety margin between the first movable arm and the second movable arm, the first movement command being based on the first location and the second location;

receive third kinematic data for the first movable arm;

receive fourth kinematic data for the second movable arm;

receive one or more first images from the image capturing device, the one or more first images capturing at least a portion of the manipulatable device;

determine, based on the third kinematic data and the one or more first images, a third location of the first movable arm;

determine, based on the fourth kinematic data and the one or more first images, a fourth location of the second movable arm; and

send a second movement command to the second movable arm that directs the second movable arm to move into a second set-up position, different from the first set-up position, to capture images of a region of interest while maintaining a second safety margin between the first movable arm and the second movable arm, the second movement command being based on the third location and the fourth location.

20. The movement control system of claim 19 wherein the controller is further configured to:

receive fifth kinematic data for the first movable arm;

receive sixth kinematic data for the second movable arm;

receive one or more second images from the image capturing device, the one or more second images capturing at least a portion of the working end;

determine, based on the fifth kinematic data and the one or more second images, a fifth location of the first movable arm;

determine, based on the sixth kinematic data and the one or more second images, a sixth location of the second movable arm; and

send a third movement command to the second movable arm that directs the second movable arm to move into a third set-up position, different from the first set-up position and the second set-up position, to capture images of the working end while maintaining a third safety margin between the first movable arm and the second movable arm, the third movement command being based on the fifth location and the sixth location.