System and method for controlling a remote medical device guidance system in three-dimensions using gestures

• US 9,439,736 B2
• Filed: 12/03/2012
• Issued: 09/13/2016
• Est. Priority Date: 07/22/2009
• Status: Active Grant

• Current Assignee: St. Jude Medical (Abbott Laboratories Incorporated)
• Original Assignee: St. Jude Medical (Abbott Laboratories Incorporated)
• Inventors: Olson, Eric S.
• Primary Examiner(s): Nguyen, Bao Long T
• Application Number:
  US13/692,356
• Publication Number:
  US 20130096575A1
• Time in Patent Office: 1,380 Days
• Field of Search: 700/250, 700/253, 700257-259, 700/264, 901 2- 3, 901 46- 47, 901/50, 318/568.11, 318/568.13, 318/568.16, 318/568.23, 318/568.25, 715/863
• US Class Current: 1/1
• CPC Class Codes: A61B 2034/258 providing specific settings for specific users
• CPC Class Codes: A61B 2034/741 Glove like input devices, e.g. "data gloves"
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/74 Manipulators with manual electric input means
• CPC Class Codes: A61B 34/76 Manipulators having means for providing feel, e.g. force or tactile feedback
• CPC Class Codes: A61B 5/283 Invasive
• CPC Class Codes: B25J 9/161 Hardware, e.g. neural networks, fuzzy logic, interfaces, processor
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2210/41 Medical
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A system for enabling a user to remotely control a robotic medical device system includes a motion capture apparatus to capture motion of a user in a sensing volume and generate indicative output data. The system includes a control unit configured to execute gesture recognition logic that recognizes a user gesture based on analysis of the indicative output data. The control unit executes interpreter logic that is configured to translate the recognized user gesture into a corresponding robotic medical device control command configured to control an aspect of the operation of the robotic medical device system.

Claims

1. A system for enabling a user to remotely control a robotic medical device system, comprising:

a motion capture apparatus configured to capture motion of a user in a sensing volume and generate output data indicative of the captured user motion, wherein said output data includes fiducial point tracking data, and wherein said fiducial point tracking data is associated with a plurality of fiducial points defined with respect to the user, and wherein said fiducial point tracking data includes, for each fiducial point, a respective position, and wherein each position includes a respective three-dimensional coordinate in a reference coordinate system, and wherein said fiducial point tracking data further includes, for each fiducial point, a respective time-based plurality of positions;

an electronic control unit including a processor and a memory;

gesture recognition logic stored in said memory and configured to execute on said processor, said gesture recognition logic being configured to recognize a user gesture based on said output data from said motion capture apparatus; and

interpreter logic stored in said memory and configured to execute on said processor, said interpreter logic being configured to translate the user gesture to a corresponding robotic medical device control command wherein said command is configured to control an aspect of the operation of the robotic medical device system, said electronic control unit being configured to communicate said command to the robotic medical device system, and wherein said gesture recognition logic is configured to;

identify a start pose based on said fiducial point tracking data;

record the motion of a predetermined plurality of fiducial points after the start pose until an end pose is identified based on said fiducial point tracking data;

compare said recorded motion of the predetermined plurality of fiducial points with a plurality of predefined gestures; and

output the user gesture when said recorded motion matches one of the plurality of gestures.

2. The system of claim 1 wherein the start pose corresponds to a start condition where a first set of fiducial points assumes a first relationship therebetween, and the end pose corresponds to an end condition where a second set of fiducial points assumes a second relationship therebetween.

3. The system of claim 1 further comprising a user-actuatable switch coupled to said electronic control unit and having a normally open state, and a user-actuatable closed state, wherein said electronic control unit is configured to at least disable said communication of said command to the robotic medical device system unless said switch is in said closed state.

4. The system of claim 1 wherein said interpreter logic is further configured to selectively translate, based on a state of context switch parameter, the user gesture into one of (i) the robotic medical device control command configured to control an aspect of the operation of the robotic medical device system and (ii) a mapping control command configured to control an aspect of an electro-anatomic mapping system.

5. The system of claim 4 wherein the sensing volume includes a context switching portion, said system for enabling the user to remotely control further comprising context switching logic stored in said memory and configured for execution by said processor and configured to detect a predetermined context switching gesture based on said output data from said motion capture apparatus originating with user-motion occurring in the context switching portion of the sensing volume.

6. The system of claim 1 wherein said output data further includes acquired imaging of the movements of the user.

7. The system of claim 1 wherein said motion capture apparatus comprises an optical sub-system configured to optically detect the motion of the user in the sensing volume.

8. The system of claim 1 wherein said robotic medical device system includes a medical device comprising one of a catheter and a sheath, and wherein said robotic medical device system further includes a manipulator assembly including at least one electrically-operated actuation unit configured for one of translation, deflection and rotation of said medical device.

9. A system for enabling a user to remotely control a robotic medical device system, comprising:

a motion capture apparatus configured to capture motion of a user in a sensing volume and generate output data indicative of the captured user motion, wherein said output data includes fiducial point tracking data, and wherein said fiducial point tracking data is associated with a plurality of fiducial points defined with respect to the user, and wherein said fiducial point tracking data includes, for each fiducial point, a respective position;

an electronic control unit including a processor and a memory;

gesture recognition logic stored in said memory and configured to execute on said processor, said gesture recognition logic being configured to recognize a user gesture based on said output data from said motion capture apparatus; and

interpreter logic stored in said memory and configured to execute on said processor, said interpreter logic being configured to translate the user gesture to a corresponding robotic medical device control command wherein said command is configured to control an aspect of the operation of the robotic medical device system, said electronic control unit being configured to communicate said command to the robotic medical device system, wherein a characteristic associated said robot medical device control command is a commanded magnitude associated with an action involving one of a catheter and a sheath under control of the robotic medical device system, and wherein the commanded magnitude corresponds to a distance between preselected fiducial points.

10. The system of claim 9 wherein the action is selected from the group comprising a catheter extension, a catheter retraction, a sheath extension, and a sheath retraction.

11. The system of claim 9 wherein said robotic medical device system includes a medical device comprising one of a catheter and a sheath, and wherein said robotic medical device system further includes a manipulator assembly including at least one electrically-operated actuation unit configured for one of translation, deflection and rotation of said medical device.

12. A system for enabling a user to remotely control a robotic medical device system, comprising:

a motion capture apparatus configured to capture motion of a user in a sensing volume and generate output data indicative of the captured user motion, wherein said output data includes fiducial point tracking data, wherein said fiducial point tracking data is associated with a plurality of fiducial points defined with respect to the user, and wherein said fiducial point tracking data includes, for each fiducial point, a respective position;

an electronic control unit including a processor and a memory;

gesture recognition logic stored in said memory and configured to execute on said processor, said gesture recognition logic being configured to recognize a user gesture based on said output data from said motion capture apparatus; and

interpreter logic stored in said memory and configured to execute on said processor, said interpreter logic being configured to translate the user gesture to a corresponding robotic medical device control command wherein said command is configured to control an aspect of the operation of the robotic medical device system, said electronic control unit being configured to communicate said command to the robotic medical device system;

wherein a characteristic associated with the robot medical device control command is a commanded rotation associated with an action involving one of a catheter and a sheath under control of the robotic medical device system, and wherein the commanded rotation corresponds to a rotation angle through which a preselected fiducial point is rotated during the user gesture.

13. The system of claim 12 wherein said robotic medical device system includes a medical device comprising one of a catheter and a sheath, and wherein said robotic medical device system further includes a manipulator assembly including at least one electrically-operated actuation unit configured for one of translation, deflection and rotation of said medical device.

14. A system for enabling a user to remotely control a robotic medical device system, comprising:

a motion capture apparatus configured to capture motion of a user in a sensing volume and generate output data indicative of the captured user motion;

an electronic control unit including a processor and a memory;

gesture recognition logic stored in said memory and configured to execute on said processor, said gesture recognition logic being configured to recognize a user gesture based on said output data from said motion capture apparatus; and

interpreter logic stored in said memory and configured to execute said processor, said interpreter logic being configured to translate the user gesture to a corresponding robotic medical device control command wherein said command is configured to control an aspect of the operation of the robotic medical device system, said electronic control unit being configured to communicate said command to the robotic medical device system, wherein said interpreter logic is further configured to selectively translate, based on a state of context switch parameter, the user gesture into one of (i) the robotic medical device control command configured to control an aspect of the operation of the robotic medical device system and (ii) a mapping control command configured to control an aspect of an electro-anatomic mapping system, and wherein said interpreter logic is further configured to generate said robotic medical device control command with respect to the visible orientation of the then-visible view of an anatomical model of at least a portion of a body of a patient produced by the electro-anatomic mapping system.

15. The system of claim 14 wherein said robotic medical device system includes a medical device comprising one of a catheter and a sheath, and wherein said robotic medical device system further includes a manipulator assembly including at least one electrically-operated actuation unit configured for one of translation, deflection and rotation of said medical device.

16. A system for enabling a user to remotely control a robotic medical device system, comprising:

motion capture means for capturing motion of a user in a sensing volume and generating output data indicative of the captured user motion;

gesture recognition means for recognizing a user gesture based on said output data from said motion capture means;

interpreter means for translating the user gesture to a corresponding robotic medical device control command wherein said command is configured to control an aspect of the operation of the robotic medical device system; and

communication means for communicating said command to the robotic medical device system, and wherein said output data includes fiducial point tracking data associated with a plurality of fiducial points defined with respect to the user, and wherein said fiducial point tracking data includes, for each fiducial point, a respective position, wherein each position includes a respective three-dimensional coordinate in a reference coordinate system, and wherein said fiducial point tracking data further includes, for each fiducial point, a respective time-based plurality of positions, and wherein said gesture recognition means is further configured for;

identifying a start pose based on said fiducial point tracking data;

recording the motion of a predetermined plurality of fiducial points after the start pose until an end pose is identified based on said fiducial point tracking data;

comparing said recorded motion of the predetermined plurality of fiducial points with a plurality of predefined gestures; and

outputting the user gesture when said recorded motion matches one of the plurality of gestures.

17. The system of claim 16 wherein said robotic medical device system includes a medical device comprising one of a catheter and a sheath, and wherein said robotic medical device system further includes a manipulator assembly including at least one electrically-operated actuation unit configured for one of translation, deflection and rotation of said medical device.