Controlling a surgical navigation system

• US 10,064,693 B2
• Filed: 11/29/2016
• Issued: 09/04/2018
• Est. Priority Date: 01/14/2010
• Status: Active Grant

• Current Assignee: Brainlab AG
• Original Assignee: Brainlab AG
• Inventors: Birkenbach, Rainer, Urban, Alexander, Druse, Alexander, Besinger, Axel, Schaal, Alexander, Gerken, Till, Christian, Georg, Lechner, Christian
• Primary Examiner(s): Amin, Bhavesh V
• Application Number:
  US15/362,911
• Publication Number:
  US 20170071686A1
• Time in Patent Office: 644 Days
• Field of Search: None
• CPC Class Codes: A61B 2017/00115 with audible or visual output with imaging means A61B90/37, A61B34/20
• CPC Class Codes: A61B 2017/00207 with hand gesture control or hand gesture recognition
• CPC Class Codes: A61B 2017/00216 with eye tracking or head position tracking control
• CPC Class Codes: A61B 2034/2048 using an accelerometer or inertia sensor
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2034/741 Glove like input devices, e.g. "data gloves"
• CPC Class Codes: A61B 2034/742 Joysticks
• CPC Class Codes: A61B 2090/372 Details of monitor hardware
• CPC Class Codes: A61B 2090/3937 Visible markers
• CPC Class Codes: A61B 2090/3945 Active visible markers, e.g. light emitting diodes
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 34/74 Manipulators with manual electric input means
• CPC Class Codes: A61B 90/39 Markers, e.g. radio-opaque or breast lesions markers
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0338 with detection of limited linear or angular displacement of an operating part of the device from a neutral position, e.g. isotonic or isometric joysticks
• CPC Class Codes: G06F 3/0346 : with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

Provided is a method for controlling a surgical navigation system. A detection unit for detecting system control inputs made by a user is associated with a surgical navigation system, the navigation system is controlled in accordance with the control inputs, and user gestures are used as said control inputs. A gesture or a initiating user gesture defines a virtual volume surrounding the initiating gesture'"'"'s limits, and further predetermined gestures made in the virtual volume are recognized as a control input. Also provided is a control system for a surgical navigation system. The navigation system is associated with a detection unit for detecting system control inputs made by a user, the navigation system is controlled in accordance with the control inputs, and the detection unit includes a gesture detection system for detecting user gestures.

Claims

1. A method for controlling a surgical navigation system, the method including:

using user gestures as touch-free system control inputs;

associating a detection unit for detecting the touch-free system control inputs made by a user with a surgical navigation system;

controlling the navigation system in accordance with the control inputs;

moving a medical instrument by the user, wherein predetermined movements of the instrument are detected and identified by the detection unit as user gestures;

carrying out an initiating user gesture that comprises drawing, with a tip of the medical instrument, a geometrical shape in a free three-dimensional working field of the user above or near a patient;

recognizing the geometrical shape by the navigation system as the initiating user gesture and defining a virtual volume surrounding the shape drawn in a sterile working field of the user; and

starting an interaction mode on the navigation system in which at least one further predetermined gestures made in the virtual volume is identified and recognized as a control input.

2. The method according to claim 1, further including:

moving a gesture support, which is directly or indirectly attached to the user or a part of the user'"'"'s body and/or which is manipulated by the user, by the user to augment a user gesture; and

detecting, by a gesture detection unit while it performs computer-assisted gesture detection, augmented user gestures augmented by the gesture support.

3. The method according to claim 2, wherein a gesture support is used which features one or any combination of the following devices or characteristics:

a glove or part of a glove or strap-on device or clip-on device or ring;

a sterile, disposable glove or part of a glove or strap-on device or clip-on device or ring;

a glove or part of a glove or strap-on device or clip-on device or ring, comprising or bearing markings which can be detected by the gesture detection system;

a glove or part of a glove or strap-on device or clip-on device or ring, comprising or bearing black-and-white or color markings or identifications which identify one or more fingers;

a glove or part of a glove or strap-on device or clip-on device or ring, comprising or bearing patterns which can be detected by the gesture detection system;

an active light-emitting or passive reflective tracking marker or part of a tracking marker for a marker tracking system associated with the surgical navigation system;

a medical instrument bearing markings or identifications which can be detected by the gesture detection system;

an acceleration sensor or gyro-sensor;

a portable or hand-held support comprising markings and/or identifications in the form of codes, patterns, colorings or tracking markers which can be detected by the gesture detection system, wherein said support is a hand-held display unit which is separately provided for displaying the markings and identifications or an instrument which is used in the surgical environment and additionally bears such markings and/or identifications.

4. The method according to claim 2, wherein a gesture is made using a combination of a portable gesture support and a marking attached to or associated with the user'"'"'s hand, and wherein a combined location or movement of the markings on the gesture support and the attached markings constitute the gesture.

5. The method according to claim 1, wherein the gestures are detected by a detection unit comprising:

a camera system for visible light; and

/or

a camera system for the invisible light spectrum; and

/or

a tracking system in the form of a time-of-flight camera, a laser scanner, a heat tracking system or heat camera system or a magnetic or electromagnetic tracking system.

6. The method according to claim 1, wherein the initiating user gesture, carried out with a gesture support comprises drawing a certain shape, which is recognized by the navigation system, whereupon

a control input is associated with the user gesture, and/or

an interaction mode is started on the navigation system, in which at least one further user gesture is identified.

7. The method according to claim 2, wherein the gesture support is an elongate navigated medical instrument, and wherein the instrument is moved in the manner of a joystick, the joystick movements being recognized as control inputs.

8. The method according to claim 7, wherein the instrument has a tip and a shaft portion, the joystick movements being made with the tip being fixed and the shaft portion moving.

9. The method according to claim 7, wherein the control inputs are distinguished by their angle of inclination with respect to a defined virtual plane including the instrument tip.

10. The method according to claim 7, wherein a joystick interaction mode is started or defined on the navigation system when the instrument enters a predefined region or follows a predefined motion.

11. The method according to claim 1, wherein a visual feedback in the form of a menu or a highlighted element on a navigation screen and/or an audible feedback in the form of a confirmation sound is given by the navigation system as to the current interaction or gesture driven control input.

12. A non-transitory computer-readable program storage medium storing a computer program for controlling a surgical navigation system, which, when running on a computer or loaded onto the computer, causes the computer to:

use user gestures as touch-free system control inputs;

associate a detection unit for detecting the touch-free system control inputs made by a user with a surgical navigation system;

control the navigation system in accordance with the control inputs;

detect and identify, by the detection unit as user gestures, predetermined movements of the instrument when the instrument is moved by the user;

recognize a geometrical shape by the navigation system as an initiating user gesture and define a virtual volume surrounding the shape drawn in a sterile working field of the user, where the initiating user gesture is carried out by drawing, with a tip of the medical instrument, the geometrical shape in a free three-dimensional working field of the user above or near a patient; and

start an interaction mode on the navigation system in which at least one further predetermined gestures made in the virtual volume is identified and recognized as a control input.

13. A control system for a surgical navigation system, the control system comprising:

a detection unit associated with the navigation system and configured to detect touch-free system control inputs made by a user and the navigation system configured to be controlled in accordance with the control inputs, the detection unit comprising;

a gesture detection system for detecting user gestures; and

a camera system for visible light; and

/or

a camera system for the invisible light spectrum; and

/or

a tracking system in the form of a time-of-flight camera, a laser scanner, a heat tracking system or heat camera system or a magnetic or electromagnetic tracking system;

wherein a medical instrument is moved by the user,

wherein predetermined movements of the instrument are detected and identified by the detection unit as user gestures,

wherein an initiating user gesture is carried out that comprises drawing, with the tip of the medical instrument, a geometrical shape in the free three-dimensional working field of the user above or near a patient,

wherein the shape is recognized by the navigation system as the initiating user gesture and defines a virtual volume surrounding the shape drawing in a sterile working field of the user, and

wherein an interaction mode is started on the navigation system in which at least one further predetermined gestures made in the virtual volume is identified and recognized as a control input.

14. The control system for a surgical navigation system according to claim 13, wherein the control system further comprises a gesture support which is moved by the user and designed to augment a user gesture, and wherein the gesture detection system is adapted to detect gestures augmented by the gesture support.

15. The control system for a surgical navigation system according to claim 14, wherein the gesture support is directly or indirectly attached to the user or a part of the user'"'"'s body.

16. The control system for a surgical navigation system according to claim 14, wherein the gesture support is a separate or integrated device which is manipulated by the user.

17. The control system for a surgical navigation system according to claim 15, wherein the gesture support features one or any combination of the following devices or characteristics:

a glove or part of a glove or strap-on device or clip-on device or ring;

a sterile, disposable glove or part of a glove or strap-on device or clip-on device or ring;

a glove or part of a glove or strap-on device or clip-on device or ring, comprising or bearing markings which can be detected by the gesture detection system;

a glove or part of a glove or strap-on device or clip-on device or ring, comprising or bearing black-and-white or color markings or identifications which identify one or more fingers;

a glove or part of a glove or strap-on device or clip-on device or ring, comprising or bearing patterns which can be detected by the gesture detection system;

an active light-emitting or passive reflective tracking marker or part of a tracking marker for a marker tracking system associated with the surgical navigation system;

a medical instrument bearing markings or identifications which can be detected by the gesture detection system;

an acceleration sensor or gyro-sensor;

a portable or hand-held support comprising markings and/or identifications in the form of codes, patterns, colorings or tracking markers which can be detected by the gesture detection system, wherein said support is a hand-held display unit which is separately provided for displaying the markings and identifications or an instrument which is used in the surgical environment and additionally bears such markings and/or identifications.