Controlling a surgical navigation system

• US 9,542,001 B2
• Filed: 01/14/2010
• Issued: 01/10/2017
• Est. Priority Date: 01/14/2010
• Status: Active Grant

• Current Assignee: Brainlab AG
• Original Assignee: Brainlab AG
• Inventors: Birkenbach, Rainer, Urban, Alexander, Druse, Alexander, Christian, Georg, Lechner, Christian, Besinger, Axel, Schaal, Alexander, Gerken, Till
• Primary Examiner(s): Amin, Bhavesh V
• Application Number:
  US13/519,400
• Publication Number:
  US 20120323364A1
• Time in Patent Office: 2,553 Days
• US Class Current: 1/1
• CPC Class Codes: A61B 2017/00115 with audible or visual output with imaging means A61B90/37, A61B34/20
• CPC Class Codes: A61B 2017/00207 with hand gesture control or hand gesture recognition
• CPC Class Codes: A61B 2017/00216 with eye tracking or head position tracking control
• CPC Class Codes: A61B 2034/2048 using an accelerometer or inertia sensor
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2034/741 Glove like input devices, e.g. "data gloves"
• CPC Class Codes: A61B 2034/742 Joysticks
• CPC Class Codes: A61B 2090/372 Details of monitor hardware
• CPC Class Codes: A61B 2090/3937 Visible markers
• CPC Class Codes: A61B 2090/3945 Active visible markers, e.g. light emitting diodes
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 34/74 Manipulators with manual electric input means
• CPC Class Codes: A61B 90/39 Markers, e.g. radio-opaque or breast lesions markers
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0338 with detection of limited linear or angular displacement of an operating part of the device from a neutral position, e.g. isotonic or isometric joysticks
• CPC Class Codes: G06F 3/0346 : with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

The invention relates to a control system for a surgical navigation system, wherein the navigation system is associated with a detection unit for detecting system control inputs made by a user, and wherein the navigation system is controlled in accordance with the control inputs and the detection unit comprises a gesture detection for detecting user gestures. The invention also relates to a method for controlling a surgical navigation system, wherein a detection unit for detecting system control inputs made by a user is associated with the surgical navigation system, the navigation system is controlled in accordance with the control inputs, and user gestures are used as said control inputs.

Claims

1. A control system for a surgical navigation system, the control system comprising:

a detection unit operably coupled with the surgical navigation system, the detection unit configured to detect system control inputs made by an associated user, and the detection unit comprising;

a gesture detection system configured to detect user gestures as control inputs; and

a hand-held gesture support comprising a shaft and a tip, the hand-held gesture support being freely movable within three dimensional space, wherein the gesture detection system is adapted to;

detect gestures augmented by the hand-held gesture support in a first mode when the hand-held gesture support is moved, the movements being recognized by the gesture detection system as a first type of control inputs;

detect gestures augmented by the hand-held gesture support in a second mode being different from the first mode, when the hand-held gesture support is moved in a manner of joystick movements, the joystick movements being made with the tip of the hand-held gesture being fixed at a point while moving the shaft of the hand-held gesture support, the joystick movements being recognized by the gesture detection system as a second type of control inputs being different from the first type of control inputs; and

process the first and the second types of detected gestures in order to control operation of the surgical navigation system.

2. The control system for the surgical navigation system according to claim 1, wherein the hand-held gesture support is a device which is manipulated by the associated user.

3. The control system for the surgical navigation system according to claim 1, wherein the hand-held gesture support comprises at least one feature selected form the group consisting of:

a medical instrument bearing markings or identifications which can be detected by the gesture detection system;

an acceleration sensor or gyro-sensor;

markings or identifications such as codes, patterns, colorings or tracking markers which can be detected by the gesture detection system, wherein said hand-held gesture support is a hand-held display unit which is separately provided for displaying the markings and identifications, or an instrument which is used in the surgical environment and additionally bears such markings or identifications.

4. The control system for the surgical navigation system according to claim 1, wherein the detection unit comprises at least one feature selected from the group consisting of:

a camera system for visible light;

a camera system for the invisible light spectrum;

a tracking system such as a time-of-flight camera, a laser scanner, a heat tracking system or heat camera system or a magnetic or electromagnetic tracking system.

19. The control system for the surgical navigation system according to claim 2, wherein the hand-held gesture support is a separate or integrated device.

20. The control system for the surgical navigation system according to claim 3, wherein the instrument which is used in the surgical environment and additionally bears such markings or identifications is a sterile pen or pointing device.

21. The control system for the surgical navigation system according to claim 4, wherein the camera system for visible light is a video camera system, or, the camera system for the invisible light spectrum is an infrared camera system.

5. A computer implemented method for controlling a surgical navigation system, the method comprising:

associating a detection unit for detecting system control inputs made by an associated user with the surgical navigation system by a computer;

receiving a first type of control inputs at one or more input ports of the computer, wherein the control inputs are in a form of a first type of user gestures, wherein a hand-held gesture support is moved by the associated user in order to augment the first type of user gesture, the movement being recognized as a first type of control inputs by the gesture detection system;

receiving a second type of control inputs, being different from the first type of control inputs, at one or more input ports of the computer, wherein the control inputs are in a form of a second type of user gestures, wherein the hand-held gesture support, comprising a tip and a shaft, that is freely movable within three dimensional space and which is moved by the associated user in order to augment a second type of user gesture, being different from the first type of user gesture, and wherein a gesture detection system is adapted to detect gestures augmented by the hand-held gesture support when the hand-held gesture support is moved in a manner of joystick movement, the joystick movement being made with the hand-held gesture support tip being fixed and moving the hand-held gesture support shaft, the joystick movement being recognized as the second type of control inputs by the gesture detection system; and

processing the first and second types of control inputs in order to control the surgical navigation system.

6. The method according to claim 5, wherein augmented user gestures are detected by the detection unit while the detection unit performs computer-assisted gesture detection.

7. The method according to claim 5, wherein the hand-held gesture support is a device configured to be manipulated by the associated user.

8. The method according to claim 5, wherein the hand-held gesture support comprises at least one feature selected form the group consisting of:

a medical instrument bearing markings or identifications which can be detected by the gesture detection system;

an acceleration sensor or gyro-sensor;

markings or identifications such as codes, patterns, colorings or tracking markers which can be detected by the gesture detection system, wherein said hand-held gesture support is a hand-held display unit which is separately provided for displaying the markings and identifications, or an instrument which is used in the surgical environment and additionally bears such markings or identifications.

9. The method according to claim 5, wherein a gesture is made using a combination of the hand-held gesture support and a marking attached to or associated with the associated user'"'"'s hand, wherein the combined location or movement of the markings on the hand-held gesture support and the attached markings constitute the gesture.

10. The method according to claim 5, further comprising detecting the gestures by the detection unit, wherein the detection unit comprises at least one feature selected from the group consisting of:

a camera system for visible light;

a camera system for the invisible light spectrum;

a tracking system such as a time-of-flight camera, a laser scanner, a heat tracking system or heat camera system or a magnetic or electromagnetic tracking system.

11. The method according to claim 5, further comprising detecting, by the gesture detection system, an initiating user gesture, carried out with the hand-held gesture support, the initiating user gesture comprising drawing a certain shape which is recognized by the navigation system, whereupon

the control input is associated with the user gesture, or

an interaction mode is started on the surgical navigation system in which at least one further user gesture is identified.

12. The method according to claim 5, further comprising defining, by the gesture or the initiating user gesture, a virtual volume surrounding limits of the initiating gesture, and wherein further predetermined gestures made in the virtual volume are recognized as a control input.

13. The method according to claim 12, further comprising defining the virtual volume in a sterile working field of the associated user.

14. The method according to claim 5, further comprising augmenting the user gesture with a user support comprising an elongate navigated medical instrument.

15. The method according to claim 5, further comprising distinguishing the control inputs by their angle of inclination with respect to a defined virtual plane including the hand-held gesture support tip.

16. The method according to claim 5, further comprising starting or initiating a joystick interaction mode on the navigation system when the hand-held gesture support enters a predefined region or follows a predefined motion.

17. The method according to claim 5, further comprising generating a feedback by the navigation system as to the current interaction or gesture driven control input, the feedback comprising at least one of:

a visual feedback;

or

an audible feedback.

22. The method according to claim 7, wherein the hand-held gesture support is a separate or integrated device.

23. The method according to claim 8, wherein the instrument which is used in the surgical environment and additionally bears such markings or identifications is a sterile pen or pointing device.

24. The method according to claim 10, wherein the camera system for visible light is a video camera system.

25. The method according to claim 11, wherein the hand-held gesture support is a navigated medical instrument, and, the drawing a certain shape which is recognized by the navigation system comprises drawing a geometrical shape.

26. The method according to claim 17, wherein the visual feedback is a menu or a highlighted element on a navigation screen.

27. The method according to claim 10, wherein the camera system for the invisible light spectrum is an infrared camera system.

28. The method according to claim 17, wherein the audible feedback is a confirmation sound.

18. A non-transitory computer readable medium storing a computer program for controlling a surgical navigation system which, when running on a computer or is loaded onto a computer, causes the computer to:

associate a detection unit for detecting system control inputs made by an associated user with the surgical navigation system by a computer;

receive a first type of control inputs at one or more input ports of the computer, wherein the control inputs are in a form of a first type of user gestures, wherein a hand-held gesture support is moved by the associated user in order to augment the first type of user gesture, the movement being recognized as a first type of control inputs by the gesture detection system;

receive a second type of control inputs, being different from the first type of control inputs, at one or more input ports of the computer, wherein control inputs are in a form of user gestures, wherein the hand-held gesture support, comprising a tip and a shaft, that is freely movable within three-dimensional space together with the associated user'"'"'s body is moved by the associated user in order to augment a second type of user gesture being different from the first type of user gesture, and wherein a gesture detection system is adapted to detect gestures augmented by movements of the hand-held gesture support when the hand-held gesture support is moved in the manner of joystick movements, the joystick movements being made with the hand-held gesture support tip being fixed and moving the hand-held gesture support shaft, the joystick movements being recognized as the second type of control inputs by the gesture detection system; and

process the first and the second types of control inputs in order to control the surgical navigation system.