Systems and user interfaces for dynamic interaction with two- and three-dimensional medical image data using hand gestures

• US 9,606,584 B1
• Filed: 06/29/2015
• Issued: 03/28/2017
• Est. Priority Date: 07/01/2014
• Status: Active Grant

• Current Assignee: Merative US LP
• Original Assignee: DR Systems Incorporated (International Business Machines Corporation)
• Inventors: Fram, Evan K.
• Primary Examiner(s): Mushambo, Martin
• Application Number:
  US14/754,178
• Time in Patent Office: 638 Days
• US Class Current: 1/1
• CPC Class Codes: A61B 2017/00207 with hand gesture control or hand gesture recognition
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 90/36 Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2203/04802 3D-info-object: information is displayed on the internal or external surface of a three dimensional manipulable object, e.g. on the faces of a cube that can be rotated by the user
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2219/008 Cut plane or projection plane definition
• CPC Class Codes: G06T 2219/028 Multiple view windows (top-side-front-sagittal-orthogonal)
• CPC Class Codes: G16H 10/60 : for patient-specific data, e.g. for electronic patient records
• CPC Class Codes: G16H 40/63 : for local operation

Abstract

Embodiments of the present disclosure relate to systems and techniques for accessing data stores of medical images and displaying the medical images in substantially real-time to provide information in an interactive user interface. Systems are disclosed that may advantageously provide highly efficient, intuitive, and rapid dynamic interaction with two- and three-dimensional medical image data using hand gestures. The systems may include interactive user interfaces that are dynamically updated to provide tracking of a user'"'"'s hand in a virtual 3D space by two- and/or three-dimensional image data. A user may use the systems described herein to more quickly, thoroughly, and efficiently interact with image data including two-dimensional images, three-dimensional image data, and/or series of image data, as compared to previous systems.

Claims

1. A medical image computing system comprising:

an electronic display;

one or more sensors configured to detect a hand of a user;

a storage device configured to store electronic software instructions;

one or more data stores storing at least one set of volumetric medical image data; and

one or more computer processors in communication with the electronic display, the one or more sensors, the storage device, and the one or more data stores, the one or more computer processors configured to execute the stored software instructions to cause the computing system to;

access, from the one or more data stores, a set of volumetric medical image data;

render a three-dimensional view of the set of volumetric medical image data;

generate user interface data for rendering an interactive user interface on the electronic display, the interactive user interface including at least the rendered three-dimensional view of the set of volumetric medical image data;

determine a virtual origin location in physical space about which a user may move their hand;

receive sensor data from the one or more sensors, the sensor data indicative of a user input provided via the hand of the user, the user input comprising at least a position of the hand;

determine, based on the sensor data, the position of the hand with respect to the virtual origin location;

calculate, based on the position of the hand with respect to the virtual origin location, a rotation of the set of volumetric medical image data;

render, based on the rotation, an updated three-dimensional view of the set of volumetric medical image data; and

update the user interface data to include the updated three-dimensional view of the set of volumetric medical image data.

2. The medical image computing system of claim 1, wherein the virtual origin location is determined in reference to at least one of the one or more sensors.

3. The medical image computing system of claim 2, wherein the at least one of the one or more sensors is located on a surface in front of and below the electronic display such that the virtual origin is located in front of the electronic display and the one or more sensors are configured to detect the hand of the user when the hand is positioned above the sensor and in front of the electronic display.

4. The medical image computing system of claim 2, wherein calculating the rotation of the set of volumetric medical image data comprises:

defining a first three-dimensional coordinate system of the at least one of the one or more sensors, wherein the virtual origin is at an origin of the first three-dimensional coordinate system;

determining x and z coordinates of the hand with reference to the first three-dimensional coordinate system;

calculating a first angle between a z-axis of the first three-dimensional coordinate system and the x and z coordinates of the hand with reference to the first three-dimensional coordinate system;

determining a y coordinate of the hand with reference to the first three-dimensional coordinate system;

converting the y coordinate to a value on a scale between a maximum value and a minimum value, wherein the maximum value and the minimum value are associated with respective minimum and maximum physical positions of the users hand along a y-axis of the first three-dimensional coordinate system;

calculating a second angle by taking an arcsin( ) of the value;

defining a second three-dimensional coordinate system of the rendered three-dimensional view of the set of volumetric medical image data;

rotating the rendered three-dimensional view of the set of volumetric medical image data by the first angle about a y-axis of the second three-dimensional coordinate system;

defining a third three-dimensional coordinate system of the electronic display; and

rotating the rendered three-dimensional view of the set of volumetric medical image data by the second angle about an x-axis of the third three-dimensional coordinate system.

5. The medical image computing system of claim 4, wherein the maximum value and minimum value are, respectively, at least one of:

−

0.99 and +0.99, −

0.999 and +0.999, or −

1 and +1.

6. The medical image computing system of claim 1, wherein the user input further comprises at least one of:

a motion of the hand, a rotation of the hand, an orientation of the hand, or a configuration of the hand.

7. The medical image computing system of claim 6, wherein:

the one or more data stores store a first set of volumetric medical image data and a second set of volumetric medical image data, and

the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to;

determine, based on the sensor data, the configuration of the hand of the user;

select, based on the configuration of the hand of the user, a one of the first set of volumetric medical image data and the second set of volumetric medical image data;

calculate, based on the position of the hand with respect to the virtual origin location, a first rotation of the one of the first set of volumetric medical image data and the second set of volumetric medical image data;

render, based on the first rotation, an updated three-dimensional view of the one of the first set of volumetric medical image data and the second set of volumetric medical image data; and

update the user interface data to include the updated three-dimensional view of the one of the first set of volumetric medical image data and the second set of volumetric medical image data.

8. The medical image computing system of claim 7, wherein:

the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to;

determine, based on the sensor data, a change in the configuration of the hand of the user;

select, based on the change in the configuration of the hand of the user, a second of the first set of volumetric medical image data and the second set of volumetric medical image data;

calculate, based on the position of the hand with respect to the virtual origin location, a second rotation of the second of the first set of volumetric medical image data and the second set of volumetric medical image data;

render, based on the second rotation, an updated three-dimensional view of the second of the first set of volumetric medical image data and the second set of volumetric medical image data; and

update the user interface data to include the updated three-dimensional view of the second of the first set of volumetric medical image data and the second set of volumetric medical image data.

9. The medical image computing system of claim 7, wherein the configuration of the hand of the user comprises at least one of:

a fist, one or more fingers extended, or a flat hand.

10. A computer-implemented method comprising:

by one or more computer processors executing software instructions;

accessing, from one or more data stores, a set of volumetric medical image data;

rendering a three-dimensional view of the set of volumetric medical image data;

generating user interface data for rendering an interactive user interface on an electronic display, the interactive user interface including at least the rendered three-dimensional view of the set of volumetric medical image data;

determining a virtual origin location in physical space about which a user may move their hand;

receiving sensor data from one or more sensors configured to detect the hand of the user, the sensor data indicative of a user input provided via the hand of the user, the user input comprising at least a position of the hand;

determining, based on the sensor data, the position of the hand with respect to the virtual origin location;

calculating, based on the position of the hand with respect to the virtual origin location, a rotation of the set of volumetric medical image data;

rendering, based on the rotation, an updated three-dimensional view of the set of volumetric medical image data; and

updating the user interface data to include the updated three-dimensional view of the set of volumetric medical image data.

11. The computer-implemented method of claim 10, wherein calculating the rotation of the set of volumetric medical image data comprises:

by the one or more computer processors executing software instructions;

defining a first three-dimensional coordinate system of the at least one of the one or more sensors, wherein the virtual origin is at an origin of the first three-dimensional coordinate system;

determining x and z coordinates of the hand with reference to the first three-dimensional coordinate system;

calculating a first angle between a z-axis of the first three-dimensional coordinate system and the x and z coordinates of the hand with reference to the first three-dimensional coordinate system;

determining a y coordinate of the hand with reference to the first three-dimensional coordinate system;

converting the y coordinate to a value on a scale between a maximum value and a minimum value, wherein the maximum value and the minimum value are associated with respective minimum and maximum physical positions of the users hand along a y-axis of the first three-dimensional coordinate system;

calculating a second angle by taking an arcsin( ) of the value;

defining a second three-dimensional coordinate system of the rendered three-dimensional view of the set of volumetric medical image data;

rotating the rendered three-dimensional view of the set of volumetric medical image data by the first angle about a y-axis of the second three-dimensional coordinate system;

defining a third three-dimensional coordinate system of the electronic display; and

rotating the rendered three-dimensional view of the set of volumetric medical image data by the second angle about an x-axis of the third three-dimensional coordinate system.

12. The computer-implemented method of claim 10, wherein the user input further comprises at least one of:

a motion of the hand, a rotation of the hand, an orientation of the hand, or a configuration of the hand.

13. The computer-implemented method of claim 12, wherein the one or more data stores store a first set of volumetric medical image data and a second set of volumetric medical image data, the computer-implemented method further comprising:

by the one or more computer processors executing software instructions;

determining, based on the sensor data, the configuration of the hand of the user;

selecting, based on the configuration of the hand of the user, a one of the first set of volumetric medical image data and the second set of volumetric medical image data;

calculating, based on the position of the hand with respect to the virtual origin location, a first rotation of the one of the first set of volumetric medical image data and the second set of volumetric medical image data;

rendering, based on the first rotation, an updated three-dimensional view of the one of the first set of volumetric medical image data and the second set of volumetric medical image data; and

updating the user interface data to include the updated three-dimensional view of the one of the first set of volumetric medical image data and the second set of volumetric medical image data.

14. The computer-implemented method of claim 13 further comprising:

by the one or more computer processors executing software instructions;

determining, based on the sensor data, a change in the configuration of the hand of the user;

selecting, based on the change in the configuration of the hand of the user, a second of the first set of volumetric medical image data and the second set of volumetric medical image data;

calculating, based on the position of the hand with respect to the virtual origin location, a second rotation of the second of the first set of volumetric medical image data and the second set of volumetric medical image data;

rendering, based on the second rotation, an updated three-dimensional view of the second of the first set of volumetric medical image data and the second set of volumetric medical image data; and

updating the user interface data to include the updated three-dimensional view of the second of the first set of volumetric medical image data and the second set of volumetric medical image data.

15. The computer-implemented method of claim 13, wherein the configuration of the hand of the user comprises at least one of:

a fist, one or more fingers extended, or a flat hand.

16. A non-transitory computer-readable storage medium having software instructions embodied therewith, the software instructions executable by one or more processors to cause the one or more processors to:

access, from one or more data stores, a set of volumetric medical image data;

render a three-dimensional view of the set of volumetric medical image data;

generate user interface data for rendering an interactive user interface on an electronic display, the interactive user interface including at least the rendered three-dimensional view of the set of volumetric medical image data;

determine a virtual origin location in physical space about which a user may move their hand;

receive sensor data from one or more sensors configured to detect the hand of the user, the sensor data indicative of a user input provided via the hand of the user, the user input comprising at least a position of the hand;

determine, based on the sensor data, the position of the hand with respect to the virtual origin location;

calculate, based on the position of the hand with respect to the virtual origin location, a rotation of the set of volumetric medical image data;

render, based on the rotation, an updated three-dimensional view of the set of volumetric medical image data; and

update the user interface data to include the updated three-dimensional view of the set of volumetric medical image data.

17. The non-transitory computer-readable storage medium of claim 16, wherein calculating the rotation of the set of volumetric medical image data comprises:

defining a first three-dimensional coordinate system of the at least one of the one or more sensors, wherein the virtual origin is at an origin of the first three-dimensional coordinate system;

determining x and z coordinates of the hand with reference to the first three-dimensional coordinate system;

calculating a first angle between a z-axis of the first three-dimensional coordinate system and the x and z coordinates of the hand with reference to the first three-dimensional coordinate system;

determining a y coordinate of the hand with reference to the first three-dimensional coordinate system;

converting the y coordinate to a value on a scale between a maximum value and a minimum value, wherein the maximum value and the minimum value are associated with respective minimum and maximum physical positions of the users hand along a y-axis of the first three-dimensional coordinate system;

calculating a second angle by taking an arcsin( ) of the value;

defining a second three-dimensional coordinate system of the rendered three-dimensional view of the set of volumetric medical image data;

rotating the rendered three-dimensional view of the set of volumetric medical image data by the first angle about a y-axis of the second three-dimensional coordinate system;

defining a third three-dimensional coordinate system of the electronic display; and

rotating the rendered three-dimensional view of the set of volumetric medical image data by the second angle about an x-axis of the third three-dimensional coordinate system.

18. The non-transitory computer-readable storage medium of claim 16, wherein the user input further comprises at least one of:

a motion of the hand, a rotation of the hand, an orientation of the hand, or a configuration of the hand.

19. The non-transitory computer-readable storage medium of claim 18, wherein the one or more data stores store a first set of volumetric medical image data and a second set of volumetric medical image data, and wherein the software instructions are further executable by one or more processors to cause the one or more processors to:

determine, based on the sensor data, the configuration of the hand of the user;

selecting, based on the configuration of the hand of the user, a one of the first set of volumetric medical image data and the second set of volumetric medical image data;

calculate, based on the position of the hand with respect to the virtual origin location, a first rotation of the one of the first set of volumetric medical image data and the second set of volumetric medical image data;

render, based on the first rotation, an updated three-dimensional view of the one of the first set of volumetric medical image data and the second set of volumetric medical image data; and

update the user interface data to include the updated three-dimensional view of the one of the first set of volumetric medical image data and the second set of volumetric medical image data.

20. The non-transitory computer-readable storage medium of claim 19, wherein the software instructions are further executable by one or more processors to cause the one or more processors to:

determine, based on the sensor data, a change in the configuration of the hand of the user;

select, based on the change in the configuration of the hand of the user, a second of the first set of volumetric medical image data and the second set of volumetric medical image data;

calculate, based on the position of the hand with respect to the virtual origin location, a second rotation of the second of the first set of volumetric medical image data and the second set of volumetric medical image data;

render, based on the second rotation, an updated three-dimensional view of the second of the first set of volumetric medical image data and the second set of volumetric medical image data; and

update the user interface data to include the updated three-dimensional view of the second of the first set of volumetric medical image data and the second set of volumetric medical image data.