Systems and user interfaces for dynamic interaction with two-and three-dimensional medical image data using hand gestures

• US 10,229,753 B2
• Filed: 02/14/2017
• Issued: 03/12/2019
• Est. Priority Date: 07/01/2014
• Status: Active Grant

• Current Assignee: Merative US LP
• Original Assignee: DR Systems Incorporated (International Business Machines Corporation)
• Inventors: Fram, Evan K.
• Primary Examiner(s): Mushambo, Martin
• Application Number:
  US15/432,764
• Publication Number:
  US 20170161448A1
• Time in Patent Office: 756 Days
• Field of Search: None
• CPC Class Codes: A61B 2017/00207 with hand gesture control or hand gesture recognition
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 90/36 Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2203/04802 3D-info-object: information is displayed on the internal or external surface of a three dimensional manipulable object, e.g. on the faces of a cube that can be rotated by the user
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2219/008 Cut plane or projection plane definition
• CPC Class Codes: G06T 2219/028 Multiple view windows (top-side-front-sagittal-orthogonal)
• CPC Class Codes: G16H 10/60 : for patient-specific data, e.g. for electronic patient records
• CPC Class Codes: G16H 40/63 : for local operation

Abstract

Embodiments of the present disclosure relate to systems and techniques for accessing data stores of medical images and displaying the medical images in substantially real-time to provide information in an interactive user interface. Systems are disclosed that may advantageously provide highly efficient, intuitive, and rapid dynamic interaction with two- and three-dimensional medical image data using hand gestures. The systems may include interactive user interfaces that are dynamically updated to provide tracking of a user'"'"'s hand in a virtual 3D space by two- and/or three-dimensional image data. A user may use the systems described herein to more quickly, thoroughly, and efficiently interact with image data including two-dimensional images, three-dimensional image data, and/or series of image data, as compared to previous systems.

Claims

1. A medical image computing system comprising:

an electronic display;

one or more sensors configured to detect a hand of a user;

one or more data stores storing a plurality of medical exams, each medical exam including one or more image series, each image series including one or more medical images, each image series associated with an imaging plane

a storage device configured to store electronic software instructions; and

one or more computer processors in communication with the electronic display, the one or more sensors, and the storage device, the one or more computer processors configured to execute the stored software instructions to cause the computing system to;

detect, by the one or more sensors, a first user input including at least an orientation, a position, and a configuration of the hand;

determine, based on the configuration of the hand, a first medical exam;

determine, based on the orientation of the hand, a first imaging plane;

determine, based on the position of the hand, a first image series of the first medical exam, wherein the first image series is associated with the first imaging plane;

determine, based on the position of the hand, a first medical image of the first image series;

generate user interface data for rendering an interactive user interface on the electronic display, the interactive user interface including a display of the first medical image;

detect, by the one or more sensors, a second user input including a change in at least one of the orientation, the position, or the configuration of the hand; and

in response to the second user input;

select a second medical image; and

update the user interface data such that the interactive user interface includes a display of the second medical image,

wherein, in response to the second user input comprising a change in the configuration of the hand, the second medical image is selected from a second medical exam such the that second medical image displays a same anatomical position in a same imaging plane as the first medical image, and the second medical image is selected from a second image series of the second medical exam of a same image series type as the first image series.

2. A medical image computing system comprising:

an electronic display;

one or more sensors configured to detect a hand of a user;

a storage device configured to store electronic software instructions;

one or more data stores storing a plurality of medical exams, each medical exam including at least one of volumetric image data or one or more image series, each image series including one or more medical images, each image series associated with an imaging plane; and

one or more computer processors in communication with the electronic display, the one or more sensors, the storage device, and the one or more data stores, the one or more computer processors configured to execute the stored software instructions to cause the computing system to;

determine, based on the hand of the user as detected by the one or more sensors;

a plane of the hand of the user, and

a position of the hand of the user;

determine, based on the plane of the hand of the user, a first imaging plane;

determine, based on the position of the hand of the user, a first image series type associated with the first imaging plane;

select a first image series of the one or more image series of a first medical exam, the first image series having the first image series type;

select, further based on the position of the hand of the user, a first medical image of the first image series; and

generate user interface data for rendering an interactive user interface on the electronic display, the interactive user interface including the first medical image.

3. The medical image computing system of claim 2, wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to:

detect, by the one or more sensors, a motion of the hand of the user along the plane of the hand of the user;

determine an updated position of the hand of the user;

determine, based on the updated position of the hand of the user, a second image series type associated with the first imaging plane;

select a second image series of the one or more image series of the first medical exam, the second image series having the second image series type;

select, further based on the updated position of the hand of the user, a second medical image of the second image series; and

update the user interface data such that the interactive user interface includes the second medical image.

4. The medical image computing system of claim 2, wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to:

detect, by the one or more sensors, a motion of the hand of the user along an axis perpendicular to the plane of the hand of the user;

select, based on the motion of the hand of the user, a second medical image of the first image series; and

update the user interface data such that the interactive user interface includes the second medical image.

5. The medical image computing system of claim 2, wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to:

detect, by the one or more sensors, a motion of the hand of the user;

determine;

an updated plane of the hand of the user, and

an updated position of the hand of the user;

determine, based on the updated plane of the hand of the user, a second imaging plane;

determine, based on the updated position of the hand of the user, a second image series type associated with the second imaging plane;

select a second image series of the one or more image series of a first medical exam, the second image series having the second image series type;

select, further based on the position of the hand of the user, a second medical image of the second image series; and

update the user interface data such that the interactive user interface includes the second medical image.

6. The medical image computing system of claim 2, wherein determining the first image series type associated with the first imaging plane comprises:

accessing a hanging protocol associated with the user; and

determining the first series type by mapping the position of the hand of the user to the hanging protocol.

7. The medical image computing system of claim 2, wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to:

detect, by the one or more sensors, a configuration of the hand of the user;

select, based on the configuration of the hand of the user and the position of the hand of the user, an operation to perform on the first medical image;

detect, by the one or more sensors, a motion of the hand of the user while in the configuration; and

implement the operation on the first medical image in proportion to the motion of the hand of the user.

8. The medical image computing system of claim 7, wherein the operation is at least one of:

an adjustment of a window, an adjustment of a level, an adjustment of a magnification, or a pan adjustment.

9. A computer-implemented method comprising:

by one or more processors executing program instructions;

communicating with one or more data stores storing a plurality of medical exams, each medical exam including at least one of volumetric image data or one or more image series, each image series including one or more medical images, each image series associated with an imaging plane;

determining, based on the hand of the user as detected by one or more sensors configured to detect a hand of a user;

a plane of the hand of the user, and

a position of the hand of the user;

determining, based on the plane of the hand of the user, a first imaging plane;

determining, based on the position of the hand of the user, a first image series type associated with the first imaging plane;

selecting a first image series of the one or more image series of a first medical exam, the first image series having the first image series type;

selecting, further based on the position of the hand of the user, a first medical image of the first image series; and

generating user interface data for rendering an interactive user interface on an electronic display, the interactive user interface including the first medical image.

10. The computer-implemented method of claim 9 further comprising:

by the one or more processors executing program instructions;

detecting, by the one or more sensors, a motion of the hand of the user along the plane of the hand of the user;

determining an updated position of the hand of the user;

determining, based on the updated position of the hand of the user, a second image series type associated with the first imaging plane;

selecting a second image series of the one or more image series of the first medical exam, the second image series having the second image series type;

selecting, further based on the updated position of the hand of the user, a second medical image of the second image series; and

updating the user interface data such that the interactive user interface includes the second medical image.

11. The computer-implemented method of claim 9 further comprising:

by the one or more processors executing program instructions;

detecting, by the one or more sensors, a motion of the hand of the user along an axis perpendicular to the plane of the hand of the user;

selecting, based on the motion of the hand of the user, a second medical image of the first image series; and

updating the user interface data such that the interactive user interface includes the second medical image.

12. The computer-implemented method of claim 9 further comprising:

by the one or more processors executing program instructions;

detecting, by the one or more sensors, a motion of the hand of the user;

determining;

an updated plane of the hand of the user, and

an updated position of the hand of the user;

determining, based on the updated plane of the hand of the user, a second imaging plane;

determining, based on the updated position of the hand of the user, a second image series type associated with the second imaging plane;

selecting a second image series of the one or more image series of a first medical exam, the second image series having the second image series type;

selecting, further based on the position of the hand of the user, a second medical image of the second image series; and

updating the user interface data such that the interactive user interface includes the second medical image.

13. The computer-implemented method of claim 9, wherein determining the first image series type associated with the first imaging plane comprises:

by the one or more processors executing program instructions;

accessing a hanging protocol associated with the user; and

determining the first series type by mapping the position of the hand of the user to the hanging protocol.

14. The computer-implemented method of claim 9 further comprising:

by the one or more processors executing program instructions;

detecting, by the one or more sensors, a configuration of the hand of the user;

selecting, based on the configuration of the hand of the user and the position of the hand of the user, an operation to perform on the first medical image;

detecting, by the one or more sensors, a motion of the hand of the user while in the configuration; and

implementing the operation on the first medical image in proportion to the motion of the hand of the user.

15. The computer-implemented method of claim 14, wherein the operation is at least one of:

an adjustment of a window, an adjustment of a level, an adjustment of a magnification, or a pan adjustment.