Systems and user interfaces for dynamic interaction with two- and three-dimensional medical image data using spatial positioning of mobile devices

• US 10,120,451 B1
• Filed: 01/09/2015
• Issued: 11/06/2018
• Est. Priority Date: 01/09/2014
• Status: Active Grant

• Current Assignee: Merative US LP
• Original Assignee: DR Systems Incorporated (International Business Machines Corporation)
• Inventors: Fram, Evan K.
• Primary Examiner(s): Lao, Lun-Yi
• Assistant Examiner(s): Suteerawongsa, Jarurat
• Application Number:
  US14/593,330
• Time in Patent Office: 1,397 Days
• CPC Class Codes: A61B 6/463 characterised by displaying multiple images or images and diagnostic data on one display
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06T 7/0012 Biomedical image inspection

Abstract

Embodiments of the present disclosure relate to systems and techniques for accessing data stores of medical images and displaying the medical images in substantially real-time to provide information in an interactive user interface. Systems are disclosed that may advantageously provide highly efficient, intuitive, and rapid dynamic interaction with two- and three-dimensional medical image data using spatial positioning of mobile devices. The systems may include interactive user interfaces that are dynamically updated to provide a virtual viewbox or window to two- and/or three-dimensional image data. A user may use the systems described herein to more quickly, thoroughly, and efficiently interact with image data including two-dimensional images, three-dimensional image data, and/or series of image data, as compared to previous systems. The systems described may also enable a user to efficiently view and interact with such image data using a small display, such as the display of a smartphone or tablet.

Claims

1. A medical image computing system comprising:

a mobile computing device including at least an electronic display;

one or more sensors configured to provide spatial position and/or motion information associated with the mobile computing device;

a storage device configured to store electronic software instructions; and

one or more computer processors in communication with the electronic display, the one or more sensors, and the storage device, the one or more computer processors configured to execute the stored software instructions to cause the computing system to;

provide a user interface on the electronic display, the user interface configured to display medical image data on the electronic display;

determine, based on the spatial position and/or motion information provided by the one or more sensors, a first spatial position of the mobile computing device in a physical space;

determine a fixed virtual spatial position of the medical image data in a virtual space corresponding to the physical space, wherein the fixed virtual spatial position of the medical image data corresponds to a fixed physical spatial position in the physical space;

receive, via the mobile computing device, a user input initially selecting the fixed virtual spatial position, wherein the fixed virtual spatial position is fixed relative to the physical space;

determine, based on the first spatial position of the mobile computing device relative to the fixed virtual spatial position of the medical image data, at least a first portion of the medical image data viewable via the electronic display;

update the user interface to include the first portion of the medical image data such that the first portion of the medical image data is displayed on the electronic display;

receive an indication from the one or more sensors of a movement of the mobile computing device from the first spatial position and relative to the fixed virtual spatial position of the medical image data initially selected via the user input;

calculate, based on the spatial position and/or motion information provided by the one or more sensors, a position difference between the first spatial position and a second spatial position of the mobile computing device;

determine, based on the position difference, the second spatial position of the mobile computing device in the physical space and relative to the fixed virtual spatial position of the medical image data initially selected via the user input;

automatically determine, based on the calculated second position difference and/or the second spatial position of the mobile computing device relative to the fixed virtual spatial position of the medical image data initially selected via the user input, at least a second portion of the medical image data viewable via the electronic display; and

update the user interface to include the second portion of the medical image data such that the second portion of the medical image data is displayed on the electronic display, wherein the second portion of the medical image data is virtually positioned relative to the fixed virtual spatial position of the medical image data initially selected via the user input.

2. The computing system of claim 1, wherein the medical image data comprises an array of medical images.

3. The computing system of claim 2, wherein the array of medical images comprises a two-dimensional array of medical images, and wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to:

in response to the position difference indicating a lateral movement, determine the second portion of the medical image data to be at a corresponding lateral position on a plane of the two-dimensional array of medical images relative to the first portion.

4. The computing system of claim 3, wherein the first and second portions of the medical image data comprise first and second medical images of the two-dimensional array.

5. The computing system of claim 4, wherein updating the user interface to include the second portion of the medical image data includes snapping to the second medical image.

6. The computing system of claim 4, wherein updating the user interface to include the second portion of the medical image data includes smoothly panning from the first medical image to the second medical image.

7. The computing system of claim 3, wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to:

in response to determining that the second spatial position is to a right of the first spatial position, determine the second portion of the medical image data to be to a right of the first portion of the medical image data;

in response to determining that the second spatial position is to a left of the first spatial position, determine the second portion of the medical image data to be to a left of the first portion of the medical image data;

in response to determining that the second spatial position is above the first spatial position, determine the second portion of the medical image data to be above the first portion of the medical image data; and

in response to determining that the second spatial position is below the first spatial position, determine the second portion of the medical image data to be below the first portion of the medical image data.

8. The computing system of claim 3, wherein translational and rotational motion of the mobile computing device is not used in determining the second portion of the medical image data.

9. The computing system of claim 2, wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to:

in response to the position difference indicating an angular movement, determine the second portion of the medical image data to be at a corresponding position on the array of medical images relative to the first portion.

10. The computing system of claim 2, wherein the array of medical images comprises a curved array of medical images.

11. The computing system of claim 1, wherein the medical image data comprises a two-dimensional array of a plurality of series of medical images, wherein the first portion of the medical image data comprises a first of the plurality of series of medical images, and wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to:

in response to the position difference indicating at least one of an angular movement, a lateral movement, a translational movement, or a rotational movement, determine the second portion of the medical image data to be a second of the plurality of series of medical images, the second of the plurality of series of medical images positioned at a corresponding position on the two-dimensional array relative to the first portion.

12. The computing system of claim 11, wherein each of the plurality of series of medical images is associated with an exam obtained at a different time or via a different modality.

13. The computing system of claim 11, wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to:

in response to an input from a user of the mobile computing device, updating the user interface to display a particular image of the second of the plurality of series of medical images.

14. The computing system of claim 13, wherein, during the input from the user, other movement of the mobile computing device that would otherwise be used to select a different of the plurality of series of medical images is ignored.

15. The computing system of claim 13, wherein the input comprises at least one of a touch input or a combination of a touch input and a movement of the mobile computing device.

16. The computing system of claim 1, wherein the medical image data comprises a plurality of two-dimensional medical images ordered one after another front to back in a virtual space, wherein the first portion comprises a first medical image of the plurality of two-dimensional medical images, and wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to:

in response to the position difference indicating a translational movement, determine the second portion of the medical image data to be at a corresponding second medical image of the plurality of two-dimensional medical images relative to the first medical image.

17. The computing system of claim 1, wherein:

the medical image data comprises a plurality of two-dimensional series of medical images,

each of the plurality of two-dimensional series shows a same anatomical part from differing orientations,

the first and second portions of the medical image data comprise medical images selected from the plurality of two-dimensional series of medical images,

the first portion of the medical image data comprises a first medical image from a first two-dimensional series, and

the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to;

in response to the second spatial position comprising a rotation of the mobile computing device with reference to the first spatial position and relative to the fixed virtual spatial position of the medical image data initially selected via the user input, determine the second portion of the medical image data to be a second medical image from a second two-dimensional series showing the anatomical part from a corresponding rotated position.

18. The computing system of claim 1, wherein:

the medical image data comprises a plurality of two-dimensional series of medical images,

at least two of the plurality of two-dimensional series shows a same anatomical part from a same orientation,

the first and second portions of the medical image data comprise medical images selected from the plurality of two-dimensional series of medical images, and

the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to;

in response to the second portion of the medical image data comprising a medical image from one of the at least two of the plurality of two-dimensional series, allowing a user of the mobile computing device to select one of the at least two of the plurality of two-dimensional series and determining the second portion of the medical image data to be a corresponding medical image from the selected on of the at least two of the plurality of two-dimensional series.

19. The computing system of claim 18, wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to:

notify the user of the existence of the at least two of the plurality of two-dimensional series before allowing the user to select the one of the at least two of the plurality of two-dimensional series.

20. The computing system of claim 1, wherein the medical image data comprises three-dimensional medical image data.

21. The computing system of claim 20, wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to:

utilize the second spatial position of the mobile computing device as a rendering parameter for rendering a view of the three-dimensional medical image data corresponding to the second spatial position.

22. The computing system of claim 20, wherein the first spatial position, the second spatial position, the position difference, the fixed virtual spatial position, and the fixed physical spatial position include at least one of location or orientation information.

23. The computing system of claim 20, wherein the first and second portions of the medical image data comprise rendered slices of the three-dimensional medical image data.

24. The computing system of claim 23, wherein the rendered slices of the three-dimensional medical image data are rendered in response to the received indication of the movement of the mobile computing device.

25. The computing system of claim 20, wherein the first and second portions of the medical image data comprise rendered volumes of the three-dimensional medical image data.

26. The computing system of claim 20, wherein the first and second portions of the medical image data comprise two-dimensional images of sections of the three-dimensional medical image data.

27. The computing system of claim 20, wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to:

in response to the position difference indicating a translational movement, determine the second portion of the medical image data to be at a corresponding translated position in the three-dimensional medical image data relative to the first portion.

28. The computing system of claim 20, wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to:

in response to the position difference indicating a rotational movement, determine the second portion of the medical image data to be at a corresponding rotated position in the three-dimensional medical image data relative to the first portion and relative to the fixed virtual spatial position of the medical image data initially selected via the user input.

29. The computing system of claim 20, wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to:

in response to the position difference indicating an arbitrary movement, determine the second portion of the medical image data to be at a corresponding arbitrary position in the three-dimensional medical image data relative to the first portion.

30. The computing system of claim 1, wherein the medical image data comprises at least one of a single medical image, an array of medical images, a series of medical images, two-dimensional medical image data, three-dimensional medical image data, a three-dimensional medical image, a three-dimensional volume, rendered portions of three-dimensional medical image data, or combinations thereof.

31. The computing system of claim 1, wherein the mobile computing device comprises at least one of a smartphone, a tablet, a smartwatch, or a wearable mobile computing device.

32. The computing system of claim 1, wherein the one or more sensors are housed in the mobile computing device and comprise at least one of an accelerometer, a gyroscope, or a Global Positioning System transceiver.

33. The computing system of claim 1, wherein the one or more sensors are located external to the mobile computing device and comprise at least one of a video camera or an infrared camera.

34. The computing system of claim 1, wherein the fixed virtual spatial position of the medical image data in the virtual space is determined based on user preferences.