VIDEO-BASED INTERACTIVE VIEWING ALONG A PATH IN MEDICAL IMAGING

• US 20170061687A1
• Filed: 09/01/2015
• Published: 03/02/2017
• Est. Priority Date: 09/01/2015
• Status: Active Grant

• Current Assignee: Siemens Healthcare GMBH (Siemens AG)
• Original Assignee: Siemens Healthcare GMBH (Siemens AG)
• Inventors: YU, DAPHNE, Hong, Wei, Paulus, John Jr., Petkov, Kaloian
• Granted Patent: US 10,204,449 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2210/41 Medical
• CPC Class Codes: G11B 27/309 Table of contents

Abstract

For interactive viewing in medical imaging, pre-rendering is used to provide greater quality of the rendered images with a given computational power. For interactivity in viewing direction, two or more views are rendered for each location and provided as video to the viewer. The viewer selects a viewing direction interactively, and an image at the selected view direction is formed from the previously rendered images of the videos for the two or more views. In an alternative or additional approach, a video graph is created with videos of different segments of the graph. The videos are sequences of images representing change in location, camera parameters (e.g., zooming), and/or rendering settings. Once provided to the user, interactivity is provided by selection of the appropriate node of the video graph. The user may interactively select different locations and/or rendering settings available in the video graph, allowing interaction while benefiting from quality provided by pre-rendered video.

Claims

1. A system for interactive viewing in medical imaging, the system comprising:

a rendering server configured to render a sequence of images from data representing a volume, the sequence being from a view point at different locations along a path through the volume where images from multiple different view directions from the view point are provided for each of the locations, the rendering server configured to transmit the sequence as a video; and

a client computer comprising a network interface configured to receive the sequence of images as the video from the rendering server, the client computer configured receive a user selection of a view orientation from a user input device, generate a display image from the images of one of the locations based on the view orientation, and display the display image on a display of the client computer.

2. The system of claim 1 wherein the multiple different view directions comprise two fisheye views of opposite hemispheres around each of the locations.

3. The system of claim 1 wherein the multiple different view directions comprises six cube map views.

4. The system of claim 1 wherein the rendering server is configured to combine the images for each of the locations into a frame for video encoding.

5. The system of claim 1 wherein the rendering server comprises a video codec configured to encode the sequence as the video in a video stream.

6. The system of claim 1 wherein the sequence comprises two or more sets of the images, one set for each of the different view directions at the locations, and wherein the rendering server comprises a video codec configured to encode two or more video streams corresponding to the two or more sets.

7. The system of claim 1 wherein the client computer is configured to generate the image as a video playback from the video.

8. The system of claim 1 wherein the client computer comprises a tablet, laptop, or smart phone.

9. The system of claim 1 wherein the client computer is configured to generate the display image from a combination of parts of two or more images from the different views for a first of the locations based on the user selection of the view orientation.

10. The system of claim 1 wherein the video is part of a video graph having a plurality of segments, a first of the segments comprising the path and a second of the segments comprising a spatial branch from the path or a change in rendering branch from one of the locations along the path.

11. A method for interactive viewing in medical imaging, the method comprising:

receiving, by a client device from a server, a video representing multiple images forming a view surrounding each of a plurality of positions along a path through a medical scan volume;

obtaining user input, on a user input of the client device, of a view direction for at least a first one of the positions;

extracting part of the view surrounding the first position from the multiple images and based on the view direction; and

generating an image on a display of the client device from the extracted part.

12. The method of claim 11 wherein receiving comprises receiving the video at two or more video streams with each of the video streams being for images of the different parts of the view, wherein extracting comprises extracting portions from the images of the two or more streams for the first position, and wherein generating comprises combining the extracted portions of the images.

13. The method of claim 11 wherein receiving the video comprises receiving intra-encoded for a first subset of the positions and predictive-encoded frames for a second subset of the positions, the video being in a video graph, and wherein obtaining the user input comprises obtaining user selection of a node of the video graph.

14. A non-transitory computer readable storage medium having stored therein data representing instructions executable by a programmed processor for interactive viewing in medical imaging, the storage medium comprising instructions for:

generating a video graph having multiple segments, at least a first of the segments branching from a second of the segments;

rendering images from medical scan data representing a volume, the rendering being for different nodes of the first and second segments of the video graph; and

encoding the images of the first segment into a first video and the images of the second segment into a second video.

15. The non-transitory computer readable storage medium of claim 14 wherein generating comprises generating the first and second segments as different spatial paths through the volume.

16. The non-transitory computer readable storage medium of claim 14 wherein generating comprises generating the first segment as a spatial path through the volume and generating the second segment as a change in rendering parameters for a location on the spatial path.

17. The non-transitory computer readable storage medium of claim 14 wherein encoding comprises encoding with a video codec, the images of some of the nodes being encoded as intra-encoded frames and the image of others of the nodes being encoded as predictive-encoded frames.

18. The non-transitory computer readable storage medium of claim 17 further comprising serving the first and second video to a client device upon request based on user selection of nodes, the intra-encoded frames corresponding to user selectable ones of the nodes of the video graph and the nodes of the predictive-encoded frames not being user selectable.

19. The non-transitory computer readable storage medium of claim 17 wherein encoding comprises encoding separate video streams for the first segment with the separate video streams progressing along the first segment in different directions.

20. The non-transitory computer readable storage medium of claim 14 wherein rendering the images comprises rendering multiple of the images for one of the nodes, the multiple images for the one node comprising fisheye images from different directions at a location.