Method and apparatus for virtual interactive medical imaging by multiple remotely-located users

• US 6,608,628 B1
• Filed: 11/05/1999
• Issued: 08/19/2003
• Est. Priority Date: 11/06/1998
• Status: Expired due to Fees

• Current Assignee: Administrator of The National Aeronautics and Space Administration
• Original Assignee: United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration
• Inventors: Senger, Steven O., Twombly, Ian Alexander, Ross, Muriel D.
• Primary Examiner(s): Bella, Matthew C.
• Assistant Examiner(s): Sajous, Wesner
• Application Number:
  US09/436,716
• Time in Patent Office: 1,383 Days
• Field of Search: 345/733, 345/734, 345/735, 345/736, 345/737, 345/740, 345/741, 345/744, 345/748, 345/750, 345/751, 345/753, 345/761, 345/619, 345/700, 345/719, 345/723, 345/732, 345/756, 709/223, 709/219, 709/315, 709/107, 709/201, 709/204, 709/205, 709/100, 709/200, 709/231, 709/232, 709/246, 703/1, 703/104.1
• US Class Current: 345/619
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2210/08 Bandwidth reduction
• CPC Class Codes: G06T 2210/36 Level of detail
• CPC Class Codes: G06T 2210/41 Medical
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 40/67 for remote operation
• CPC Class Codes: G16H 50/50 for simulation or modelling of medical disorders
• CPC Class Codes: G16H 80/00 ICT specially adapted for facilitating communication between medical practitioners or patients, e.g. for collaborative diagnosis, therapy or health monitoring

Abstract

A virtual interactive imaging system allows the displaying of high-resolution, three-dimensional images of medical data to a user and allows the user to manipulate the images, including rotation of images in any of various axes. The system includes a mesh component that generates a mesh to represent a surface of an anatomical object, based on a set of data of the object, such as from a CT or MRI scan or the like. The mesh is generated so as to avoid tears, or holes, in the mesh, providing very high-quality representations of topographical features of the object, particularly at high-resolution. The system further includes a virtual surgical cutting tool that enables the user to simulate the removal of a piece or layer of a displayed object, such as a piece of skin or bone, view the interior of the object, manipulate the removed piece, and reattach the removed piece if desired. The system further includes a virtual collaborative clinic component, which allows the users of multiple, remotely-located computer systems to collaboratively and simultaneously view and manipulate the high-resolution, three-dimensional images of the object in real-time.

Claims

1. A method of enabling users of a plurality of networked computer systems to collaboratively view and manipulate images of an object, the method comprising:

causing corresponding images of the object to be displayed on each of a plurality of computer systems at a first resolution;

receiving, at one of the computer systems, user input specifying a manipulation of the image;

transmitting information indicative of the user input to each of the other computer systems;

updating the image displayed on each of the other computer systems substantially simultaneously to depict the manipulation, including displaying the image at a second resolution lower than the first resolution while said manipulation is being depicted; and

displaying the image on each of the computer systems at the first resolution when depiction of the manipulation is complete.

2. A method as recited in claim 1, further comprising maintaining a model of the object at the first resolution and a model of the object at the second resolution, wherein said updating comprises switching from using the first model to using the second model to display the image.

3. A method as recited in claim 1, wherein the image displayed by each of the computer systems may be a three-dimensional image.

4. A method as recited in claim 1, wherein said displaying comprises displaying changes of the object over time.

5. A method of enabling a plurality of geographically distributed users to collaboratively view and manipulate images of an object, the method comprising:

maintaining a data structure including data representing the object, the data structure including a plurality of variables shared by each of a plurality of remote processing systems, the data structure further including a plurality of models of the object, each model corresponding to a different image resolution;

multicasting data to each of the remote processing systems based on the data structure to allow the image to be displayed on each of the remote processing systems, including dynamically selecting from the plurality of models; and

coordinating transmission of user inputs and values of shared variables applied at each of the client systems to allow the image displayed on each of the client systems to be updated in real-time in response to user inputs applied at each other client system.

6. A method as recited in claim 5, further comprising selecting said models of the object for said multicasting so as to cause the image to be displayed on each of the client systems at a reduced resolution during a user manipulation of the image on one of the client systems.

7. A method as recited in claim 6, wherein at least one of the client systems is configured to display changes of the object over time.

8. A method as recited in claim 5, further comprising, for each of a plurality of consecutive portions of the data representing the object:

compressing the portion until the portion reaches the specified image size; and

combining the compressed portion into a packet with information indicating a position of said portion within the image.

9. A method of enabling a plurality of geographically distributed users to collaboratively view and manipulate images of an object in real-time, the method comprising, at a server:

maintaining a data structure including data representing the object, the data structure including a plurality of variables shared by each of a plurality of remote client systems, the data structure further including a plurality of models of the object, each model corresponding to a different image resolution;

sequentially preparing a packet of each of a plurality of consecutive portions of the data representing the object for transmission to the remote client systems, based on the data structure, including selecting said models of the object according to display capabilities of the remote client systems;

multicasting each packet that is ready for transmission to each of the remote client systems to allow each of the client systems to display the image of the object based on the packets; and

coordinating transmission of user inputs applied at each of the client systems to allow the image displayed on each of the client systems to be updated in real-time in response to user inputs applied at each of the other client systems.

10. A method as recited in claim 9, wherein said maintaining comprises selecting said models of the object so as to cause the image to be displayed on each of the client systems at a reduced resolution during a user manipulation of the image at one of the client systems.

11. A method as recited in claim 9, wherein said preparing comprises:

compressing the portion until the portion reaches the specified image size;

combining the compressed portion into a packet with information indicating a position of said portion within the image; and

marking the packet as ready for transmission.

12. A method as recited in claim 9, wherein said multicasting further comprises regulating a data transmission rate to control the number of packets lost during transmission.

13. A method as recited in claim 9, wherein said coordinating comprises:

receiving values of said shared variables from each of the remote client systems, the values representing user inputs applied at each said remote client system; and

using the values to update the data structure.

18. An apparatus as recited in claim 11, further comprising means for selecting said models of the object for said multicasting so as to cause the image to be displayed on each of the client systems at a reduced resolution during a user manipulation of the image on one of the client systems.

14. A method of enabling a plurality of geographically distributed users to collaboratively view and manipulate images of an object in real-time, the method comprising, at a server processing system:

maintaining a data structure including data representing the object, the data structure including a plurality of properties of the object and variables shared by each of a plurality of remote client systems, the data structure further including a plurality of models of the object, each model representing an image of the object at a different resolution;

setting a specified image size;

sequentially preparing each of a plurality of consecutive portions of the data representing the object for transmission, based on the data structure, by

compressing the portion using run length encoding until the portion reaches the specified image size,

combining the compressed portion into a packet with information indicating a position of said portion within the image, and

marking the packet as ready for transmission;

sequentially multicasting each said packet that is ready for transmission to each of the remote client systems to allow each of the client systems to display an image of the object based on the packets; and

coordinating transmission of user inputs applied at each of the client systems to allow the image displayed by each of the client systems to be updated in real-time in response to user inputs applied at each other client system, said coordinating including

receiving values of said shared variables from each of the remote client systems, the values representing user inputs applied at each said remote client system, and

using the values to update the data structure.

15. A method as recited in claim 14, wherein said preparing comprises appropriately accessing said models of the object so as to cause the image to be displayed on each of the client systems at a lowered resolution during a user manipulation of the image one of the client systems.

16. A method as recited in claim 14, wherein said transmitting comprises regulating a data transmission rate to control the number of packets lost during transmission.

17. An apparatus for enabling a plurality of geographically distributed users to collaboratively view and manipulate images of an object, the method comprising:

means for maintaining a data structure including data representing the object, the data structure including a plurality of variables shared by each of a plurality of remote processing systems, the data structure further including a plurality of models of the object, each model corresponding to a different image resolution;

means for multicasting data to each of the remote processing systems based on the data structure to allow the image displayed on each of the remote processing systems, including dynamically selecting from the plurality of models; and

means for coordinating transmission of user inputs applied at each of the client systems to allow the image displayed on each of the client systems to be updated in real-time in response to user inputs applied at each other client system.

19. An apparatus as recited in claim 17, further comprising:

means for compressing each of a plurality of consecutive portions of the data representing the object until each said the portion reaches a specified image size; and

means for combining each compressed portion into a packet with information indicating a position of said portion within the image.

20. An apparatus as recited in claim 17, wherein said means for coordinating comprises:

means for receiving values of said shared variables from each of the remote processing systems, the values representing user inputs applied at each said remote client system; and

means for using the values to update the data structure.