Systems and methods for collaborative interactive visualization of 3D data sets over a network ("DextroNet")

• US 20070248261A1
• Filed: 01/03/2007
• Published: 10/25/2007
• Est. Priority Date: 12/31/2005
• Status: Abandoned Application

• Current Assignee: Bracco Imaging S.p.A. (Bracco SPA)
• Original Assignee: Bracco Imaging S.p.A. (Bracco SPA)
• Inventors: Zhou, Lu, Serra, Luis, Goh, Lin
• Application Number:
  US11/649,425
• Publication Number:
  US 20070248261A1
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2210/41 Medical
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 2219/028 Multiple view windows (top-side-front-sagittal-orthogonal)
• CPC Class Codes: G06T 2219/2021 Shape modification
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 40/67 for remote operation
• CPC Class Codes: G16H 80/00 ICT specially adapted for facilitating communication between medical practitioners or patients, e.g. for collaborative diagnosis, therapy or health monitoring
• CPC Class Codes: H04L 12/1822 Conducting the conference, e.g. admission, detection, selection or grouping of participants, correlating users to one or more conference sessions, prioritising transmission

Abstract

Exemplary systems and methods are provided by which multiple persons in remote physical locations can collaboratively interactively visualize a 3D data set substantially simultaneously. In exemplary embodiments of the present invention, there can be, for example, a main workstation and one or more remote workstations connected via a data network. A given main workstation can be, for example, an augmented reality surgical navigation system, or a 3D visualization system, and each workstation can have the same 3D data set loaded. Additionally, a given workstation can combine real-time imagining with previously obtained 3D data, such as, for example, real-time or pre-recorded video, or information such as that provided by a managed 3D ultrasound visualization system. A user at a remote workstation can perform a given diagnostic or therapeutic procedure, such as, for example, surgical navigation or fluoroscopy, or can receive instruction from another user at a main workstation where the commonly stored 3D data set is used to illustrate the lecture. A user at a main workstation can, for example, see the virtual tools used by each remote user as well as their motions, and each remote user can, for example, see the virtual tool of the main user and its respective effects on the data set at the remote workstation. For example, the remote workstation can display the main workstation'"'"'s virtual tool operating on the 3D data set at the remote workstation via a virtual control panel of said local machine in the same manner as if said virtual tool was a probe associated with that remote workstation. In exemplary embodiments of the present invention each user'"'"'s virtual tools can be represented by their IP address, a distinct color, and/or other differentiating designation. In exemplary embodiments of the present invention the data network can be either low or high bandwidth. In low bandwidth embodiments a 3D data set can be pre-loaded onto each user'"'"'s workstation and only the motions of a main user'"'"'s virtual tool and manipulations of the data set sent over the network. In high bandwidth embodiments, for example, real-time images, such as, for example, video, ultrasound or fluoroscopic images, can be also sent over the network as well.

Claims

1. Apparatus for interactively manipulating a three-dimensional image, the image comprising volumetric data generated from imaging data regarding a subject, the apparatus being configured to:

receive, over a communications link, positional data for one or more remote probes of one or more remote machines;

generate, for display on a local display, a combined three-dimensional scene comprising said at least one remote probe and the three-dimensional image;

manipulate the three-dimensional image in response to manipulations of a local probe by a user local to the apparatus; and

send, over the communications link, data regarding said manipulations by said user local to the apparatus sufficient to allow the said at least one remote machine to display a combined three-dimensional scene comprising an image of the local probe performing manipulations on said three-dimensional image.

2. Apparatus according to claim 1, further configured to filter the data sent regarding said manipulations by said user local to the apparatus in response to network conditions.

3. Apparatus according to claim 2, wherein said filtering drops data packets which do not substantially modify the three-dimensional image.

4. Apparatus according to claim 3, wherein said filtering drops packets that relate to one of tool movement and tool status.

5. Apparatus according to claim 1, further configured to display at least one of an IP address, user name and user designator of the one or more remote machines.

6. Apparatus according to claim 1, wherein the data regarding said manipulations by said user local to the apparatus sent to the said one or more remote machines is sufficient to allow the one or more remote machines to display the image of the local probe performing manipulations on said three-dimensional image by interacting with a virtual control panel of said local machine in the same manner as if said local probe was a probe associated with said one or more remote machines.

7. Apparatus according to claim 1, further configured to receive a snapshot of the display of said one or more remote machines in response to a command of said user local to the apparatus.

8. Apparatus according to claim 1, further configured to cause a synchronization of the three-dimensional image between the apparatus and any of the one or more remote machines in response to a command of said user local to the apparatus.

9. Apparatus according to claim 8, wherein causing said synchronization includes sending a compressed copy of the three-dimensional image as stored on the apparatus to the one or more remote machines.

10. Apparatus according to claim 8, wherein causing said synchronization includes sending a list of interactive commands that were executed on the three-dimensional image at the apparatus.

11. Apparatus according to claim 1, said apparatus being further configured to, in response to a request by a remote user of a remote machine, switch the roles of the local machine and the said remote machine.

24. Apparatus according to claim 1, said apparatus being further configured to receive additional data regarding the subject from a remote apparatus local to the subject and local to one of the remote machines, said additional data being co-registered to the three-dimensional image of the subject.

25. Apparatus according to claim 24, wherein said additional data regarding the subject is acquired in substantially real-time.

26. Apparatus according to claim 24, wherein said additional data regarding the subject is one or more of real-time video, pre-recorded video, position data of a probe or instrument local to the subject, fluoroscopic images, ultrasound images and multimodal images.

12. Apparatus for interactively manipulating a three-dimensional image, the image comprising volumetric data generated from imaging data regarding a subject, the apparatus being configured to:

receive, over a communications link, positional data for a remote probe of a remote machine;

receive positional data for a local probe;

generate, for display on a display, a combined three-dimensional scene comprising the remote probe, the local probe and the three-dimensional image; and

manipulate the three-dimensional image in response to manipulations of the remote probe in a manner substantially equivalent to manipulations of the three-dimensional image by a user local to the remote machine via the local probe.

13. Apparatus according to claim 12, wherein the manipulation of the three-dimensional image in response to the manipulations of the remote probe is displayed as the image of the remote probe interacting with a virtual control panel of said local machine in the same manner as if said remote probe was a probe associated with said apparatus.

14. Apparatus according to claim 12, further configured to display at least one of an IP address, username and user designator of the remote machine.

15. Apparatus according to claim 12, further configured to send a snapshot of the display in response to a command of said remote machine.

16. Apparatus according to claim 12, further configured to synchronize the three-dimensional image using image data sent over the communications link from the remote machine in response to a command from said remote machine.

17. Apparatus according to claim 12, further configured to display the three-dimensional image with one of the same viewpoint as that of a user local to the remote machine and an arbitrary viewpoint different from that of a user local to the remote machine.

18. Apparatus according to claim 17, further configured to display a virtual control panel.

19. Apparatus according to claim 17, further configured to display a specialized local control panel, responsive to commands of a user local to the apparatus, when the three-dimensional image is displayed at an arbitrary viewpoint different from that of a user local to the remote machine.

20. Apparatus according to claim 12, further configured to allow a user to perform local operations on the three-dimensional image.

21. Apparatus according to claim 20, wherein said local operations comprise manipulations that do not affect which voxels are considered as being part of an object.

22. Apparatus according to claim 20, wherein said local operations comprise one or more of translations and rotations of objects and settings of magnification and transparency.

23. Apparatus according to claim 12, further configured to create an additional local copy of the three-dimensional image and manipulate said additional copy in response to manipulations received form a user local to the apparatus.

27. Apparatus according to claim 12, further configured to display additional data regarding the subject from a remote apparatus local to the subject and local to one of the remote machines.

28. Apparatus according to claim 27, wherein said additional data is co-registered to the three-dimensional image of the subject.

29. Apparatus according to claim 27, wherein said additional data is substantially real-time.

30. Apparatus according to claim 27, wherein said additional data is wherein said additional data regarding the subject is one or more of real-time video, pre-recorded video, position data of a probe or instrument local to the subject, fluoroscopic images, ultrasound images and multimodal images.

31. A system for interactively manipulating a three-dimensional image, the image comprising volumetric data generated from imaging data regarding a subject, the system comprising:

a main workstation comprising;

a first apparatus being configured to;

receive, over a communications link, positional data for one or more remote probes of one or more remote machines;

generate, for display on a local display, a combined three-dimensional scene comprising said at least one remote probe and the three-dimensional image;

manipulate the three-dimensional image in response to manipulations of a local probe by a user local to the apparatus; and

send, over the communications link, data regarding said manipulations by said user local to the apparatus sufficient to allow the said at least one remote machine to display a combined three-dimensional scene comprising an image of the local probe performing manipulations on said three-dimensional image;

one or more distant workstations, each comprising;

a second apparatus, configured to;

receive, over a communications link, positional data for a remote probe of a remote machine;

receive positional data for a local probe;

generate, for display on a display, a combined three-dimensional scene comprising the remote probe, the local probe and the three-dimensional image; and

manipulate the three-dimensional image in response to manipulations of the remote probe in a manner substantially equivalent to manipulations of the three-dimensional image by a user local to the remote machine via the local probe; and

a data network,

wherein the main workstation and each of said one or more distant workstations are connected via the data network.

32. A method for interactively manipulating a three-dimensional image, the image comprising volumetric data generated from imaging data regarding a subject, the method comprising:

providing a first apparatus, said first apparatus being configured to;

receive, over a communications link, positional data for one or more remote probes of one or more remote machines;

generate, for display on a local display, a combined three-dimensional scene comprising said at least one remote probe and the three-dimensional image;

manipulate the three-dimensional image in response to manipulations of a local probe by a user local to the apparatus; and

send, over the communications link, data regarding said manipulations by said user local to the apparatus sufficient to allow the said at least one remote machine to display a combined three-dimensional scene comprising an image of the local probe performing manipulations on said three-dimensional image;

providing one or more second apparati, each of said second apparati being configured to;

receive, over a communications link, positional data for a remote probe of a remote machine;

receive positional data for a local probe;

generate, for display on a display, a combined three-dimensional scene comprising the remote probe, the local probe and the three-dimensional image; and

manipulate the three-dimensional image in response to manipulations of the remote probe in a manner substantially equivalent to manipulations of the three-dimensional image by a user local to the remote machine via the local probe; and

providing a data network and connecting the first apparatus and the one or more second apparati via the data network,

wherein in operation a user at the first apparatus and a user at the one or more second apparati collaboratively visualize a common 3D data set.

33. The method of claim 32, wherein the user at the first apparatus and a user at a second apparatus switch roles at the initiation of the user at the first apparatus.