System and method of providing real-time dynamic imagery of a medical procedure site using multiple modalities

• US 10,733,700 B2
• Filed: 11/01/2018
• Issued: 08/04/2020
• Est. Priority Date: 08/02/2006
• Status: Active Grant

• Current Assignee: Inneroptic Technology, Inc.
• Original Assignee: Inneroptic Technology, Inc.
• Inventors: Keller, Kurtis P., Razzaque, Sharif A., State, Andrei, Green, Caroline K., Ackerman, Jeremy D.
• Primary Examiner(s): Braniff, Christopher
• Application Number:
  US16/177,894
• Publication Number:
  US 20190180411A1
• Time in Patent Office: 642 Days
• Field of Search: None
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: A61B 1/042 characterised by a proximal camera, e.g. a CCD camera
• CPC Class Codes: A61B 1/313 for introducing through surgical openings, e.g. laparoscopes
• CPC Class Codes: A61B 8/12 in body cavities or body tracts, e.g. by using catheters catheters per se A61M25/00
• CPC Class Codes: A61B 8/463 characterised by displaying multiple images or images and diagnostic data on one display composite display in devices for short-range imaging with acoustic waves G01S7/52074
• CPC Class Codes: A61B 8/5238 for combining image data of patient, e.g. merging several images from different acquisition modes into one image for image registration algorithms G06T7/30
• CPC Class Codes: G06T 3/16 Spatio-temporal transformations, e.g. video cubism
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A system and method of providing composite real-time dynamic imagery of a medical procedure site from multiple modalities which continuously and immediately depicts the current state and condition of the medical procedure site synchronously with respect to each modality and without undue latency is disclosed. The composite real-time dynamic imagery may be provided by spatially registering multiple real-time dynamic video streams from the multiple modalities to each other. Spatially registering the multiple real-time dynamic video streams to each other may provide a continuous and immediate depiction of the medical procedure site with an unobstructed and detailed view of a region of interest at the medical procedure site at multiple depths. A user may thereby view a single, accurate, and current composite real-time dynamic imagery of a region of interest at the medical procedure site as the user performs a medical procedure.

Claims

1. A method for image guidance comprising:

obtaining a first real-time imaging stream from a first medical device;

obtaining a second real-time imaging stream from a second medical device;

determining a relative position and orientation of the first medical device with respect to the second medical device based at least in part on first position and orientation data associated with the first medical device and second position and orientation data associated with the second medical device;

determining a relative position and orientation of the first real-time imaging stream with respect to the second real-time imaging stream based at least in part on the determined relative position and orientation of the first medical device with respect to the second medical device; and

causing one or more displays to display a composite medical image comprising;

at least a portion of the first real-time imaging stream, and

at least a portion of the second real-time imaging stream spatially registered with the at least a portion of the first real-time imaging stream based at least in part on the determined relative position and orientation of the first real-time imaging stream with respect to the second real-time imaging stream.

2. The method of claim 1, wherein the first real-time imaging stream depicts at least a portion of a scene from a first point-of-view of the first medical device, wherein the second real-time imaging stream depicts at least a portion of the scene from a second point-of-view corresponding to the second medical device.

3. The method of claim 1, wherein the composite medical image depicts the at least a portion of the second real-time imaging stream from a point-of-view of the first medical device.

4. The method of claim 1, wherein the first real-time imaging stream is based at least in part on a first modality, wherein the second real-time imaging stream is based at least in part on a second modality that is different from the first modality.

5. The method of claim 1, wherein the composite medical image depicts the at least a portion of the second real-time imaging stream within the at least a portion of the first real-time imaging stream.

6. The method of claim 1, wherein the at least a portion of the first real-time imaging stream is aligned with the at least a portion of the second real-time imaging stream such that the composite medical image forms a composite representation of a scene.

7. The method of claim 1, wherein said determining the relative position and orientation of the first medical device with respect to the second medical device comprises determining a relationship between a pose of the first medical device with respect to a first coordinate system and a pose of the second medical device with respect to a second coordinate system.

8. The method of claim 1, wherein the first real-time imaging stream is based at least in part on a modality that provides two-dimensional imagery.

9. The method of claim 1, wherein the first real-time imaging stream is based at least in part on a modality that provides three-dimensional imagery.

10. The method of claim 1, wherein said causing the one or more displays to display a composite medical image comprises causing the one or more displays to display the composite medical image in a virtual 3D space.

11. The method of claim 1, wherein at least one of the first real-time imaging stream or the second real-time imaging stream corresponds to a real-time dynamic video stream of a scene.

12. The method of claim 1, wherein at least one of the first position and orientation data or the second position and orientation data comprises spatial state data associated with at least one of the first medical device or the second medical device.

13. A system for medical procedure image guidance comprising an image guidance system having one or more processors, the one or more processors configured to:

determine a relative position and orientation of a first medical device with respect to a second medical device based at least in part on first position and orientation data associated with the first medical device and second position and orientation data associated with the second medical device;

determine a relative position and orientation of a first real-time imaging stream received from the first medical device with respect to a second real-time imaging stream received from the second medical device based at least in part on the determined relative position and orientation of the first medical device with respect to the second medical device; and

cause one or more displays to display a composite medical image comprising;

at least a portion of the first real-time imaging stream, and

at least a portion of a second real-time imaging stream spatially registered with the at least a portion of the first real-time imaging stream based at least in part on the determined relative position and orientation of the first real-time imaging stream with respect to the second real-time imaging stream.

14. The system of claim 13, wherein the first real-time imaging stream depicts at least a portion of a scene from a first point-of-view of the first medical device, wherein the second real-time imaging stream depicts at least a portion of the scene from a second point-of-view corresponding to the second medical device.

15. The system of claim 13, wherein the composite medical image depicts a scene from a point-of-view of the first medical device.

16. The system of claim 13, wherein the composite medical image depicts the at least a portion of the second real-time imaging stream within the at least a portion of the first real-time imaging stream.

17. The system of claim 13, wherein the at least a portion of the first real-time imaging stream is aligned with the at least a portion of the second real-time imaging stream such that the composite medical image forms a composite representation of a scene.

18. The system of claim 13, wherein the first real-time imaging stream is based at least in part on a modality that provides three-dimensional imagery.

19. The system of claim 13, wherein at least one of the first real-time imaging stream or the second real-time imaging stream corresponds to a real-time dynamic video stream of a scene.