System and method for image registration of multiple video streams

• US 10,181,361 B2
• Filed: 05/04/2017
• Issued: 01/15/2019
• Est. Priority Date: 08/12/2011
• Status: Active Grant

• Current Assignee: Help Lightning, Inc., UAB Research Foundation (University of Alabama System)
• Original Assignee: Help Lightning, Inc., UAB Research Foundation (University of Alabama System)
• Inventors: Dillavou, Marcus W., Shum, Phillip Corey, Guthrie, Barton L., Shenai, Mahesh B., Deaton, Drew Steven, May, Matthew Benton
• Primary Examiner(s): Merouan, Abderrahim
• Application Number:
  US15/587,056
• Publication Number:
  US 20170308673A1
• Time in Patent Office: 621 Days
• CPC Class Codes: G06F 3/1431 using a single graphics controller
• CPC Class Codes: G09G 2340/02 Handling of images in compressed format, e.g. JPEG, MPEG
• CPC Class Codes: G09G 2370/045 using multiple communication channels, e.g. parallel and serial
• CPC Class Codes: G09G 2380/08 Biomedical applications
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 5/397 Arrangements specially adapted for transferring the contents of two or more bit-mapped memories to the screen simultaneously, e.g. for mixing or overlay G09G5/02 takes precedence
• CPC Class Codes: G16H 20/00 ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
• CPC Class Codes: G16H 20/30 relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
• CPC Class Codes: G16H 20/40 relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 30/20 for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 40/63 for local operation
• CPC Class Codes: G16H 40/67 for remote operation
• CPC Class Codes: G16H 50/50 for simulation or modelling of medical disorders
• CPC Class Codes: G16H 80/00 ICT specially adapted for facilitating communication between medical practitioners or patients, e.g. for collaborative diagnosis, therapy or health monitoring
• CPC Class Codes: G16Z 99/00 Subject matter not provided for in other main groups of this subclass
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/167 Synchronising or controlling image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals

Abstract

Provided herein are methods and systems for image registration from multiple sources. A method for image registration includes rendering a common field of interest that reflects a presence of a plurality of elements, wherein at least one of the elements is a remote element located remotely from another of the elements and updating the common field of interest such that the presence of the at least one of the elements is registered relative to another of the elements.

Claims

1. A method comprising:

capturing one or more local images of a physical environment of a local device, the one or more local images comprising a representation of a first element, wherein the one or more local images have a first alignment and orientation when viewed relative to a coordinate system;

receiving, from a remote device, a remote image including a virtual element, wherein, prior to transmission, the remote image had a second alignment and orientation when viewed relative to the coordinate system, and wherein the remote image is received in a transformed state resulting, at least in part, from a skew, a shift, a crop, a translation, or combination thereof of the remote image during transmission such that display of the remote image in the transformed state results in a transformed alignment and orientation different from the second alignment and orientation;

generating a common field of interest based on the one or more local images and the remote image such that at least one of the first element and the virtual element is spatially registered relative to the other of the first element and the virtual element, wherein generating the common field of interest comprises applying one or more image transforms to one or more of the remote image and the one or more local images, and wherein the one or more image transforms are based on a first transformation characteristic of one or more of the local device and the remote device and a second transformation characteristic resulting from transmission of one or more of the remote image and the one or more local images, wherein the applying the one or more image transforms facilitates display of the remote image having the second alignment and orientation when viewed relative to the coordinate system; and

causing the common field of interest to be rendered via the local device.

2. The method of claim 1, wherein the one or more local images are part of a live video capture.

3. The method of claim 1, wherein the local device is a mobile device or tablet.

4. The method of claim 1, further comprising the step of calibrating the common field of interest by aligning a calibration feature of a remote field and a calibration feature of a local field.

5. The method of claim 4, wherein the calibration features of the remote field and the local field are aligned relative to the coordinate system.

6. The method of claim 4, wherein the calibration features of the remote field and the local field are aligned relative to a pre-defined virtual fiducial.

7. The method of claim 1, wherein the at least one of the first element and the virtual element is spatially registered relative to the other of the first element and the virtual element based upon at least a registration feature inserted onto at least one of the remote image and the one or more local images.

8. The method of claim 7, wherein the registration feature is a virtual image inserted onto at least one of the remote image and the one or more local images.

9. The method of claim 7, wherein the registration feature is a virtual image rendered in the common field of interest.

10. A method comprising:

capturing a video of a local physical environment using a mobile device;

rendering a physical image representing a portion of the captured video of the local physical environment via the mobile device, the physical image comprising a representation of a physical element, wherein the physical image has a first alignment and orientation when viewed relative to a coordinate system;

receiving a virtual image from a remote device via a network, the virtual image comprising a representation of a virtual element, wherein, prior to transmission, the virtual image had a second alignment and orientation when viewed relative to the coordinate system, and wherein the virtual image is received in a transformed state resulting, at least in part, from a skew, a shift, a crop, a translation, or combination thereof of the virtual image during transmission such that display of the virtual image in the transformed state results in a transformed alignment and orientation different from the second alignment and orientation;

generating a common field of interest based on the physical image and the virtual image such that at least one of the physical element and the virtual element is spatially registered relative to the other of the physical element and the virtual element, wherein generating the common field of interest comprises applying one or more image transforms to one or more of the physical image and the virtual image, and wherein the one or more image transforms are based on a first transformation characteristic of one or more of the mobile device and the remote device and a second transformation characteristic resulting from transmission of one or more of the physical image and the virtual image, wherein the applying the one or more image transforms facilitates display of the virtual image having the second alignment and orientation when viewed relative to the coordinate system; and

causing the common field of interest to be rendered via the mobile device.

11. The method of claim 10, wherein the mobile device is a handheld device or wearable device.

12. The method of claim 10, further comprising the step of calibrating the common field of interest by aligning a calibration feature of a remote field and a calibration feature of a local field.

13. The method of claim 12, wherein the calibration features of the remote field and the local field are aligned relative to the coordinate system.

14. The method of claim 12, wherein the calibration features of the remote field and the local field are aligned relative to a pre-defined virtual fiducial.

15. The method of claim 10, wherein the at least one of the physical element and the virtual element is spatially registered relative to the other of the physical element and the virtual element based upon at least a registration feature inserted onto at least one of the virtual image and the physical image.

16. The method of claim 15, wherein the registration feature is an additional virtual image inserted onto at least one of the virtual image and the physical image.

17. The method of claim 15, wherein the registration feature is an additional virtual image rendered in the common field of interest.

18. A method comprising:

receiving, via a network, a video of a remote physical environment;

rendering a remote virtual image representing at least a portion of the video of the remote physical environment via a local device, the remote virtual image comprising a representation of a first virtual element, wherein, prior to transmission, the remote virtual image had a first alignment and orientation when viewed relative to a coordinate system, and wherein the remote virtual image is received in a transformed state resulting, at least in part, from a skew, a shift, a crop, a translation, or combination thereof of the remote virtual image during transmission such that display of the remote virtual image in the transformed state results in a transformed alignment and orientation different from the first alignment and orientation;

causing a local virtual image to be generated via the local device, wherein the local virtual image overlays the rendered remote virtual image, the local virtual image comprising a representation of a second virtual element, wherein the local virtual image has a second alignment and orientation when viewed relative to the coordinate system;

causing information representing the local virtual image and representing a spatial relationship between the local virtual image and the remote virtual image to be transmitted to a remote device;

generating a common field of interest based on the remote virtual image and the information representing the local virtual image and representing the spatial relationship between the local virtual image and the remote virtual image such that at least one of the first virtual element and the second virtual element is spatially registered relative to the other of the first virtual element and the second virtual element, wherein generating the common field of interest comprises applying one or more image transforms to one or more of the remote virtual image and the local virtual image, and wherein the one or more image transforms are based on a first transformation characteristic of one or more of the local device and the remote device and a second transformation characteristic resulting from transmission of one or more of the remote virtual image and the local virtual image, wherein the applying the one or more image transforms facilitates display of the remote virtual image having the first alignment and orientation when viewed relative to the coordinate system; and

causing the common field of interest to be rendered via the remote device.

19. The method of claim 18, wherein the local device is a handheld device or wearable device.

20. The method of claim 18, further comprising the step of calibrating the common field of interest by aligning a calibration feature of a remote field and a calibration feature of a local field.