System and method for video image registration and/or providing supplemental data in a heads up display

• US 10,139,629 B2
• Filed: 04/10/2017
• Issued: 11/27/2018
• Est. Priority Date: 02/28/2007
• Status: Active Grant

• Current Assignee: Science Applications International Corporation
• Original Assignee: Science Applications International Corporation
• Inventors: Scales, John Richard, Rodgers, Michael Harris
• Primary Examiner(s): Tran, Nam T
• Application Number:
  US15/483,086
• Publication Number:
  US 20170212353A1
• Time in Patent Office: 596 Days
• CPC Class Codes: F41G 3/165 using a TV-monitor
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0159 with mechanical means other than scaning means for positioning the whole image
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0176 characterised by mechanical features
• CPC Class Codes: G02B 27/0189 Sight systems
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Video sources and inertial sensors are attached to a weapon and to goggles. A computer receives video images from the weapon- and goggles-mounted sources and inertial data from the sensors. The computer calculates a location for an image from the weapon-mounted source within an image from the goggles-mounted source using the inertial sensor data. The sensor-based location is checked (and possibly adjusted) based on a comparison of the images. A database contains information about real-world objects in a field of view of the goggles-mounted source, and is used to generate icons or other graphics concerning such objects.

Claims

1. A method of processing video, comprising:

receiving a first video signal from a first video source;

receiving a second video signal from a second video source;

identifying a region of a first image from the first video source with a region of a second image from the second video source;

comparing data associated with the identified region of the first image and data associated with the identified region of the second image;

selecting, based on the comparing, a part of the first image and a part of the second image that represent a related portion of the external environment; and

displaying at least a portion of the first image and the selected part of the second image such that the selected part of the second image replaces the selected part of the first image and is in registration with a region of the first image surrounding the selected part of the first image.

2. The method of claim 1, wherein the first video source comprises a video camera configured to be mounted to a helmet.

3. The method of claim 1, wherein the second video source comprises a video camera configured to be mounted on a weapon.

4. The method of claim 1, further comprising receiving first motion data corresponding to the first video source and second motion data corresponding to the second video source.

5. The method of claim 1, wherein the comparing further comprises:

determining a first point within the first image and a second point within the second image that represent a related part of the external environment.

6. The method of claim 5, wherein the comparing further comprises:

determining a twist rotation about the first and second points.

7. A system for processing video, the system comprising:

a processor; and

a memory having instructions stored thereon, that, when executed by the processor, cause the processor to perform operations, the operations comprising;

receiving a first video signal from a first video source;

receiving a second video signal from a second video source;

identifying a region of a first image from the first video source with a region of a second image from the second video source;

comparing data associated with the identified region of the first image and data associated with the identified region of the second image;

selecting, based on the comparing, a part of the first image and a part of the second image that represent a related portion of the external environment; and

displaying at least a portion of the first image and the selected part of the second image such that the selected part of the second image replaces the selected part of the first image and is in registration with a region of the first image surrounding the selected part of the first image.

8. The system of claim 7, wherein the first video source comprises a video camera configured to be mounted to a helmet.

9. The system of claim 7, wherein the second video source comprises a video camera configured to be mounted on a weapon.

10. The system of claim 7, further comprising receiving first motion data corresponding to the first video source and second motion data corresponding to the second video source.

11. The system of claim 7, wherein the comparing further comprises:

determining a first point within the first image and a second point within the second image that represent a related part of the external environment.

12. The system of claim 11, wherein the comparing further comprises:

determining a twist rotation about the first and second points.

13. A non-transitory machine-readable medium having machine executable instructions for performing a method comprising:

receiving a first video signal from a first video source;

receiving a second video signal from a second video source;

identifying a region of a first image from the first video source with a region of a second image from the second video source;

comparing data associated with the identified region of the first image and data associated with the identified region of the second image;

selecting, based on the comparing, a part of the first image and a part of the second image that represent a related portion of the external environment; and

displaying at least a portion of the first image and the selected part of the second image such that the selected part of the second image replaces the selected part of the first image and is in registration with a region of the first image surrounding the selected part of the first image.

14. The non-transitory machine-readable medium of claim 13, wherein the first video source comprises a video camera configured to be mounted to a helmet.

15. The non-transitory machine-readable medium of claim 13, wherein the second video source comprises a video camera configured to be mounted on a weapon.

16. The non-transitory machine-readable medium of claim 13, further comprising receiving first motion data corresponding to the first video source and second motion data corresponding to the second video source.

17. The non-transitory machine-readable medium of claim 13, wherein the comparing further comprises:

determining a first point within the first image and a second point within the second image that represent a related part of the external environment.

18. The non-transitory machine-readable medium of claim 17, wherein the comparing further comprises:

determining a twist rotation about the first and second points.