System and method for video image registration and/or providing supplemental data in a heads up display

• US 9,618,752 B2
• Filed: 11/24/2015
• Issued: 04/11/2017
• Est. Priority Date: 02/28/2007
• Status: Active Grant

• Current Assignee: Science Applications International Corporation
• Original Assignee: Science Applications International Corporation
• Inventors: Scales, John Richard, Rodgers, Michael Harris
• Primary Examiner(s): TRAN, NAM T
• Application Number:
  US14/950,643
• Publication Number:
  US 20160077343A1
• Time in Patent Office: 504 Days
• CPC Class Codes: F41G 3/165 using a TV-monitor
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0159 with mechanical means other than scaning means for positioning the whole image
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0176 characterised by mechanical features
• CPC Class Codes: G02B 27/0189 Sight systems
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Video sources and inertial sensors are attached to a weapon and to goggles. A computer receives video images from the weapon- and goggles-mounted sources and inertial data from the sensors. The computer calculates a location for an image from the weapon-mounted source within an image from the goggles-mounted source using the inertial sensor data. The sensor-based location is checked (and possibly adjusted) based on a comparison of the images. A database contains information about real-world objects in a field of view of the goggles-mounted source, and is used to generate icons or other graphics concerning such objects.

Claims

1. A system, comprising:

a first video source configured to generate first video data of images representing portions of an external environment within a field of view of the first video source;

a second video source configured to generate second video data of images representing portions of the external environment within a field of view of the second video source;

a video display; and

a controller in communication with the first and second video sources and the display and configured to perform operations that include

receiving the first video data, the second video data, first motion data corresponding to the first video source, and second motion data corresponding to the second video source,

identifying, based on the received first motion data and the received second motion data, a region of a first image generable from the first video data for comparison with a region of a second image generable from the second video data,

comparing data corresponding to the identified region of the first image and data corresponding to the region of the second image,

selecting, based on the comparing, a part of the first image and a part of the second image that represent a same portion of the external environment,

displaying at least a portion of the first image and the selected part of the second image such that the selected part of the second image replaces the selected part of the first image and is in registration with regions of the first image surrounding the selected part of the first image.

2. The system of claim 1, wherein the controller is configured to perform operations that include

updating, based on the comparing, calibration data used to identify the region of the first image based on the received first motion data and the received second motion data,

receiving, after the updating, additional first video data, additional second video data, additional first motion data, and additional second motion data, and

identifying, based on the received additional first motion data and the received additional second motion data, a region of a third image generable from the additional first video data for comparison with a region of a fourth image generable from the additional second video data.

3. The system of claim 1, wherein the comparing comprises

determining a first point within the first image and a second point within the second image that represent a same part of the external environment, and

determining a twist rotation about the first and second points.

4. The system of claim 3, wherein

determining the first point and the second point comprises selecting a predetermined point in the second image as the second point and identifying the first point as a point in the first image for which a rotationally invariant similarity metric relative to the second point is maximized, and

determining the twist rotation comprises determining a phase shift between a signal corresponding to a portion of the first image surrounding the first point and a signal corresponding to a portion of the second image surrounding the second point.

5. The system of claim 1, wherein

the comparing comprises storing, for each of multiple pairs of a location in the first image and a location in the second image, a data set that includes a value for a peak to sidelobe ratio (PSR) and a center point location,

the comparing further comprises identifying one of the PSR values that exceeds others of the PSR values by a predetermined percentage, and

the selecting comprises selecting the part of the first image based on the center point location from the data set of the identified PSR value.

6. The system of claim 5, wherein the comparing comprises, for each of the pairs of a location in the first image and a location in the second image, generating the data set based on a comparison of gray-scale values in a portion of the first image surrounding the location in the first image and gray-scale values in a portion of the second image surrounding the location in the second image.

7. A method comprising:

receiving first video data of images representing portions of an external environment within a field of view of a first video source;

receiving second video data of images representing portions of the external environment within a field of view of a second video source;

receiving first motion data corresponding to the first video source and second motion data corresponding to the second video source;

identifying, based on the received first motion data and the received second motion data, a region of a first image generable from the first video data for comparison with a region of a second image generable from the second video data;

comparing data corresponding to the identified region of the first image and data corresponding to the region of the second image;

selecting, based on the comparing, a part of the first image and a part of the second image that represent a same portion of the external environment; and

displaying at least a portion of the first image and the selected part of the second image such that the selected part of the second image replaces the selected part of the first image and is in registration with regions of the first image surrounding the selected part of the first image.

8. The method of claim 7, further comprising

updating, based on the comparing, calibration data used to identify the region of the first image based on the received first motion data and the received second motion data;

receiving, after the updating, additional first video data, additional second video data, additional first motion data, and additional second motion data; and

identifying, based on the received additional first motion data and the received additional second motion data, a region of a third image generable from the additional first video data for comparison with a region of a fourth image generable from the additional second video data.

9. The method of claim 7, wherein the comparing comprises

determining a first point within the first image and a second point within the second image that represent a same part of the external environment, and

determining a twist rotation about the first and second points.

10. The method of claim 9, wherein

determining the first point and the second point comprises selecting a predetermined point in the second image as the second point and identifying the first point as a point in the first image for which a rotationally invariant similarity metric relative to the second point is maximized, and

determining the twist rotation comprises determining a phase shift between a signal corresponding to a portion of the first image surrounding the first point and a signal corresponding to a portion of the second image surrounding the second point.

11. The method of claim 7, wherein

the comparing comprises storing, for each of multiple pairs of a location in the first image and a location in the second image, a data set that includes a value for a peak to sidelobe ratio (PSR) and a center point location,

the comparing further comprises identifying one of the PSR values that exceeds others of the PSR values by a predetermined percentage, and

the selecting comprises selecting the part of the first image based on the center point location from the data set of the identified PSR value.

12. The method of claim 11, wherein the comparing comprises, for each of the pairs of a location in the first image and a location in the second image, generating the data set based on a comparison of gray-scale values in a portion of the first image surrounding the location in the first image and gray-scale values in a portion of the second image surrounding the location in the second image.

13. A non-transitory machine-readable medium having machine executable instructions for performing a method comprising:

receiving first video data of images representing portions of an external environment within a field of view of a first video source;

receiving second video data of images representing portions of the external environment within a field of view of a second video source;

receiving first motion data corresponding to the first video source and second motion data corresponding to the second video source;

identifying, based on the received first motion data and the received second motion data, a region of a first image generable from the first video data for comparison with a region of a second image generable from the second video data;

comparing data corresponding to the identified region of the first image and data corresponding to the region of the second image;

selecting, based on the comparing, a part of the first image and a part of the second image that represent a same portion of the external environment; and

displaying at least a portion of the first image and the selected part of the second image such that the selected part of the second image replaces the selected part of the first image and is in registration with regions of the first image surrounding the selected part of the first image.

14. The non-transitory machine-readable medium of claim 13, further comprising machine executable instructions for performing:

updating, based on the comparing, calibration data used to identify the region of the first image based on the received first motion data and the received second motion data;

receiving, after the updating, additional first video data, additional second video data, additional first motion data, and additional second motion data; and

identifying, based on the received additional first motion data and the received additional second motion data, a region of a third image generable from the additional first video data for comparison with a region of a fourth image generable from the additional second video data.

15. The non-transitory machine-readable medium of claim 13, wherein the comparing comprises

determining a first point within the first image and a second point within the second image that represent a same part of the external environment, and

determining a twist rotation about the first and second points.

16. The non-transitory machine-readable medium of claim 15, wherein

determining the first point and the second point comprises selecting a predetermined point in the second image as the second point and identifying the first point as a point in the first image for which a rotationally invariant similarity metric relative to the second point is maximized, and

determining the twist rotation comprises determining a phase shift between a signal corresponding to a portion of the first image surrounding the first point and a signal corresponding to a portion of the second image surrounding the second point.

17. The non-transitory machine-readable medium of claim 13, wherein

the comparing comprises storing, for each of multiple pairs of a location in the first image and a location in the second image, a data set that includes a value for a peak to sidelobe ratio (PSR) and a center point location,

the comparing further comprises identifying one of the PSR values that exceeds others of the PSR values by a predetermined percentage, and

the selecting comprises selecting the part of the first image based on the center point location from the data set of the identified PSR value.

18. The non-transitory machine-readable medium of claim 17, wherein the comparing comprises, for each of the pairs of a location in the first image and a location in the second image, generating the data set based on a comparison of gray-scale values in a portion of the first image surrounding the location in the first image and gray-scale values in a portion of the second image surrounding the location in the second image.