System and Method for Video Image Registration and/or Providing Supplemental Data in a Heads Up Display

• US 20080204361A1
• Filed: 02/28/2007
• Published: 08/28/2008
• Est. Priority Date: 02/28/2007
• Status: Active Grant

• Current Assignee: Science Applications International Corporation
• Original Assignee: Science Applications International Corporation
• Inventors: Scales, John Richard, Rodgers, Michael Harris
• Granted Patent: US 9,229,230 B2
• Time in Patent Office: Days
• US Class Current: 345/8
• CPC Class Codes: F41G 3/165 using a TV-monitor
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0159 with mechanical means other than scaning means for positioning the whole image
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0176 characterised by mechanical features
• CPC Class Codes: G02B 27/0189 Sight systems
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Video sources and inertial sensors are attached to a weapon and to goggles. A computer receives video images from the weapon- and goggles-mounted sources and inertial data from the sensors. The computer calculates a location for an image from the weapon-mounted source within an image from the goggles-mounted source using the inertial sensor data. The sensor-based location is checked (and possibly adjusted) based on a comparison of the images. A database contains information about real-world objects in a field of view of the goggles-mounted source, and is used to generate icons or other graphics concerning such objects.

Claims

1. A system, comprising:

a first video source configured to generate images representing portions of an external environment;

a second video source, movable independent of the first video source, configured to generate images representing portions of the external environment;

a video display; and

a controller coupled to the first and second video sources and to the display, wherein the controller is configured to

(a) receive video images from the first video source and from the second video source,

(b) receive motion data indicative of motion of the first and second video sources,

(c) identify, based on the received motion data, a part of a first video source image that potentially represents a portion of the external environment represented in a part of a second video source image;

(d) evaluate, based on a comparison of data from the first and second video source images, the identification performed in operation (c); and

(e) display at least a portion of the first video source image and at least a portion of the second video source image such that the second video source image portion overlays a corresponding region of the first video source image portion, wherein the corresponding region represents a portion of the external environment represented in the second video source portion.

2. The system of claim 1, further comprising a first inertial measurement unit (IMU) sensor coupled to and movable with the first video source and a second IMU sensor coupled to and movable with the second video source, and wherein the controller is configured to receive motion data in operation (b) from the first and second IMU sensors.

3. The system of claim 2, wherein

operation (c) includes using the received motion data to determine an amount by which the second video source image portion should be rotated relative to the first video source image portion so that the second video source image portion is in registration with areas of the first video source image portion adjacent to the corresponding region, and

operation (d) includes determining, based on a phase shift of the first video source image relative to the second video source image, a rotation of the second video source image region relative to the first video source image.

4. The system of claim 1, wherein operation (d) includes calculating a rotationally invariant similarity metric for at least one region of the second video source image relative to at least one region of the first video source image.

5. The system of claim 1, wherein

operation (c) includes identifying at least one high-contrast region of one of the first and second video source images, the high contrast region having a reference point contained therein,

operation (c) further includes identifying, based on the received motion data, a corresponding reference point in the other of the first and second video source images,

operation (d) includes identifying elements of the other of the first and second video source images relative to the corresponding reference point, and

operation (d) further includes comparing image data associated with elements of the high-contrast region with image data associated with the identified elements.

6. The system of claim 5, wherein operation (d) includes

obtaining a first set of gray-scale values associated with the high-contrast region elements and obtaining a second set of gray-scale values associated with the identified elements, and

performing one or more Fast Fourier Transforms (FFT) on the first and second sets of gray-scale values.

7. The system of claim 5, wherein the elements of the high-contrast region are positioned in a dimensioned arrangement relative to the reference point and the elements of the other of the first and second video source images are in the dimensioned arrangement relative in the corresponding reference point.

8. The system of claim 7, wherein operation (d) includes

obtaining a first set of gray-scale values associated with the high-contrast region elements and obtaining a second set of gray-scale values associated with the identified elements, and

performing one or more Fast Fourier Transforms (FFT) on the first and second sets of gray-scale values.

9. The system of claim 1, wherein operation (d) includes

(d1) selecting a first location in the first video source image and a second location in the second video source image,

(d2) calculating a peak to sidelobe ratio (PSR) using regions of the first and second video source images surrounding the first and second locations, and

(d3) assessing, using the PSR of operation (d2), whether the first and second locations represent the same portion of the external environment.

10. The system of claim 9, wherein the controller is further configured to

(f) determine, using the PSR of operation (d2), that the first and second locations do not represent the same portion of the external environment,

(g) select additional locations in the first video source image,

(h) calculate PSRs using the region of the second video source image surrounding the second location and regions of the first video source image surrounding the additional locations, and

(i) determine, using a PSR associated with an identified one of the additional locations, that the identified and second locations represent the same portion of the external environment.

11. The system of claim 1, wherein the controller is further configured to adjust, based on the evaluation of operation (d), the manner in which the controller makes the identification of operation (c).

12. The system of claim 2, wherein the second video source is mounted on a weapon, wherein the first video source and the display are contained in a pair of goggles wearable by a user, and wherein the controller is configured to

(f) determine a position of the system in the external environment,

(g) determine an orientation of the goggles in the external environment,

(h) identify objects in the external environment based on the position and orientation determined in operations (f) and (g), and

(i) generate graphic indicia on the display providing information about objects identified in operation (h).

13. The system of claim 12, wherein the graphic indicia are generated on the display in positions associated with the first video source image representations of the external objects about which the indicia provide information.

14. The system of claim 13, wherein the graphic indicia include icons, and wherein the sizes of the icons are scaled based on distances between the system and the external objects about which the icons provide information.

15. A method, comprising:

(a) receiving video images from a first video source and from a second video source representing portions of an external environment;

(b) receiving motion data indicative of motion of the first and second video sources;

(c) identifying, based on the received motion data, a part of a first video source image that potentially represents a portion of the external environment represented in a part of a second video source image;

(d) evaluating, based on a comparison of data from the first and second video source images, the identification performed in step (c); and

(e) displaying at least a portion of the first video source image and at least a portion of the second video source image such that the second video source image portion overlays a corresponding region of the first video source image portion, wherein the corresponding region represents a portion of the external environment represented in the second video source portion.

16. The method of claim 15, wherein step (b) includes receiving data from first and second inertial measurement unit (IMU) sensors respectively coupled to and movable with the first and second video sources.

17. The method of claim 16, wherein

step (c) includes using the received motion data to determine an amount by which the second video source image portion should be rotated relative to the first video source image portion so that the second video source image portion is in registration with areas of the first video source image portion adjacent to the corresponding region, and

step (d) includes determining, based on a phase shift of the first video source image relative to the second video source image, a rotation of the second video source image region relative to the first video source image.

18. The method of claim 15, wherein step (d) includes calculating a rotationally invariant similarity metric for at least one region of the second video source image relative to at least one region of the first video source image.

19. The method of claim 15, wherein

step (c) includes identifying at least one high-contrast region of one of the first and second video source images, the high contrast region having a reference point contained therein,

step (c) further includes identifying, based on the received motion data, a corresponding reference point in the other of the first and second video source images,

step (d) includes identifying elements of the other of the first and second video source images relative to the corresponding reference point, and

step (d) further includes comparing image data associated with elements of the high-contrast region with image data associated with the identified elements.

20. The method of claim 19, wherein step (d) includes

obtaining a first set of gray-scale values associated with the high-contrast region elements and obtaining a second set of gray-scale values associated with the identified elements, and

performing one or more Fast Fourier Transforms (FFT) on the first and second sets of gray-scale values.

21. The method of claim 19, wherein the elements of the high-contrast region are positioned in a dimensioned arrangement relative to the reference point and the elements of the other of the first and second video source images are in the dimensioned arrangement relative in the corresponding reference point.

22. The method of claim 21, wherein step (d) includes

obtaining a first set of gray-scale values associated with the high-contrast region elements and obtaining a second set of gray-scale values associated with the identified elements, and

performing one or more Fast Fourier Transforms (FFT) on the first and second sets of gray-scale values.

23. The method of claim 15, wherein step (d) includes

(d1) selecting a first location in the first video source image and a second location in the second video source image,

(d2) calculating a peak to sidelobe ratio (PSR) using regions of the first and second video source images surrounding the first and second locations, and

(d3) assessing, using the PSR of step (d2), whether the first and second locations represent the same portion of the external environment.

24. The method of claim 23, further including

(f) determining, using the PSR of step (d2), that the first and second locations do not represent the same portion of the external environment;

(g) selecting additional locations in the first video source image;

(h) calculating PSRs using the region of the second video source image surrounding the second location and regions of the first video source image surrounding the additional locations; and

(i) determining, using a PSR associated with an identified one of the additional locations, that the identified and second locations represent the same portion of the external environment.

25. The method of claim 15, further including adjusting, based on the evaluation of step (d), the manner in which the identification of step (c) is performed.

26. The method of claim 16, wherein the second video source is mounted on a weapon, wherein the first video source and a display are contained in a pair of goggles wearable by a user, and further including

(f) determining a position of the system in the external environment;

(g) determining an orientation of the goggles in the external environment;

(h) identifying objects in the external environment based on the position and orientation determined in steps (f) and (g); and

(i) generating graphic indicia on the display providing information about objects identified in step (h).

27. The method of claim 26, wherein the graphic indicia are generated on the display in positions associated with the first video source image representations of the external objects about which the indicia provide information.

28. The method of claim 27, wherein the graphic indicia include icons, and wherein the sizes of the icons are scaled based on distances between the system and the external objects about which the icons provide information.

29. A machine-readable medium having machine-executable instructions for performing a method, comprising:

(a) receiving video images from a first video source and from a second video source representing portions of an external environment;

(b) receiving motion data indicative of motion of the first and second video sources;

(c) identifying, based on the received motion data, a part of a first video source image that potentially represents a portion of the external environment represented in a part of a second video source image;

(d) evaluating, based on a comparison of data from the first and second video source images, the identification performed in step (c); and

(e) displaying at least a portion of the first video source image and at least a portion of the second video source image such that the second video source image portion overlays a corresponding region of the first video source image portion, wherein the corresponding region represents a portion of the external environment represented in the second video source portion.

30. The machine-readable medium of claim 29, wherein step (b) includes receiving data from first and second inertial measurement unit (IMU) sensors respectively coupled to and movable with the first and second video sources.

31. The machine-readable medium of claim 30, wherein

step (c) includes using the received motion data to determine an amount by which the second video source image portion should be rotated relative to the first video source image portion so that the second video source image portion is in registration with areas of the first video source image portion adjacent to the corresponding region, and

step (d) includes determining, based on a phase shift of the first video source image relative to the second video source image, a rotation of the second video source image region relative to the first video source image.

32. The machine-readable medium of claim 29, wherein step (d) includes calculating a rotationally invariant similarity metric for at least one region of the second video source image relative to at least one region of the first video source image.

33. The machine-readable medium of claim 29, wherein

step (c) includes identifying at least one high-contrast region of one of the first and second video source images, the high contrast region having a reference point contained therein,

step (c) further includes identifying, based on the received motion data, a corresponding reference point in the other of the first and second video source images,

step (d) includes identifying elements of the other of the first and second video source images relative to the corresponding reference point, and

step (d) further includes comparing image data associated with elements of the high-contrast region with image data associated with the identified elements.

34. The machine-readable medium of claim 33, wherein step (d) includes

obtaining a first set of gray-scale values associated with the high-contrast region elements and obtaining a second set of gray-scale values associated with the identified elements, and

performing one or more Fast Fourier Transforms (FFT) on the first and second sets of gray-scale values.

35. The machine-readable medium of claim 33, wherein the elements of the high-contrast region are positioned in a dimensioned arrangement relative to the reference point and the elements of the other of the first and second video source images are in the dimensioned arrangement relative in the corresponding reference point.

36. The machine-readable medium of claim 35, wherein step (d) includes

obtaining a first set of gray-scale values associated with the high-contrast region elements and obtaining a second set of gray-scale values associated with the identified elements, and

performing one or more Fast Fourier Transforms (FFT) on the first and second sets of gray-scale values.

37. The machine-readable medium of claim 29, wherein step (d) includes

(d1) selecting a first location in the first video source image and a second location in the second video source image,

(d2) calculating a peak to sidelobe ratio (PSR) using regions of the first and second video source images surrounding the first and second locations, and

(d3) assessing, using the PSR of step (d2), whether the first and second locations represent the same portion of the external environment.

38. The machine-readable medium of claim 37, comprising additional instructions for performing steps that include

(f) determining, using the PSR of step (d2), that the first and second locations do not represent the same portion of the external environment;

(g) selecting additional locations in the first video source image;

(h) calculating PSRs using the region of the second video source image surrounding the second location and regions of the first video source image surrounding the additional locations; and

(i) determining, using a PSR associated with an identified one of the additional locations, that the identified and second locations represent the same portion of the external environment.

39. The machine-readable medium of claim 29, comprising additional instructions for performing steps that include adjusting, based on the evaluation of step (d), the manner in which the identification of step (c) is performed.

40. The machine-readable medium of claim 30, wherein the second video source is mounted on a weapon, wherein the first video source and a display are contained in a pair of goggles wearable by a user, and comprising additional instructions for performing steps that include

(f) determining a position of the system in the external environment;

(g) determining an orientation of the goggles in the external environment;

(h) identifying objects in the external environment based on the position and orientation determined in steps (f) and (g); and

(i) generating graphic indicia on the display providing information about objects identified in step (h).

41. The machine-readable medium of claim 40, wherein the graphic indicia are generated on the display in positions associated with the first video source image representations of the external objects about which the indicia provide information.

42. The machine-readable medium of claim 41, wherein the graphic indicia include icons, and wherein the sizes of the icons are scaled based on distances between the system and the external objects about which the icons provide information.