Systems and methods for providing enhanced vision imaging with decreased latency

• US 7,148,861 B2
• Filed: 03/01/2003
• Issued: 12/12/2006
• Est. Priority Date: 03/01/2003
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Yelton, Dennis J., Bernier, Kenneth L.
• Primary Examiner(s): Hjerpe, Richard
• Assistant Examiner(s): NGUYEN, KIMNHUNG T
• Application Number:
  US10/377,412
• Publication Number:
  US 20040169617A1
• Time in Patent Office: 1,382 Days
• Field of Search: 345/156, 345418-419, 345/425, 345 7- 9, 340/946, 340/961
• US Class Current: 345/8
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/205 Image-based rendering

Abstract

The systems and methods of the present invention provide an image processor that displays enhanced vision images based on a plurality images sources. The image processor comprises both serial and parallel processors. The serial processor performs low data volume calculations needed by the parallel processor for image display. The serial processor precalculates transformations needed to convert source data from each source to a primary coordinate system. The parallel processor uses these transforms and correlates source data with the display on a pixel by pixel basis as each frame from the sources is received to thereby provide display data with decreased latency. The image processor reduces parallax effect and is capable of: 1) stitching images of different fields of view to provide a mosaic image; 2) fusing images taken of the same field of view by different types of sources into a fused image; and 3) correcting anomalies in the displayed image.

Claims

1. A system for providing vision imaging comprising:

at least one display having a plurality of pixels for displaying images provided by said image source, wherein at least one image source pixel of said image source corresponds to a pixel of said display, said display further having a display coordinate system;

at least two image sources each having image source pixels that correspond to a pixel of said display, wherein the image source pixel from each image source has a unique characteristic,

a serial processing element in communication with said display, said serial processing element capable of calculating transformations for said display that relate the coordinates of said display to a primary coordinate system; and

a parallel processing element, in electrical communication with said serial processing element and said display, said parallel processing element capable of receiving image data from the image source and using the transformations calculated by said serial processing element for said display to transform the image source pixel corresponding to the display pixel into the display coordinate system for display at the display pixel location,

said parallel processing element determines which image source pixels of said image sources correspond to said display pixel, and wherein said parallel processing element combines the image source pixels of each image source together into a resultant pixel containing the unique characteristic of each image source pixel.

2. A method for providing vision imaging comprising:

providing at least one display having a plurality of display pixels, wherein the display has a display coordinate system;

providing at least two image sources each having a plurality of image source pixels, the image sources each providing images from a field of view for display on the display;

defining a mapping surface at a distance from the display and the image sources such that pixels of the display and image source have respective line of sights that intercept the mapping surface;

determining for a given display pixel which of the image source pixels correspond to a location where the line of sight of the display pixel intercepts the mapping surface, wherein said determining step comprises;

calculating transformations for the display that relate the coordinates of the display to a primary coordinate system;

receiving image data from the image source; and

transforming the image source pixel to the display coordinate system using the transformations to thereby display the image source pixel at the display pixel location,

wherein the at least two image sources each have pixels that correspond to a pixel of the display, and for each image source said determining step determines which pixels of the image source correspond to the display pixel and combines the pixels of each image source together to form data for displaying at the display pixel.

3. A method according to claim 2, wherein said determining step weights each image source pixel of each image source that corresponds to the display pixel based on a location of the image source pixels within the field of view of its respective image source and sums the image source pixels of each image source together based on their associated weights.

4. A method for providing vision imaging comprising:

providing at least one display having a plurality of display pixels, wherein the display has a display coordinate system;

providing at least one image source having a plurality of image source pixels, the image source providing images from a field of view for display on the display;

defining a mapping surface at a distance from the display and the image source such that pixels of the display and image source have respective line of sights that intercept the mapping surface,

determining for a given display pixel which of the image source pixels correspond to a location where the line of sight of the display pixel intercepts the mapping surface, wherein said determining step comprises;

calculating transformations for the display that relate the coordinates of the display to a primary coordinate system;

receiving image data from the image source; and

transforming the image source pixel to the display coordinate system using the transformations to thereby display the image source pixel at the display pixel location,

wherein the image source has an image source coordinate system and wherein said calculating step calculates transformations for the image source that relate the coordinates of the image source to a primary coordinate system,

wherein for each display pixel said determining step;

extends a first vector from the display pixel to a mapping surface;

determines a point where the vector intercepts the mapping surface;

extends a second vector from the image source to the intercept;

transforms the second vector to the coordinate system of the image source; and

determines which pixels of the image source correspond to the transformed second vector.

5. A method according to claim 4, wherein in said determining step if the transformed second vector does not correspond to specific pixels of the image source, said determining determines pixels of the image source that have an associated vector extending to the mapping surface that approximates the second transformed vector.

6. A method according to claim 4, wherein in said determining step if the transformed second vector does not correspond to specific pixels of the image source, said determining step further:

determines potential pixels of said image source that should correspond to the transformed second vector;

and for each potential pixel;

extends a third vector from the potential pixels to the mapping surface; and

compares the third vector to the transformed second vector to determine a line of sight vector error; and

selects the pixels having the lowest line of sight vector error.

7. A method for providing vision imaging comprising:

providing at least one display having a plurality of display pixels, wherein the display has a display coordinate system;

providing at least two image sources each having a plurality of image source pixels, the image sources each providing images from a field of view for display on the display;

defining a mapping surface at a distance from the display and the image source such that pixels of the display and image source have respective line of sights that intercept the mapping surface;

determining for a given display pixel which of the image source pixels correspond to a location where the line of sight of the display pixel intercepts the mapping surface, wherein said determining step comprises;

calculating transformations for the display that relate the coordinates of the display to a primary coordinate system;

receiving image data from the image source; and

transforming the image source pixel to the display coordinate system using the transformations to thereby display the image source pixel at the display pixel location,

wherein the at least two image sources each have image source pixels that correspond to a pixel of the display, wherein the image source pixel from each image source has a unique characteristic, said determining step determines which image source pixels of the image sources correspond to the display pixel and combines the image source pixels of each image source together into a resultant pixel containing the unique characteristic of each image source pixel.

8. A system for providing vision imaging comprising:

at least one image source having a plurality of pixels, wherein said image source has an associated image source coordinate system;

at least one display having a plurality of pixels for displaying images provided by said image source, wherein at least one image source pixel of said image source corresponds to a pixel of said display, said display further having a display coordinate system;

an image processor in communication with said display, said image processor capable of;

calculating transformations for said display that relate the coordinates of said display to a primary coordinate system and transformations for said image source that relate the coordinates of said image source to the primary coordinate system;

receiving image data from the image source; and

transforming the image source pixel corresponding to the display pixel into the display coordinate system for display at the display pixel location using the calculated transformations, said transforming comprising;

extending a first vector from said display pixel to a mapping surface;

determining a point where the first vector intercepts the mapping surface;

extending a second vector from said image source to the intercept;

transforming the second vector to the coordinate system of said image source; and

determining which pixels of said image source correspond to the transformed second vector.

9. A system according to claim 8, wherein if the transformed second vector does not correspond to specific pixels of said image source, said image processor determines pixels of said image source that have an associated vector extending to the mapping surface that approximates the second transformed vector.

10. A system according to claim 8, wherein if the transformed second vector does not correspond to specific pixels of said image source, said image processor further:

determines potential pixels of said image source that should correspond to the transformed second vector;

and for each potential pixel;

extends a third vector from the potential pixels to the mapping surface;

compares the third vector to the transformed second vector to determine a line of

sight vector error; and

selects the pixels having the least line of sight vector error.

11. A system for providing vision imaging comprising:

at least two image sources each having a plurality of pixels;

at least one display having a plurality of pixels for displaying images provided by said image sources, wherein at least one image source pixel of each of said image sources corresponds to a pixel of said display, said display further having a display coordinate system;

an image processor in communication with said display, said image processor capable of;

calculating transformations for said display that relate the coordinates of said display to a primary coordinate system;

receiving image data from the image sources; and

transforming the image source pixels corresponding to the display pixel into the display coordinate system for display at the display pixel location using the calculated transformations,

wherein when said at least two image sources each have pixels that correspond to a pixel of said display, for each image source said image processor is capable of;

determining which pixels of said image sources correspond to said display pixel; and

combining the pixels of each image source together to form data for displaying at said display pixel.

12. A system according to claim 11, wherein said image processor weights each image source pixel of each image source that corresponds to the display pixel based on a location of the image source pixels within the field of view of its respective image source and sums the image source pixels of each image source together based on their associated weights.

13. A system according to claim 11, wherein at least two image sources each have image source pixels that correspond to a pixel of said display, wherein the image source pixel from each image source has a unique characteristic, said image processor determines which image source pixels of said image sources correspond to said display pixel, and wherein said image processor combines the image source pixels of each image source together into a resultant pixel containing the unique characteristic of each image source pixel.

14. A system for providing vision imaging comprising:

at least two image sources each having image source pixels that correspond to a pixel of said display, wherein the image source pixel from each image source has a unique characteristic;

at least one display having a plurality of pixels for displaying images provided by said image sources, wherein at least one image source pixel of each of said image sources corresponds to a pixel of said display, said display further having a display coordinate system;

an image processor in communication with said display, said image processor being capable of;

calculating transformations for said display that relate the coordinates of said display to a primary coordinate system and transformations for said image sources that relate the coordinates of said image sources to the primary coordinate system;

receiving image data from the image sources; and

transforming the image source pixels corresponding to the display pixel into the display coordinate system for display at the display pixel location using the calculated transformations,

said image processor further being capable of;

determining which image source pixels of said image sources correspond to said display pixel; and

combining the image source pixels of each image source together into a resultant pixel containing the unique characteristic of each image source pixel.