Systems and methods for providing enhanced vision imaging with decreased latency

• US 20040169617A1
• Filed: 03/01/2003
• Published: 09/02/2004
• Est. Priority Date: 03/01/2003
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Yelton, Dennis J., Bernier, Kenneth L.
• Granted Patent: US 7,148,861 B2
• Time in Patent Office: Days
• US Class Current: 345/1.1
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/205 Image-based rendering

Abstract

The systems and methods of the present invention provide an image processor that displays enhanced vision images based on a plurality images sources. The image processor comprises both serial and parallel processors. The serial processor performs low data volume calculations needed by the parallel processor for image display. The serial processor precalculates transformations needed to convert source data from each source to a primary coordinate system. The parallel processor uses these transforms and correlates source data with the display on a pixel by pixel basis as each frame from the sources is received to thereby provide display data with decreased latency. The image processor reduces parallax effect and is capable of: 1) stitching images of different fields of view to provide a mosaic image; 2) fusing images taken of the same field of view by different types of sources into a fused image; and 3) correcting anomalies in the displayed image.

Claims

1. A system for providing vision imaging comprising:

at least one image source having a plurality of pixels;

at least one display having a plurality of pixels for displaying images provided by said image source, wherein at least one image source pixel of said image source corresponds to a pixel of said display, said display further having a display coordinate system;

a serial processing element in communication with said display, said serial processing element capable of calculating transformations for said display that relate the coordinates of said display to a primary coordinate system; and

a parallel processing element, in electrical communication with said serial processing element and said display, said parallel processing element capable of receiving image data from the image source and using the transformations calculated by said serial processing element for said display to transform the image source pixel corresponding to the display pixel into the display coordinate system for display at the display pixel location.

2. A system according to claim 1, wherein said image source has an image source coordinate system and wherein serial processing element is further capable of calculating transformations for said image source that relate the coordinates of said image source to a primary coordinate system.

3. A system according to claim 2, wherein said parallel processing element uses the transformations calculated by said serial processing element to transform image source pixels from the image source coordinate system to the primary coordinate system and to transform display pixels from the display coordinate system to the primary coordinate system to thereby correlate the image data with the display.

4. A system according to claim 2, wherein for each display pixel said parallel processing element:

extends a first vector from said display pixel to a mapping surface;

determines a point where the first vector intercepts the mapping surface;

extends a second vector from said image source to the intercept;

transforms the second vector to the coordinate system of said image source; and

determines which pixels of said image source correspond to the transformed second vector.

5. A system according to claim 4, wherein if the transformed second vector does not correspond to specific pixels of said image source, said parallel processing element determines pixels of said image source that have an associated vector extending to the mapping surface that approximates the second transformed vector.

6. A system according to claim 4, wherein if the transformed second vector does not correspond to specific pixels of said image source, said parallel processing element further:

determines potential pixels of said image source that should correspond to the transformed second vector;

and for each potential pixel;

extends a third vector from the potential pixels to the mapping surface; and

compares the third vector to the transformed second vector to determine a line of sight vector error; and

selects the pixels having the least line of sight vector error.

7. A system according to claim 1, wherein at least two image sources each have pixels that correspond to a pixel of said display, and for each image source said parallel processor element determines which pixels of said image source correspond to said display pixel, and wherein said parallel processing element combines the pixels of each image source together to form data for displaying at said display pixel.

8. A system according to claim 7, wherein said parallel processing element weights each image source pixel of each image source that corresponds to the display pixel based on a location of the image source pixels within the field of view of its respective image source and sums the image source pixels of each image source together based on their associated weights.

9. A system according to claim 1, wherein at least two image sources each have image source pixels that correspond to a pixel of said display, wherein the image source pixel from each image source has a unique characteristic, said parallel processing element determines which image source pixels of said image sources correspond to said display pixel, and wherein said parallel processing element combines the image source pixels of each image source together into a resultant pixel containing the unique characteristic of each image source pixel.

10. A system for providing vision imaging comprising:

at least one display having a plurality of display pixels;

at least one image source having a plurality of image source pixels, said image source providing images from a field of view for display on said display; and

an image processor in communication with said display and said image source, wherein said image processor defines a mapping surface at a distance from said display and said image source such that pixels of said display and image source have respective line of sights that intercept the mapping surface,

wherein for a given display pixel, said image processor determines which of said image source pixels correspond to a location where the line of sight of said display pixel intercepts the mapping surface.

11. A system according to claim 10, wherein said display has a display coordinate system, wherein said image processor is capable of calculating transformations for said display that relate the coordinates of said display to a primary coordinate system, said image processor further capable of receiving image data from the image source and using the transformations for the display to transform the image source pixel to the display coordinate system for display at the display pixel location.

12. A system according to claim 11, wherein said image source has an image source coordinate system and wherein said image processor is further capable of calculating transformations for said image source that relate the coordinates of said image source to a primary coordinate system.

13. A system according to claim 12, wherein said image processor uses the transformations to transform image source pixels from the image source coordinate system to the primary coordinate system and to transform display pixels from the display coordinate system to the primary coordinate system to thereby correlate the image data with the display.

14. A system according to claim 12, wherein for each display pixel said image processor:

extends a first vector from said display pixel to a mapping surface;

determines a point where the first vector intercepts the mapping surface;

extends a second vector from said image source to the intercept;

transforms the second vector to the coordinate system of said image source; and

determines which pixels of said image source correspond to the transformed second vector.

15. A system according to claim 14, wherein in if the transformed second vector does not correspond to specific pixels of said image source, said image processor determines pixels of said image source that have an associated vector extending to the mapping surface that approximates the second transformed vector.

16. A system according to claim 14, wherein if the transformed second vector does not correspond to specific pixels of said image source, said image processor further:

determines potential pixels of said image source that should correspond to the transformed second vector;

and for each potential pixel;

extends a third vector from the potential pixels to the mapping surface; and

compares the third vector to the transformed second vector to determine a line of sight vector error; and

selects the pixels having the lowest line of sight vector error.

17. A system according to claim 11, wherein at least two image sources each have pixels that correspond to a pixel of said display, and for each image source said image processor determines which pixels of said image source correspond to said display pixel, and wherein said image processor combines the pixels of each image source together to form data for displaying at said display pixel.

18. A system according to claim 17, wherein said image processor weights each image source pixel of each image source that corresponds to the display pixel based on a location of the image source pixels within the field of view of its respective image source and sums the image source pixels of each image source together based on their associated weights.

19. A system according to claim 11, wherein at least two image sources each have image source pixels that correspond to a pixel of said display, wherein the image source pixel from each image source has a unique characteristic, said image processor determines which image source pixels of said image sources correspond to said display pixel, and wherein said image processor combines the image source pixels of each image source together into a resultant pixel containing the unique characteristic of each image source pixel.

20. A method for providing vision imaging comprising:

providing at least one display having a plurality of display pixels and at least one image source having a plurality of image source pixels, the image source providing images from a field of view for display on the display;

defining a mapping surface at a distance from the display and the image source such that pixels of the display and image source have respective line of sights that intercept the mapping surface,

determining for a given display pixel which of the image source pixels correspond to a location where the line of sight of the display pixel intercepts the mapping surface.

21. A method according to claim 20, wherein the display has a display coordinate system, said method further comprising:

calculating transformations for the display that relate the coordinates of the display to a primary coordinate system;

receiving image data from the image source; and

transforming the image source pixel to the display coordinate system using the transformations to thereby display the image source pixel at the display pixel location.

22. A method according to claim 21, wherein said image source has an image source coordinate system and wherein said calculating step calculates transformations for the image source that relate the coordinates of the image source to a primary coordinate system.

23. A method according to claim 22, wherein said transforming step uses the transformations to transform image source pixels from the image source coordinate system to the primary coordinate system and to transform display pixels from the display coordinate system to the primary coordinate system to thereby correlate the image data with the display.

24. A method according to claim 22, wherein for each display pixel said determining step:

extends a first vector from the display pixel to a mapping surface;

determines a point where the vector intercepts the mapping surface;

extends a second vector from the image source to the intercept;

transforms the second vector to the coordinate system of the image source; and

determines which pixels of the image source correspond to the transformed second vector.

25. A method according to claim 24, wherein in said determining step if the transformed second vector does not correspond to specific pixels of the image source, said determining determines pixels of the image source that have an associated vector extending to the mapping surface that approximates the second transformed vector.

26. A method according to claim 24, wherein in said determining step if the transformed second vector does not correspond to specific pixels of the image source, said determining step further:

determines potential pixels of said image source that should correspond to the transformed second vector;

and for each potential pixel;

extends a third vector from the potential pixels to the mapping surface; and

compares the third vector to the transformed second vector to determine a line of sight vector error; and

selects the pixels having the lowest line of sight vector error.

27. A method according to claim 21, wherein at least two image sources each have pixels that correspond to a pixel of the display, and for each image source said determining step determines which pixels of the image source correspond to the display pixel and combines the pixels of each image source together to form data for displaying at the display pixel.

28. A method according to claim 27, wherein said determining step weights each image source pixel of each image source that corresponds to the display pixel based on a location of the image source pixels within the field of view of its respective image source and sums the image source pixels of each image source together based on their associated weights.

29. A method according to claim 21, wherein at least two image sources each have image source pixels that correspond to a pixel of the display, wherein the image source pixel from each image source has a unique characteristic, said determining step determines which image source pixels of the image sources correspond to the display pixel and combines the image source pixels of each image source together into a resultant pixel containing the unique characteristic of each image source pixel.