Systems and methods for providing enhanced vision imaging

• US 20040169663A1
• Filed: 03/01/2003
• Published: 09/02/2004
• Est. Priority Date: 03/01/2003
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Bernier, Kenneth L.
• Granted Patent: US 7,619,626 B2
• Time in Patent Office: Days
• US Class Current: 345/629
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/205 Image-based rendering

Abstract

The present invention provides systems and methods that provide images of an environment to the viewpoint of a display. The systems and methods define a mapping surface at a distance from the image source and display that approximates the environment within the field of view of the image source. The system methods define a model that relates the different geometries of the image source, display, and mapping surface to each other. Using the model and the mapping surface, the systems and methods tile images from the image source, correlate the images to the display, and display the images. In instants where two image sources have overlapping fields of view on the mapping surface, the systems and methods overlap and stitch the images to form a mosaic image. If two overlapping image sources each have images with unique characteristics, the systems and methods fuse the images into a composite image.

Claims

1. A system for providing images of an environment to the viewpoint of a display, said system comprising:

at least one image source having a field of view and providing an image of the environment; and

a processor in communication with said image source and the display, wherein said processor defines a mapping surface at a distance from said image source and display that approximates the environment within the field of view of said image source, and wherein said processor defines a model that relates a geometry of said image source, a geometry of said display, and a geometry of said mapping surface to each other.

2. A system according to claim 1, wherein the environment to the viewpoint of the display comprises a geometric shape, and wherein as said display approaches the geometric shape, said processor defines a mapping surface that more closely approximates the geometric shape.

3. A system according to claim 1, wherein at least one of said image source and display has optical distortion associated therewith, and wherein the model defined by said processor accounts for the optical distortion.

4. A system according to claim 1, wherein said image source, display and mapping surface all have different coordinate systems, and wherein the model defined by said processor includes transformations for transforming said image source, display, and mapping surface to a different coordinate system.

5. A system according to claim 1, wherein for a selected coordinate of the mapping surface within the field of view of said image source, said processor using the model determines a coordinate of said image source that corresponds to the selected coordinate on the mapping surface, said processor further relates the selected mapping surface coordinate with a corresponding coordinate of the display and displays the image data associated with the determined coordinate of the image source at the corresponding coordinate on the display.

6. A system according to claim 1, wherein said processor comprises:

a central processor; and

a graphics processor,

wherein said central processor defines a mapping surface that is a three-dimensional space comprising a plurality of vertex vectors each representing a three-dimensional coordinate of the mapping space and for a selected vertex of the mapping space within the field of display of said image source, said central processor determines a texture vector of the image provided by said image source that corresponds to the selected vertex of the mapping surface, and provides a collection of vectors to said graphics processor, said collection comprising the selected vertex of the mapping surface, the corresponding vector of the image, and a color vector, and

wherein said graphics processor renders the collection of vectors as a 3D video-textured triangle.

7. A system according to claim 6, wherein at least two of said source, display, and mapping space have a different coordinate system, and wherein said processor calculates transformations for transforming the vectors of said two of said source, display, and mapping space to a primary coordinate system such that the vectors can be correlated.

8. A system according to claim 6, wherein the field of view of said image source defines an image projected on the mapping space, wherein said processor defines a tile that is smaller than the image projected on the mapping surface, wherein all texture vectors of the image within the tile are associated with a respective vertex of the mapping surface and all texture vectors projecting on the mapping surface at locations outside the tile are not associated with a vertex of the mapping space.

9. A system according to claim 6 further comprising at least two image sources having respective fields of view that overlap each other on the mapping space, such that said image sources provide respective images having texture vectors that correspond to the same vertex of the mapping space, wherein said processor, for each image source, provides the vertex of the mapping space and the texture vector of respective image source to the graphics processor for display, such that the images from said sources overlap on said display.

10. A system according to claim 9, wherein said process defines a blending zone representing a plurality of vertices on the mapping surface in a location where the fields of view of said two image sources overlap, wherein for each vertex in the blending zone, said processor alters the intensity value of the vertex color to thereby substantially eliminate a seam between the overlapping tiles.

11. A system according to claim 9, wherein if the images from said sources have different brightness values, said processor reduces a color magnitude of the vertices of the brighter image located in the blend zone by a scale factor based on a ratio of the relative brightness of the adjacent images at the vertices.

12. A system according to claim 6 further comprising at least two image sources having respective fields of view that overlap each other on the mapping space such that said image sources provide respective images having texture vectors that correspond to the same vertex of the mapping space, wherein respective images provided by each of said image sources have a unique characteristic, wherein said processor, for each image source, provides the vertex of the mapping space and the texture vector of respective image source to the graphics processor for display, such that the images from said sources overlap on said display, and wherein said processor combines the images into a resultant image containing the unique characteristic of each image.

13. A method for providing images of an environment to the viewpoint of a display, said method comprising:

providing at least one image source having a field of view and providing an image of the environment;

defining a mapping surface at a distance from the image source and display that approximates the environment within the field of view of the image source; and

defining a model that relates a geometry of the image source, a geometry of the display, and a geometry of the mapping surface to each other.

14. A method according to claim 13, wherein the environment to the viewpoint of the display comprises a geometric shape, and wherein as said display approaches the geometric shape, said defining a mapping surface defines a mapping surface that more closely approximates the geometric shape.

15. A method according to claim 13, wherein at least one of the image source and display has optical distortion associated therewith, and wherein the model defined in said defining step accounts for the optical distortion.

16. A method according to claim 13, wherein said image source, display and mapping surface all have different coordinate systems, and wherein the model defined by defining step includes transformations for transforming the image source, display, and mapping surface to a different coordinate system.

17. A method according to claim 13, wherein for a selected coordinate of the mapping surface within the field of view of the image source, said method further comprising:

using the model to determine a coordinate of the image source that corresponds to the selected coordinate on the mapping surface;

relating the selected mapping surface coordinate with a corresponding coordinate of the display; and

displaying the image data associated with the determined coordinate of the image source at the corresponding coordinate on the display.

18. A system for providing images of an environment to the viewpoint of a display, said system comprising:

at least one image source having a field of view and providing an image of the environment; and

a processor in communication with said image source and the display, wherein said processor defines a mapping surface at a distance from said image source,

wherein the field of view of said image source defines an image projected on the mapping surface, and wherein said processor defines a tile that defines a subset of an area covered by the image projected on the mapping surface by said image source.

19. A system according to claim 18 comprising at least two image sources having respective fields of view that overlap each other on the mapping space, wherein said processor defines respective tiles for each image such that the tiles having overlapping regions.

20. A system according to claim 19, wherein said processor defines blend zones on the mapping surface within the overlap regions and modulates the intensity of the images in the blend zones to hide seams betweens the images.

21. A system according to claim 19, wherein said image sources provide respective images that each have at least one unique characteristic, and wherein said processor combines the images into a resultant image containing the unique characteristic of each image.

22. A method for providing images of an environment to the viewpoint of a display, said method comprising:

providing at least one image source having a field of view and providing an image of the environment;

defining a mapping surface at a distance from said image source, wherein the field of view of the image source defines an image projected on the mapping surface; and

defining a tile that defines a subset of an area covered by the image projected on the mapping surface by said image source.

23. A method according to claim 22, wherein said providing step provides at least two image sources having respective fields of view that overlap each other on the mapping space, wherein said defining tile step defines respective tiles for each image such that the tiles having overlapping regions.

24. A method according to claim 23 further comprising defining blending zones on the mapping surface within the overlap regions and modulating the intensity of the images in the blend zone to hide seams betweens the images.

25. A method according to claim 23, wherein the image sources provide respective images that each have at least one unique characteristic, and wherein said method comprises combining the images into a resultant image containing the unique characteristic of each image.

26. A system for providing images of an environment to the viewpoint of a display, said system comprising:

at least two image sources having respective fields of view and providing images of the environment; and

a processor in communication with said image sources and the display, wherein said processor defines a mapping surface at a distance from said image sources,

wherein the respective fields of view of said image sources define respective images that project on to the mapping surface and have adjacent regions that overlap, and wherein said processor defines blend zones on the mapping surface within the overlap regions and modulates the intensity of the images in the blend zones to hide seams betweens the images.

27. A system according to claim 26, wherein for regions of each image located in the blend zone, said processor tapers the intensity of the image at a point in the image at an edge the blend zone to the outer edge of the image.

28. A method for providing images of an environment to the viewpoint of a display, said method comprising:

providing at least two image sources having respective fields of view and providing images of the environment;

defining a mapping surface at a distance from said image sources, wherein the respective fields of view of said image sources define respective images that project on to the mapping surface and have adjacent regions that overlap;

defining blend zones on the mapping surface within the overlap regions; and

modulating the intensity of the images in the blend zones to hide seams betweens the images.

29. A method according to claim 28, wherein for regions of each image located in the blend zone, said modulating step tapers the intensity of the image at a point in the image at an edge the blend zone to the outer edge of the image.

30. A system for providing images of an environment to the viewpoint of a display, said system comprising:

at least two image sources having respective fields of view that at least partially overlap, wherein said image sources provide respective images that each have at least one unique characteristic; and

a processor in communication with said image sources and the display, wherein said processor defines a mapping surface at a distance from said image sources,

wherein the respective fields of view of said image sources define respective images that project on to the mapping surface and have regions that overlap, and wherein said processor combines the images into a resultant image containing the unique characteristic of each image.

31. A system according to claim 30, wherein said processor displays one of the images with increased intensity relative to the other image to thereby enhance the resultant image.

32. A system according to claim 30, wherein said processor evaluates pixels of each image and weight pixels based on their associated intensity such that pixels having greater intensity are enhanced in the combined image.

33. A method for providing images of an environment to the viewpoint of a display, said method comprising:

providing at least two image sources having respective fields of view that at least partially overlap, wherein said image sources provide respective images that each have at least one unique characteristic;

defining a mapping surface at a distance from said image sources, wherein the respective fields of view of said image sources define respective images that project on to the mapping surface and have regions that overlap;

combining the images into a resultant image containing the unique characteristic of each image.

34. A method according to claim 33, wherein said combining step displays one of the images with increased intensity relative to the other image to thereby enhance the resultant image.

35. A method according to claim 33 further comprising evaluating pixels of each image and weighting pixels based on their associated intensity such that pixels having greater intensity are enhanced in the combined image.