Multi-camera image stitching for a distributed aperture system

• US 20060066730A1
• Filed: 03/18/2005
• Published: 03/30/2006
• Est. Priority Date: 03/18/2004
• Status: Active Grant

• Current Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Original Assignee: Northrop Grumman Corporation
• Inventors: Evans, Daniel B. Jr., Weygandt, James
• Granted Patent: US 7,499,079 B2
• Time in Patent Office: Days
• US Class Current: 348/218.100
• CPC Class Codes: H04N 23/58 Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/2624 for obtaining an image which is composed of whole input images, e.g. splitscreen

Abstract

Distributed Aperture Systems use multiple staring sensors distributed around a vehicle to provide automatic detection of targets, and to provide an imaging capability at all aspects. The sensor image data is “stitched” to make the camera joints transparent to the operator. For example, images from three different cameras may be combined into a single seamless mosaic. The output mosaic is suitable for rendering on a head-steered helmet mounted display or a multifunction console display.

Claims

1. A method for image stitching, comprising:

determining a virtual line-of-sight (LOS) and a corresponding field-of-view (FOV) of an output mosaic;

obtaining a plurality of input images, wherein each image of the plurality of input images contributes to at least one output mosaic pixel; and

mapping contributions from the plurality of input images for each output mosaic pixel.

2. The method of claim 1, wherein the plurality of the input images are substantially simultaneously captured.

3. The method of claim 1, wherein a FOV of each image of the plurality of input images is directly adjacent to or overlaps a FOV of another one of the plurality of input images.

4. The method of claim 3, wherein:

the plurality of input images are captured by a corresponding plurality of imaging sensors, and

each of the plurality of imaging sensors has a LOS that is different from other imaging sensors.

5. The method of claim 1, wherein the step of mapping contributions from the plurality of input images for each output mosaic pixel includes:

for each input image of the plurality of input images, determining whether the input image belongs to a collection of contributing images to the output mosaic pixel; and

determining a value of the output mosaic pixel based on the collection of contributing images.

6. The method of claim 5, wherein the step of determining whether the input image belongs in the collection of contributing images includes:

determining whether the output mosaic pixel is within the input image; and

including the input image in the collection of contributing images when it is determined that the output mosaic pixel is within the input image.

7. The method of claim 6, wherein the step of determining whether the output mosaic pixel is within the input image includes:

determining whether the output mosaic pixel overlaps any portion of the input image.

8. The method of claim 7, wherein the step of determining whether the output mosaic pixel overlaps any portion of the input image includes:

determining an incremental LOS and an incremental FOV of the output mosaic pixel; and

determining whether the incremental LOS and FOV of the output mosaic pixel overlaps any portion of the input image.

9. The method of claim 6, wherein the step of determining whether the output mosaic pixel is within the input image includes:

translating a position of the output mosaic pixel to a corresponding position of the input image; and

determining whether the corresponding position of the input image is within a FOV of the input image.

10. The method of claim 9, wherein the step of translating the position of the output mosaic pixel to the corresponding position of the input image includes:

translating the output mosaic pixel from Euler AZ/EL coordinates to Cartesian FRD coordinates within a output display view coordinate system;

translating the FRD coordinates of the output mosaic pixel to a corresponding FRD coordinates of an input image view coordinate system; and

translating the output mosaic pixel from the FRD coordinates to corresponding Euler coordinates within the input image view coordinate system.

11. The method of claim 5, wherein the step of determining the value of the output mosaic pixel based on the collection of contributing images includes:

inserting an empty value for the output mosaic pixel when the collection of contributing images is empty;

determining an image contribution of each contributing image for the output mosaic pixel when the collection of contributing images is not empty; and

incorporating the image contributions of the contributing images for the output mosaic pixel when the collection of contributing images is not empty.

12. The method of claim 11, wherein the step of determining the image contribution of each contributing image includes:

transforming the output mosaic pixel into a corresponding position within the contributing image;

selecting a pixel within the contributing image that is closest to the corresponding position as a contributing pixel; and

determining the image contribution of the contributing image based on the contributing pixel.

13. The method of claim 12, wherein the step of determining the image contribution of each contributing image further includes:

determining if a number of contributing images in the collection of contributing images is more than a predetermined number; and

removing from the collection of contributing images one or more images so that the number of contributing images is equal to the predetermined number when it is determined that the number of contributing images in the list is more than the predetermined number.

14. The method of claim 13, wherein the step of removing images from the collection of contributing images includes:

determining a radial distance of the corresponding position of the output mosaic pixel from a center of the contributing image for each contributing image; and

removing from the collection of contributing images those images with the greatest minimum radial distances until the number of images of the collection of contributing images is equal to the predetermined number.

15. The method of claim 14, wherein the predetermined number is two.

16. The method of claim 15, wherein the step of determining the image contribution of the contributing image includes:

if a number of the contributing images is one;

providing a value of the contributing pixel as the image contribution of the contributing image; and

if the number of the contributing images is two;

determining a difference of the radial distances of the corresponding positions of the contributing pixels;

determining whether the difference of the radial distances of the corresponding positions of the contributing pixels is within a predetermined seam-width factor;

blending the output contributions of the contributing pixels if the difference is within the predetermined seam-width factor; and

providing the value of the contributing pixel with the closer radial distance and providing zero as the value for the contributing pixel with the further radial distance if the difference is not within the predetermined seam-width factor.

17. The method of claim 16, wherein the step of blending the output contributions of the contributing images includes:

determining a contribution weight for each contributing pixel of the contributing images based on the difference of the radial distances; and

providing as the image contribution the value of the contributing pixel multiplied by the contribution weight.

18. An apparatus for image stitching, comprising:

a control processor for determining a virtual line-of-sight (LOS) and a corresponding field-of-view (FOV) of an output mosaic;

a plurality of imaging sensors for generating a corresponding plurality of input images, wherein each image of the plurality of input images contributes to at least one output mosaic pixel of the output mosaic; and

an image processor for mapping contributions from the plurality of input images for each output mosaic pixel.

19. The apparatus of claim 18, wherein plurality of imaging sensors generate the plurality of the input images substantially simultaneously.

20. The apparatus of claim 19, wherein a FOV of each imaging sensor is directly adjacent to or overlaps a FOV of another imaging sensor.

21. The apparatus of claim 20, wherein each imaging sensor has a LOS that is different from other imaging sensors.

22. The apparatus of claim 18, wherein the image processor is configured for:

determining, for all of the plurality of input images, whether the output mosaic pixel is within a particular input image; and

including the particular input image in a collection of contributing images when it is determined that the output mosaic pixel is within the particular input image.

23. The apparatus of claim 22, wherein the image processor determines whether the output mosaic pixel is within the particular input image as follows:

translating the output mosaic pixel from Euler coordinates to equivalent Cartesian coordinates in an output mosaic coordinate system;

translating the Cartesian coordinates of the output mosaic pixel to a corresponding Cartesian coordinates in an input image coordinate system corresponding the input image; and

translating the output mosaic pixel from the Cartesian coordinates to corresponding Euler coordinates in the input image coordinate system; and

determining whether the corresponding Euler coordinates is within a FOV of the input image.

24. The apparatus claim 22, wherein the image processor maps contributions from the plurality of input images for each output mosaic pixel as follows:

inserting an empty value for the output mosaic pixel when the collection of contributing images is empty; and

determining and inserting an image contribution of each contributing image for the output mosaic pixel when the collection of contributing images is not empty.

25. The apparatus of claim 24, wherein the image processor determines the contribution of each contributing image as follows:

transforming the output mosaic pixel into a corresponding position within the contributing image;

selecting a pixel within the contributing image that is closest to the corresponding position as a contributing pixel;

determining if a number of contributing images in the collection of contributing images is more than a predetermined number;

removing from the collection of contributing images one or more images so that the number of contributing images is equal to the predetermined number when it is determined that the number of contributing images in the list is more than the predetermined number as follows;

determining a radial distance of the corresponding position of the output mosaic pixel from a center of the contributing image for each contributing image; and

removing from the collection of contributing images those images with the greatest minimum radial distances until the number of images of the collection of contributing images is equal to the predetermined number; and

determining the image contribution of the contributing image based on the contributing pixel.

26. The apparatus of claim 25, wherein the predetermined number is 2, and wherein the image processor determines the value of each output mosaic pixel as follows:

if a number of the contributing images for the output mosaic pixel is one;

providing a value of the contributing pixel of the contributing image as the value of the output mosaic pixel; and

if the number of the contributing images for the output pixel is two;

determining a difference of the radial distances of the corresponding positions of the contributing pixels;

determining whether the difference of the radial distances of the corresponding positions of the contributing pixels is within a predetermined seam-width factor;

feathering the output contributions of the contributing pixels if the difference is within the predetermined seam-width factor; and

providing the value of the contributing pixel having the closer radial distance as the value of the output mosaic pixel if the difference is not within the predetermined seam-width factor.

27. The apparatus of claim 26, wherein the image processor blends contributing pixels of two contributing images as follows:

determining a contribution weight for each contributing pixel of the contributing images based on the difference of the radial distances;

multiplying the value of each contributing pixel by the corresponding weight; and

adding the weighted values of the contributing pixels and providing the result as the value of the output mosaic pixel.

28. The apparatus of claim 18, wherein the control processor or the image processor or both are not co-located with the plurality of image sensors.