System and method for de-noising multiple copies of a signal

• US 7,362,918 B2
• Filed: 06/24/2003
• Issued: 04/22/2008
• Est. Priority Date: 06/24/2003
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Herley, Cormac
• Primary Examiner(s): Wu; Jingge
• Assistant Examiner(s): YUAN, KATHLEEN S
• Application Number:
  US10/606,002
• Publication Number:
  US 20040264806A1
• Time in Patent Office: 1,764 Days
• Field of Search: 382/282, 382/154, 382/284, 345/629, 345/421, 348/586
• US Class Current: 382/284
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 5/40 using histogram techniques
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 5/90 Dynamic range modification of images or parts thereof
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration

Abstract

A “composite signal generator” automatically combines two or more copies of a signal to produce a composite that is better than the individual copies. For example, given two or more input images of a scene captured from approximately the same viewpoint, the composite signal generator automatically produces a composite image having reduced or eliminated areas of occlusion with respect to any occlusions existing in the input images. First, the input images are registered using conventional image registration techniques. Differences between the registered images are then used to identify regions of potential occlusion in one or more of the images. A determination of which image is actually occluded is made by identifying which image has a larger discontinuity along a border of the potentially occluded regions. A composite image is then created by choosing one image and mosaicing parts from the other images where it is occluded and they are not.

Claims

1. A physical computer-readable medium having stored thereon computer executable instructions for automatically constructing an image mosaic from a set of images of a scene, said computer executable instructions comprising:

inputting a set of images of a scene;

registering the set of images;

comparing the set of images to identify areas of difference between the images for all images, said areas of difference representing regions of potential occlusion in each image, wherein an occlusion represents an area in any of the images that is at least partially obstructed from view by one or more objects in one or more of the other images;

determining, for each image, whether regions of potential occlusion in each image represent actual regions of occlusion, or whether the regions of potential occlusion in each image represent regions of non-occlusion, by determining a level of discontinuity along an exterior border of each region of potential occlusion in each image; and

creating a mosaic image by replacing at least one actual region of occlusion in one image from the set of images with corresponding regions of non-occlusion from at least one other image from the set of images of the scene.

2. The computer-readable medium of claim 1 wherein registering the images comprises at least one of translating the images, rotating the images, scaling the images and warping the images, for aligning the images.

3. The computer-readable medium of claim 1 further comprising cropping the images after registering the set of images so that all images cover approximately the same view of the scene.

4. The computer-readable medium of claim 1 further comprising balancing the set of images.

5. The computer-readable medium of claim 4 wherein balancing the images comprises histogram averaging of the images based on corresponding targets for each image.

6. The computer-readable medium of claim 4 wherein balancing the images comprises white balancing the images based on corresponding targets for each image.

7. The computer-readable medium of claim 1 wherein comparing the set of images to identify areas of difference between the images for all images further comprises comparing corresponding blocks of image pixels between each image to determine whether the corresponding blocks match.

8. The computer-readable medium of claim 1 wherein comparing the set of images to identify areas of difference between the images for all images further comprises automatically constructing at least one image mask for identifying non-occluded regions and regions of potential occlusion in the set of images.

9. The computer-readable medium of claim 7 determining whether the corresponding blocks match further comprises determining whether the corresponding blocks match within a predetermined matching threshold.

10. The computer-readable medium of claim 1 wherein replacing at least one actual region of occlusion in one image from the set of images with corresponding regions of non-occlusion further comprises at least one of blending, feathering, and pixel averaging, along the edge of the corresponding regions of non-occlusion used to replace any actual regions of occlusion.

11. The computer-readable medium of claim 1 further comprising a user interface for excluding one or more actual regions of occlusion from being replaced with corresponding regions of non-occlusion.

12. A system for removing occlusions from a composite image formed from a set of images of a scene, comprising:

acquiring at least two images of a scene from approximately the same viewpoint;

aligning each of the images to a base image selected from the set of images;

identifying areas of potential occlusion in each of the aligned images, wherein an occlusion represents an area in any of the images that is at least partially obstructed from view by one or more objects in one or more of the other images;

selecting a seed image from the set of images;

determining whether each area of potential occlusion in the seed image is an actual area of occlusion by examining each area of potential occlusion in the seed image to determine whether a level of discontinuity along an outer edge of each area of discontinuity exceeds a predetermined threshold;

replacing areas of actual occlusion in the seed image with corresponding non-occluded areas from one of the other images in the set to form a composite image from the seed image.

13. The system of claim 12 wherein the system is embedded in a digital camera.

14. The system of claim 12 wherein aligning each of the images to the base images comprises applying a geometric transform to each image relative to the base image for registering each of the images to the base image.

15. The system of claim 14 further comprising individually cropping each image so that all images cover approximately the same view of the scene.

16. The system of claim 12 further comprising balancing the images by applying any of histogram averaging of the images based on corresponding targets for each image, and white balancing of the images based on corresponding targets for each image.

17. The system of claim 12 wherein identifying areas of potential occlusion in each of the aligned images further comprises dividing each image in the set of images into a number of pixel blocks, and comparing corresponding pixel blocks between each image to determine whether the corresponding blocks match within a predetermined threshold.

18. The system of claim 12 wherein identifying areas of potential occlusion in each of the aligned images further comprises a pixel-by-pixel comparison of corresponding pixels between each image to determine whether the corresponding pixels match within a predetermined threshold.

19. The system of claim 12 wherein identifying areas of potential occlusion in each of the aligned images further comprises automatically constructing an image mask for identifying all regions of potential occlusion in the set of images.

20. The system of claim 12 wherein replacing areas of actual occlusion in the seed image with corresponding non-occluded areas from one of the other images in the set further comprises at least one of blending, feathering, and pixel averaging an outer edge of the corresponding non-occluded areas with respect to the seed image.

21. A computer-implemented process for removing occlusions from a mosaic image created from a set of images of a scene, comprising:

inputting a set of two or more images of a scene;

aligning each of the images to a base image selected from the set of images;

comparing each of the aligned images to identify areas of potential occlusion in each of the aligned images, wherein an occlusion represents an area in any of the images that is at least partially obstructed from view by one or more objects in one or more of the other images;

determining a level of discontinuity along an outer edge of each area of potential occlusion for each image, said level of discontinuity indicating an area of actual occlusion where the level of discontinuity exceeds a predetermined discontinuity threshold, and said level of discontinuity indicating an area of non-occlusion where the level of discontinuity is less than the predetermined discontinuity threshold;

creating an image mask for each image, said image masks indicating areas of occlusion and areas of non-occlusion for each image; and

using the image mask for each image for creating a mosaic image by replacing areas of actual occlusion in one of the images with corresponding areas of non-occlusion from one of the other images.

22. The computer-implemented process of claim 21 wherein the computer is integral to a digital camera, and wherein the composite image is formed automatically from images of a scene captured using the digital camera.

23. The computer-implemented process of claim 21 wherein replacing areas of actual occlusion in one of the images with corresponding areas of non-occlusion from one of the other images further comprises blending pixels along an outer edge of the corresponding areas of non-occlusion with pixels surrounding the areas of actual occlusion being replaced.

24. The computer-implemented process of claim 21 wherein replacing areas of actual occlusion in one of the images with corresponding areas of non-occlusion from one of the other images further comprises feathering pixels along an outer edge of the corresponding areas of non-occlusion with pixels surrounding the areas of actual occlusion being replaced.

25. The computer-implemented process of claim 21 wherein replacing areas of actual occlusion in one of the images with corresponding areas of non-occlusion from one of the other images further comprises averaging pixels along an outer edge of the corresponding areas of non-occlusion with pixels surrounding the areas of actual occlusion being replaced.

26. The computer-implemented process of claim 21 further comprising balancing the set of images after registering the set of images.

27. The computer-implemented process of claim 26 wherein balancing the images comprises histogram averaging of the images based on corresponding targets for each image.

28. The computer-implemented process of claim 26 wherein balancing the images comprises white balancing the images based on corresponding targets for each image.