Virtual image artifact detection

• US 7,292,735 B2
• Filed: 04/16/2004
• Issued: 11/06/2007
• Est. Priority Date: 04/16/2004
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Criminisi, Antonio, Blake, Andrew
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US10/826,963
• Publication Number:
  US 20050232509A1
• Time in Patent Office: 1,299 Days
• Field of Search: 382/275, 382/154, 382260-265, 382/254, 382/286, 382/294, 382/162, 382/167, 382/270, 382/284, 358/3.26, 345/611, 348 46- 50, 348/135, 348/139, 348/143, 348241-252
• US Class Current: 382/275
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 10/10 Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 10/98 Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
• CPC Class Codes: G06V 2201/12 Acquisition of 3D measurements of objects
• CPC Class Codes: H04N 13/10 Processing, recording or transmission of stereoscopic or multi-view image signals
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

Artifacts are detected in a cyclopean virtual image generated from stereo images. A disparity map is generated from the stereo images. Individual projected images are determined based on the disparity map and the corresponding stereo images. A difference map is computed between the individual projected images to indicate the artifacts. A source patch in the virtual image is defined relative to an artifact. A replacement target patch is generated using a split-patch search technique as a composite of a background exemplar patch and a foreground exemplar patch. Each exemplar patch may be identified from an image patch selected from at least one of the stereo images. The source patch of the virtual image is replaced by the replacement target patch to correct the detected artifact.

Claims

1. A method of detecting one or more artifacts in a virtual image synthesized from stereo images, the method comprising:

generating a disparity map from the stereo images;

generating a projected image for each of the stereo images by projecting each stereo image into a target viewpoint based on the disparity map;

computing color-distances between corresponding pixels in the projected images to produce a difference map; and

designating one or more locations in the difference map associated with a computed color-distance exceeding a threshold as the one or more artifacts.

2. The method of claim 1 further comprising:

generating an occlusion map from the stereo images; and

combining the occlusion map into the projected image.

3. The method of claim 1 further comprising:

identifying an artifact pixel in the virtual image that corresponds with one of the artifacts;

defining a source patch relative to the artifact pixel in the virtual image;

identifying a disparity point in the disparity map that corresponds to the artifact pixel;

defining a disparity patch relative to the disparity point in the disparity map; and

generating a filter map from the source patch and the disparity patch.

4. The method of claim 3 wherein the filter map represents a foreground filter map.

5. The method of claim 3 wherein the filter map represents a background filter map.

6. The method of claim 3 further comprising:

filtering the disparity patch before generating the filter map.

7. The method of claim 1 further comprising:

generating a filter map from a source patch of the virtual image and a disparity patch of the disparity map;

determining a candidate exemplar patch from each stereo image based on the filter map.

8. The method of claim 7 wherein the filter map represents a background filter map and further comprising:

filtering the source patch using the background filter map; and

selecting one of the candidate exemplar patches as an uncontaminated background exemplar patch based on comparison to the background-filtered source patch.

9. The method of claim 7 further comprising:

extracting an unoccluded background exemplar patch from an unoccluded foreground candidate exemplar patch;

approximating an occluded background exemplar patch from an occluded foreground candidate exemplar patch; and

determining an uncontaminated foreground exemplar patch from the unoccluded background exemplar patch and the occluded background exemplar patch.

10. The method of claim 9 further comprising:

determining a transparency weight from the unoccluded background exemplar patch and the occluded background exemplar patch.

11. The method of claim 7 further comprising:

generating a target patch as a composite of a background exemplar patch and a foreground exemplar patch.

12. The method of claim 11 further comprising:

replacing the source patch of the virtual image with the target patch.

13. The method of claim 1 further comprising:

generating a target patch as a weighted average of a background exemplar patch and a foreground exemplar patch, based on a transparency weight.

14. A computer readable medium encoded with a computer program for executing on a computer system a computer process for detecting one or more artifacts in a virtual image synthesized from stereo images, the computer process comprising:

generating a projected image for each of the stereo images by projecting each stereo image into a target viewpoint based on a disparity map of the stereo images;

computing differences between corresponding pixels in the projected images to produce a difference map, wherein a computed difference exceeding a threshold indicates an artifact;

generating an occlusion man from the stereo images; and

combining the occlusion man into the projected image.

26. The computer readable medium of claim 14 wherein the computer process further comprises:

generating a target patch as a weighted average of a background exemplar patch and a foreground exemplar patch, based on a transparency weight.

15. A system for detecting one or more artifacts in a virtual image synthesized from stereo images, the system comprising:

an image warp module that generates a projected image for each of the stereo images by projecting each stereo image into a target viewpoint based on a disparity map of the stereo images;

an image distancing module that computes color-distances between corresponding pixels in the projected images to produce a difference map; and

a thresholding module that designates one or more locations in the difference map associated with a computed color-distance exceeding a threshold as the one or more artifacts.

16. A computer readable medium encoded with a computer program for executing on a computer system a computer process for detecting one or more artifacts in a virtual image synthesized from stereo images, the computer process comprising:

generating a projected image for each of the stereo images by projecting each stereo image into a target viewpoint based on a disparity map of the stereo images;

computing differences between corresponding pixels in the projected images to produce a difference map, wherein a computed difference exceeding a threshold indicates an artifact;

identifying an artifact pixel in the virtual image that corresponds with one of the artifacts;

defining a source patch relative to the artifact pixel in the virtual image;

identifying a disparity point in the disparity map that corresponds to the artifact pixel;

defining a disparity patch relative to the disparity point in the disparity map; and

generating a filter map from the source patch and the disparity patch.

17. The computer readable medium of claim 16 wherein the filter map represents a foreground filter map.

18. The computer readable medium of claim 16 wherein the filter map represents a background filter map.

19. The computer readable medium of claim 16 wherein the computer process further comprises:

filtering the disparity patch before generating the filter map.

20. A computer readable medium encoded with a computer program for executing on a computer system a computer process for detecting one or more artifacts in a virtual image synthesized from stereo images, the computer process comprising:

generating a projected image for each of the stereo images by projecting each stereo image into a target viewpoint based on a disparity man of the stereo images;

computing differences between corresponding pixels in the projected images to produce a difference map, wherein a computed difference exceeding a threshold indicates an artifact;

generating a filter map from a source patch of the virtual image and a disparity patch of the disparity map; and

determining a candidate exemplar patch from each stereo image based on the filter map.

21. The computer readable medium of claim 20 wherein the filter map represents a background filter map and the computer process further comprises:

filtering the source patch using the background filter map; and

selecting one of the candidate exemplar patches as an uncontaminated background exemplar patch based on comparison to the background-filtered source patch.

22. The computer readable medium of claim 20 wherein the computer process further comprises:

extracting an unoccluded background exemplar patch from an unoccluded foreground candidate exemplar patch;

approximating an occluded background exemplar patch from an occluded foreground candidate exemplar patch; and

determining an uncontaminated foreground exemplar patch from the unoccluded background exemplar patch and the occluded background exemplar patch.

23. The computer readable medium of claim 22 wherein the computer process further comprises:

determining a transparency weight from the unoccluded background exemplar patch and the occluded background exemplar patch.

24. The computer readable medium of claim 20 wherein the computer process further comprises:

generating a target patch as a composite of a background exemplar patch and a foreground exemplar patch.

25. The computer readable medium of claim 24 wherein the computer process further comprises:

replacing the source patch of the virtual image with the target patch.