Virtual image generation

• US 7,257,272 B2
• Filed: 04/16/2004
• Issued: 08/14/2007
• Est. Priority Date: 04/16/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Blake, Andrew, Criminisi, Antonio
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US10/826,981
• Publication Number:
  US 20050232510A1
• Time in Patent Office: 1,215 Days
• Field of Search: 382/275, 382/254, 382260-265, 382/286, 382/294, 348241-252, 345/611, 358/3.26
• US Class Current: 382/275
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking

Abstract

Artifacts are detected in a cyclopean virtual image generated from stereo images. A disparity map is generated from the stereo images. Individual projected images are determined based on the disparity map and the corresponding stereo images. A difference map is computed between the individual projected images to indicate the artifacts. A source patch in the virtual image is defined relative to an artifact. A replacement target patch is generated using a split-patch search technique as a composite of a background exemplar patch and a foreground exemplar patch. Each exemplar patch may be identified from an image patch selected from at least one of the stereo images. The source patch of the virtual image is replaced by the replacement target patch to correct the detected artifact.

Claims

1. A method of correcting an artifact in a virtual image synthesized from stereo images, the method comprising:

detecting the artifact in the virtual image based on a disparity map of the stereo images;

designating a source patch relative to the artifact in the virtual image;

generating a target patch as a composite of a background exemplar patch and a foreground exemplar patch, each exemplar patch being identified from an image patch from at least one of the stereo images; and

replacing the source patch of the virtual image with the target patch.

2. The method of claim 1 wherein the virtual image includes a cyclopean virtual image.

3. The method of claim 1 further comprising:

identifying a disparity point in the disparity map that corresponds to the artifact pixel;

defining a disparity patch relative to the disparity point in the disparity map; and

generating a filter map from the source patch and the disparity patch.

4. The method of claim 3 wherein the filter map represents a foreground filter map.

5. The method of claim 3 wherein the filter map represents a background filter map.

6. The method of claim 3 further comprising:

filtering the disparity patch before generating the filter map.

7. The method of claim 1 further comprising:

generating a filter map from a source patch of the virtual image and a disparity patch of the disparity map;

determining a candidate exemplar patch from each stereo image based on the filter map.

8. The method of claim 7 wherein the filter map represents a background filter map and further comprising:

filtering the source patch using the background filter map; and

selecting one of the candidate exemplar patches as the background exemplar patch based on comparison to the background-filtered source patch.

9. The method of claim 7 further comprising:

extracting an unoccluded background exemplar patch from an unoccluded foreground candidate exemplar patch;

approximating an occluded background exemplar patch from an occluded foreground candidate exemplar patch; and

determining the foreground exemplar patch from the unoccluded background exemplar patch and the occluded background exemplar patch.

10. The method of claim 9 further comprising:

determining a transparency weight from the unoccluded background exemplar patch and the occluded background exemplar patch.

11. The method of claim 1 wherein the generating operation comprises:

generating a target patch as a weighted average of a background exemplar patch and a foreground exemplar patch, based on a transparency weight.

12. A computer program product encoding a computer program for executing on a computer system a computer process for correcting an artifact in a virtual image synthesized from stereo images, the computer process comprising:

detecting the artifact in the virtual image based on a disparity map of the stereo images;

designating a source patch relative to the artifact in the virtual image;

generating a target patch as a composite of a background exemplar patch and a foreground exemplar patch, each exemplar patch being identified from an image patch from at least one of the stereo images; and

replacing the source patch of the virtual image with the target patch.

13. The computer program product of claim 12 wherein the virtual image includes a cyclopean virtual image.

14. The computer program product of claim 12 wherein the computer process further comprises:

identifying a disparity point in the disparity map that corresponds to the artifact pixel;

defining a disparity patch relative to the disparity point in the disparity map; and

generating a filter map from the source patch and the disparity patch.

15. The computer program product of claim 14 wherein the filter map represents a foreground filter map.

16. The computer program product of claim 14 wherein the filter map represents a background filter map.

17. The computer program product of claim 14 wherein the computer process further comprises:

filtering the disparity patch before generating the filter map.

18. The computer program product of claim 12 wherein the computer process further comprises:

generating a filter map from a source patch of the virtual image and a disparity patch of the disparity map;

determining a candidate exemplar patch from each stereo image based on the filter map.

19. The computer program product of claim 18 wherein the filter map represents a background filter map and further comprising:

filtering the source patch using the background filter map; and

selecting one of the candidate exemplar patches as the background exemplar patch based on comparison to the background-filtered source patch.

20. The computer program product of claim 18 wherein the computer process further comprises:

extracting an unoccluded background exemplar patch from an unoccluded foreground candidate exemplar patch;

approximating an occluded background exemplar patch from an occluded foreground candidate exemplar patch; and

determining the foreground exemplar patch from the unoccluded background exemplar patch and the occluded background exemplar patch.

21. The computer program product of claim 20 wherein the computer process further comprises:

determining a transparency weight from the unoccluded background exemplar patch and the occluded background exemplar patch.

22. The computer program product of claim 12 wherein the generating operation comprises:

generating a target patch as a weighted average of a background exemplar patch and a foreground exemplar patch, based on a transparency weight.

23. A system for correcting an artifact in a cyclopean virtual image synthesized from stereo images, the system comprising:

an artifact correction module that detects the artifact in the virtual image based on a disparity map of the stereo images, designates a source patch relative to the artifact in the virtual image, generates a target patch as a composite of a background exemplar patch and a foreground exemplar patch, and replaces the source patch of the virtual image with the target patch.