System and method for depth extraction of images with forward and backward depth prediction

• US 9,659,382 B2
• Filed: 05/28/2008
• Issued: 05/23/2017
• Est. Priority Date: 05/28/2008
• Status: Active Grant

• Current Assignee: InterDigital Madison Patent Holdings (InterDigital, Inc.)
• Original Assignee: Thomson Licensing (Vantiva SA)
• Inventors: Zhang, Dong-Qing, Izzat, Izzat, Yoon, Youngshik
• Primary Examiner(s): McDowell, Jr., Maurice L
• Application Number:
  US12/994,894
• Publication Number:
  US 20110069064A1
• Time in Patent Office: 3,282 Days
• Field of Search: 345422, 345475
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/593 from stereo images

Abstract

A system and method for spatiotemporal depth extraction of images with forward and backward depth prediction are provided. The system and method of the present disclosure provide for acquiring a plurality of frames, generating a first depth map of a current frame in the plurality of frames based on a depth map of a previous frame in the plurality of frames, generating a second depth map of the current frame in the plurality of frames based on a depth map of a subsequent frame in the plurality of frames, and processing the first depth map and the second depth map to produce a third depth map for the current frame.

Claims

1. In a computer, a method of matching at least two images, the method comprising:

acquiring a plurality of frames, each frame in the plurality of frames includes a first image and a second image;

generating a first depth map of a current frame in the plurality of frames based on a depth map of a previous frame in the plurality of frames;

generating a second depth map of the current frame in the plurality of frames based on a depth map of a subsequent frame in the plurality of frames; and

processing the first depth map and the second depth map to produce a third depth map for the current frame,

wherein generating first and second depth maps further include;

estimating a disparity of at least one point in the first image with at least one corresponding point in the second image, the step of estimating including performing a low-cost optimization function and computing a belief propagation function based on the results of the low-cost optimization function;

generating a disparity map based on the estimated disparity; and

converting the estimated disparity map into a depth map; and

wherein the first depth map contains an error based on the depth map of the previous frame and the second depth map contains a correction for reducing the error.

2. The method of claim 1, wherein the estimating the disparity includes computing a pixel matching cost function.

3. The method of claim 1, wherein the estimating the disparity includes computing a smoothness cost function.

4. The method of claim 1, wherein the estimating the disparity includes computing a temporal cost function.

5. The method of claim 1, wherein the first image is a left eye view and second image is a right eye view of a stereoscopic pair.

6. The method of claim 1, wherein the second depth map contains the correction for reducing the error due to a smoothing effect accumulated during processing of the plurality of frames.

7. A system for matching at least two images, the system comprising:

means for acquiring a plurality of frames, each frame in the plurality of frames includes a first image and a second image;

means for generating a first depth map of a current frame in the plurality of frames based on a depth map of a previous frame in the plurality of frames;

means for generating a second depth map of the current frame in the plurality of frames based on a depth map of a subsequent frame in the plurality of frames; and

means for processing the first depth map and the second depth map to produce a third depth map for the current frame,

wherein the means for generating the first depth map and means for generating the second depth map further include;

means for estimating a disparity of at least one point in the first image with at least one corresponding point in the second image, the means for estimating further including means for performing a low-cost optimization function and means for computing a belief propagation function based on the results of the low-cost optimization function;

means for generating a disparity map based on the estimated disparity; and

means for converting the estimated disparity map into a depth map; and

wherein the first depth map contains an error based on the depth map of the previous frame and the second depth map contains a correction for reducing the error.

8. The system of claim 7, wherein the means for estimating the disparity includes means for computing a pixel matching cost function.

9. The system of claim 7, wherein the means for estimating the disparity includes means for computing a smoothness cost function.

10. The system of claim 7, wherein the means for estimating the disparity includes computing a temporal cost function.

11. The system of claim 7, wherein the first image is a left eye view and second image is a right eye view of a stereoscopic pair.

12. The system of claim 7, wherein the first image is a left eye view and second image is a right eye view of a stereoscopic pair.

13. The system of claim 7, wherein the second depth map contains the correction for reducing the error due to a smoothing effect accumulated during processing of the plurality of frames.

14. An apparatus matching at least two images, the apparatus comprising:

a memory configured for storing data and instructions;

a processor configured to acquire a plurality of frames, each frame in the plurality of frames includes a first image and a second image;

generate a first depth map of a current frame in the plurality of frames based on a depth map of a previous frame in the plurality of frames;

generate a second depth map of the current frame in the plurality of frames based on a depth map of a subsequent frame in the plurality of frames; and

process the first depth map and the second depth map to produce a third depth map for the current frame,

wherein generating first and second depth maps further include;

estimating a disparity of at least one point in the first image with at least one corresponding point in the second image, the step of estimating including performing a low-cost optimization function and computing a belief propagation function based on the results of the low-cost optimization function;

generating a disparity map based on the estimated disparity; and

converting the estimated disparity map into a depth map; and

wherein the first depth map contains an error based on the depth map of the previous frame and the second depth map contains a correction for reducing the error.

15. The apparatus of claim 14, wherein the processor if further configured to estimate the disparity by computing a pixel matching cost function.

16. The apparatus of claim 14, wherein the processor if further configured to estimate the disparity by computing a smoothness cost function.

17. The apparatus of claim 14, wherein the processor if further configured to estimate the disparity by computing a temporal cost function.

18. The apparatus of claim 14, wherein the second depth map contains the correction for reducing the error due to a smoothing effect accumulated during processing of the plurality of frames.