Dynamic depth recovery from multiple synchronized video streams

• US 6,847,728 B2
• Filed: 12/09/2002
• Issued: 01/25/2005
• Est. Priority Date: 12/09/2002
• Status: Expired due to Term

• Current Assignee: SRI International, Inc.
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Kumar, Rakesh, Tao, Hai, Sawhney, Harpreet Singh
• Primary Examiner(s): Bali, Vikkram
• Application Number:
  US10/315,291
• Publication Number:
  US 20040109585A1
• Time in Patent Office: 778 Days
• Field of Search: 382/106, 382/154, 382/171, 382/172, 382/173, 382/107, 356/3, 356/12, 345/419, 345/427, 345/426, 348/42, 348/47
• US Class Current: 382/106
• CPC Class Codes: G06T 7/579 from motion

Abstract

A method of generating a dynamic depth map for a sequence of images from multiple cameras models a scene as a collection of 3D piecewise planar surface patches induced by color based image segmentation. This representation is continuously estimated using an incremental formulation in which the 3D geometric, motion, and global visibility constraints are enforced over space and time. The proposed algorithm optimizes a cost function that incorporates the spatial color consistency constraint and a smooth scene motion model.

Claims

1. A method of generating a dynamic depth map of a scene from a sequence of sets of segmented images of the scene, including a current set of segmented images and a prior set of segmented images, each set of segmented images representing a plurality of different views of the scene at a respective instant, wherein each image in each set of segmented images includes a respective coordinate set, the method comprising the steps of:

a) determining a temporal correspondence between segments in at least one image in the current set of segmented images and corresponding segments in at least one image of the prior set of segmented images to obtain a temporal depth predicted value for each segment in the current set of segmented images;

b) determining a spatial correspondence among segments in the current set of segmented images to obtain a spatial depth predicted value for each segment in at least one image of the current set of images;

c) for each segment in the at least one image of the current set of segmented images, selecting one of the temporal depth predicted value and the spatial depth predicted value as an initial depth estimate of the segment;

d) generating a measure of difference between the at least one image in the current set of segmented images and each other image in the current set of segmented images;

e) repeating steps c) and d) selecting different initial depth values until a minimum measure of difference is generated in step d).

2. A method according to claim 1, where in the step of generating the measure of difference between the at least one image in the current set of segmented images and each other image in the current set of segmented images includes the steps of:

warping the at least one image in the current set of segmented images into the coordinate system of each of the other images in the current set of segmented images to obtain a set of respective warped images;

measuring a difference between the warped images and their respective other image in the current set of segmented images and summing the result to obtain a first term;

measuring an average distance between points in corresponding segments of the warped images and the respective other images in the current set of segmented images and summing the result, to obtain a second term; and

summing the first term and the second term to obtain the measure of difference.

3. A method according to claim 1, wherein:

the step of selecting one of the temporal depth predicted value and the spatial depth predicted value selects the value for one segment in the at least one image while holding the depth values of other, adjacent segments unchanged; and

the step of generating the measure of difference between the at least one image in the current set of images and each other image in the current set of images generates a difference only of the one segment and segments in the at least one image adjacent to the one segment.

4. A method according to claim 1, wherein the step of determining a temporal correspondence between segments in the at least one image in the current set of segmented image and corresponding segments in the at least one image of the prior set of segmented images includes the steps of:

classifying each segment in the at least one image in the prior set of segmented images as a textured segment or an untextured segment;

for each untextured segment, associating at least one corresponding segment in the at least one image of the current set of segmented images;

for each segment in the at least one image of the current set of segmented images that is not associated with one of the untextured segments, identifying the segment as a textured segment and identifying each textured segment in the at least one image of the current set of images with a corresponding segment from the at least one image in the prior set of images using an optical flow technique; and

for each segment in the at least one image of the current set of segmented images, assigning a depth value of the corresponding segment of the at least one image in the prior set of images as the temporal depth predicted value.

5. A method according to claim 2, wherein:

the step of selecting one of the temporal depth predicted value and the spatial depth predicted value selects the value for one segment in the at least one image while holding the depth values of other, adjacent segments unchanged; and

the step of generating the measure of difference between the at least one image in the current set of images and each other image in the current set of images generates a difference only of the one segment and segments in the at least one image adjacent to the one segment.

6. A method according to claim 5, wherein the step of determining a temporal correspondence between segments in the at least one image in the current set of segmented image and corresponding segments in the at least one image of the prior set of segmented images includes the steps of:

classifying each segment in the at least one image in the prior set of segmented images as a textured segment or an untextured segment;

for each untextured segment, associating at least one corresponding segment in the at least one image of the current set of segmented images;

for each segment in the at least one image of the current set of segmented images that is not associated with one of the untextured segments, identifying the segment as a textured segment and identifying each textured segment in the at least one image of the current set of images with a corresponding segment from the at least one image in the prior set of images using an optical flow technique; and

for each segment in the at least one image of the current set of segmented images, assigning a depth value of the corresponding segment of the at least one image in the prior set of images as the temporal depth predicted value.