System and method for median fusion of depth maps

• US 20020015048A1
• Filed: 06/27/2001
• Published: 02/07/2002
• Est. Priority Date: 06/28/2000
• Status: Active Grant

• Current Assignee: Telefonaktiebolaget LM Ericsson
• Original Assignee: Telefonaktiebolaget LM Ericsson
• Inventors: Nister, David
• Granted Patent: US 6,868,191 B2
• Time in Patent Office: Days
• US Class Current: 345/625
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 7/55 from multiple images

Abstract

The present invention is directed toward providing a system for constructing a graphical representation of an image, as perceived from a desired view, from a plurality of calibrated views. An optimum, median fused depth map of a new, desired view of an object or a scene is constructed from a plurality of depth maps of the object or scene, utilizing median values for each of the image pixels within the image domain of the new view. Each of the known depth maps are rendered into the new view, and the pixels of the rendered image are processed one at a time until a median value for each pixel in the new view of the image of the object or scene is calculated. The calculated median values are then assembled into a median fused depth map of the new view of the object or scene, with the median fused depth map available to construct a new two or three dimension images and models of the object or scene as perceived from the new view.

Claims

1. A method for computing a fused depth map of a view of an object, comprising:

obtaining camera calibration data for a new view of an object;

rendering each of a plurality of known depth maps of the object into the new view of the object based on the camera calibration of the new view; and

computing the depth map elements of a fused depth map of the new view of the object, wherein each said depth map element corresponds to a pixel in the new view of the object and wherein each said depth map element comprises a median value of a set of rendered depth map elements, corresponding to the pixel, each from one of the plurality of rendered depth maps.

2. The method according to claim 1, wherein the step of computing the depth map elements further comprises:

for each pixel in the new view of the object, initializing the depth map element of the fused depth map of the new view corresponding to the pixel with the minimum value of the plurality of rendered depth map elements corresponding to the pixel;

initializing a support matrix to zero, wherein the support matrix is defined on the domain of the new view of the object;

initializing the elements of a next candidate depth map to infinity;

for each known depth map rendered into the new view of the object and for each pixel in the new view of the object, initializing a minimum depth map value matrix with the smallest depth map value for each said pixel;

decrementing the support matrix at the pixel by one if the pixel is occluded by the rendered depth map;

setting the element of the next candidate depth map at the pixel to the corresponding minimum depth map value matrix element if the pixel is not occluded by the rendered depth map and if the corresponding minimum depth map value matrix element corresponding to the pixel is less than the next candidate depth map element corresponding to the pixel;

incrementing the support matrix at the pixel by one if the pixel is passed by the rendered depth map; and

for each pixel in the new view where the corresponding support matrix element is greater than zero, updating the fused depth map element of the new view with the corresponding next candidate depth map value.

3. A method for determining a fused depth map of a view of an object, comprising:

selecting a desired view of an object for which to determine a fused depth map;

selecting a plurality of known depth maps, wherein each selected known depth map includes at least a partial view of the object;

rendering the plurality of known depth maps into the desired view of the object, wherein for each pixel in the desired view, the corresponding fused depth map element of the desired view is initialized with the lowest depth map value among the plurality of rendered depth map elements corresponding to said pixel;

setting to zero the elements of a support matrix corresponding in dimension to the fused depth map;

setting to infinity the elements of a next candidate depth map corresponding in dimension to the fused depth map;

for each of the plurality of known depth maps, performing the following;

rendering the known depth map into the desired view of the object; and

for each pixel in the desired view, updating the corresponding element of the support matrix based on the value of the rendered depth map at each said pixel;

for each pixel in the desired view, performing the following;

if the value of the support matrix at the pixel is greater than zero, then setting the fused depth map value at the pixel to be equal to the corresponding element of the next candidate depth map.

4. The method according to claim 3, wherein the step of updating the corresponding element of the support matrix comprises:

initializing a minimum depth map value matrix element at said pixel with the smallest of the rendered depth map values at each said pixel;

if the minimum depth map value matrix element at said pixel is less than or equal to the fused depth map value at the corresponding pixel, then decrementing the corresponding element of the support matrix by one; and

if the minimum depth map value matrix element at said pixel is greater than the fused depth map value at the corresponding pixel, then updating the next candidate depth map and the support matrix element corresponding to said pixel.

5. The method according to claim 4, wherein the step of updating the next candidate depth map and the support matrix element comprises:

setting the next candidate depth map at the pixel to the minimum of the next candidate depth map value and the minimum depth map value matrix element corresponding to the pixel; and

if the pixel is passed by the rendered depth map, then incrementing the support matrix element corresponding to the pixel by one.

6. The method according to claim 3, wherein the steps beginning with the setting to zero the elements of a support matrix are repeated n- I times, where n is the number of the plurality of known depth maps.

7. A method for determining a fused depth map of a view of an object, comprising:

selecting a desired view of an object for which to determine a fused depth map;

selecting a plurality of known depth maps, wherein each selected known depth map includes at least a partial view of the object;

rendering the plurality of known depth maps into the desired view of the object, wherein for each pixel in the desired view, the corresponding fused depth map element of the desired view is initialized with the lowest depth map value among the plurality of rendered depth map elements corresponding to said pixel;

setting to zero the elements of a support matrix corresponding in dimension to the fused depth map;

setting to infinity the elements of a next candidate depth map corresponding in dimension to the fused depth map;

for each of the plurality of known depth maps, performing the following;

rendering the known depth map into the desired view of the object; and

for each pixel in the desired view of the object, performing the following;

initializing a minimum depth map value matrix element at said pixel with the smallest of the rendered depth map values at each said pixel;

if the minimum depth map value matrix element at said pixel is less than or equal to the fused depth map value at the corresponding pixel, then decrementing the corresponding element of the support matrix by one; and

if the minimum depth map value matrix element at said pixel is greater than the fused depth map value at the corresponding pixel, then performing the following;

setting the next candidate depth map at the pixel to the minimum of the next candidate depth map value and the minimum depth map value matrix element corresponding to the pixel; and

if the pixel is passed by the rendered depth map, then incrementing the support matrix element corresponding to the pixel by one;

for each pixel in the desired view, performing the following;

if the value of the support matrix at the pixel is greater than zero, then setting the fused depth map value at the pixel to be equal to the corresponding element of the next candidate depth map.

8. The method according to claim 7, wherein the steps of setting to zero the elements of a support matrix through the step of setting the fused depth map value at the pixel to be equal to the corresponding element of the next candidate depth map are repeated n−

1 times, where n is the number of the plurality of known depth maps.

9. A method for constructing a graphical representation of an object, as perceived from a desired view, from a plurality of calibrated views, comprising:

obtaining camera calibration data for a desired view of the object;

computing a plurality of depth maps, each depth map respectively corresponding to each of a plurality of calibrated views of the object;

for each of the plurality of computed depth maps, rendering a selected, computed depth map into the desired view; and

for each pixel in the desired view, computing a median fused depth map value at each pixel in the desired view, wherein as many rendered depth maps occlude said pixel as pass it by.

10. The method according to claim 9, further including projecting the graphical representation of the object as perceived from the desired view, utilizing the computed median fused depth map values.

11. A system for computing a fused depth map of a view of an object, comprising:

a measurement device obtaining camera calibration data for a new view of an object;

a merging device rendering each of a plurality of known depth maps of the object into the new view of the object based on the camera calibration of the new view; and

a processor computing the depth map elements of a fused depth map of the new view of the object, wherein each said depth map element corresponds to a pixel in the new view of the object and wherein each said depth map element comprises a median value of a set of rendered depth map elements, corresponding to the pixel, each from one of the plurality of rendered depth maps.

12. The system according to claim 11, wherein the processor further comprises:

a first initialization device for each pixel in the new view of the object, initializing the depth map element of the fused depth map of the new view corresponding to the pixel with the minimum value of the plurality of rendered depth map elements corresponding to the pixel;

a second initialization device initializing a support matrix to zero, wherein the support matrix is defined on the domain of the new view of the object;

a third initialization device initializing the elements of a next candidate depth map to infinity;

a fourth initialization device for each known depth map rendered into the new view of the object and for each pixel in the new view of the object, initializing a minimum depth map value matrix with the smallest depth map value for each said pixel;

a first math processor decrementing the support matrix at the pixel by one if the pixel is occluded by the rendered depth map;

an assignment device setting the element of the next candidate depth map at the pixel to the corresponding minimum depth map value matrix element if the pixel is not occluded by the rendered depth map and if the corresponding minimum depth map value matrix element corresponding to the pixel is less than the next candidate depth map element corresponding to the pixel;

a second math processor incrementing the support matrix at the pixel by one if the pixel is passed by the rendered depth map; and

an update device for each pixel in the new view where the corresponding support matrix element is greater than zero, updating the fused depth map element of the new view with the corresponding next candidate depth map value.

14. The system according to claim 13, wherein the second processor comprises:

a third initialization device initializing a minimum depth map value matrix element at said pixel with the smallest of the rendered depth map values at each said pixel;

a first math processor decrementing the corresponding element of the support matrix by one if the minimum depth map value matrix element at said pixel is less than or equal to the fused depth map value at the corresponding pixel; and

a third processor updating the next candidate depth map and the support matrix element corresponding to said pixel if the minimum depth map value matrix element at said pixel is greater than the fused depth map value at the corresponding pixel.

15. The system according to claim 14, wherein the processor comprises:

an assignment device setting the next candidate depth map at the pixel to the minimum of the next candidate depth map value and the minimum depth map value matrix element corresponding to the pixel; and

a second math processor incrementing the support matrix element corresponding to the pixel by one if the pixel is passed by the rendered depth map.

16. The system according to claim 13, wherein the first initialization device, the second initialization device, the first processor, and the update device perform their functions n−

1 times, where n is the number of the plurality of known depth maps.

18. The system according to claim 17, wherein the respective functions of the first initialization device through the update device are repeated n−

1 times, where n is the number of the plurality of known depth maps.

20. The system according to claim 19, further including an image projection device projecting the graphical representation of the object as perceived from the desired view, utilizing the computed median fused depth map values.

13. A system for determining a fused depth map of a view of an object, comprising:

a first selection device selecting a desired view of an object for which to determine a fused depth map;

a second selection device selecting a plurality of known depth maps, wherein each selected known depth map includes at least a partial view of the object;

a first merging device rendering the plurality of known depth maps into the desired view of the object, wherein for each pixel in the desired view, the corresponding fused depth map element of the desired view is initialized with the lowest depth map value among the plurality of rendered depth map elements corresponding to said pixel;

a first initialization device setting to zero the elements of a support matrix corresponding in dimension to the fused depth map;

a second initialization device setting to infinity the elements of a next candidate depth map corresponding in dimension to the fused depth map;

a first processor comprising the following for each of the plurality of known depth maps;

a second merging device rendering the known depth map into the desired view of the object; and

a second processor updating, for each pixel in the desired view, the element of the support matrix corresponding to each said pixel, based on the value of the rendered depth map at each said pixel;

an update device setting, for each pixel in the desired view, the fused depth map value at the pixel to be equal to the corresponding element of the next candidate depth map if the value of the support matrix at the pixel is greater than zero.

17. A system for determining a fused depth map of a view of an object, comprising:

a first selection device selecting a desired view of an object for which to determine a fused depth map;

a second selection device selecting a plurality of known depth maps, wherein each selected known depth map includes at least a partial view of the object;

a first merging device rendering the plurality of known depth maps into the desired view of the object, wherein for each pixel in the desired view, the corresponding fused depth map element of the desired view is initialized with the lowest depth map value among the plurality of rendered depth map elements corresponding to said pixel;

a first initialization device setting to zero the elements of a support matrix corresponding in dimension to the fused depth map;

a second initialization device setting to infinity the elements of a next candidate depth map corresponding in dimension to the fused depth map;

a first processor comprising the following for each of the plurality of known depth maps;

a second merging device rendering the known depth map into the desired view of the object; and

a second processor comprising the following for each pixel in the desired view of the object;

a third initialization device initializing a minimum depth map value matrix element at said pixel with the smallest of the rendered depth map values at each said pixel;

a first math processor decrementing the corresponding element of the support matrix by one if the minimum depth map value matrix element at said pixel is less than or equal to the fused depth map value at the corresponding pixel; and

a third processor comprising the following if the minimum depth map value matrix element at said pixel is greater than the fused depth map value at the corresponding pixel;

an assignment device setting the next candidate depth map at the pixel to the minimum of the next candidate depth map value and the minimum depth map value matrix element corresponding to the pixel; and

a second math processor incrementing the support matrix element corresponding to the pixel by one if the pixel is passed by the rendered depth map;

a fourth processor comprising the following for each pixel in the desired view;

an update device setting the fused depth map value at said pixel to be equal to the corresponding element of the next candidate depth map if the value of the support matrix at said the pixel is greater than zero.

19. A system for constructing a graphical representation of an object, as perceived from a desired view, from a plurality of calibrated views, comprising:

a measurement device obtaining camera calibration data for a desired view of the object;

a first processor computing a plurality of depth maps, each depth map respectively corresponding to each of a plurality of calibrated views of the object;

a merging device rendering a selected, computed depth map into the desired view for each of the plurality of computed depth maps; and

a third processor computing a median fused depth map value at each pixel in the desired view, wherein as many rendered depth maps occlude said pixel as pass it by.