METHOD FOR GENERATING DEPTH MAPS FROM MONOCULAR IMAGES AND SYSTEMS USING THE SAME

• US 20130162629A1
• Filed: 11/18/2009
• Published: 06/27/2013
• Est. Priority Date: 11/18/2009
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Ludovic Angot
• Inventors: Huang, Wei-Jia, Liu, Kai-Che, Angot, Ludovic
• Granted Patent: US 9,030,469 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20028 Bilateral filtering
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion

Abstract

Methods for generating depth maps from monocular still image or monocular video and systems using the same are provided. First, an initial depth map is estimated or arbitrary defined. For video inputs, motion information can be used, for still image the initial background can be arbitrary set by default, chosen by the user or can be estimated. Estimation of the initial depth map can be based on a contrast map or a blur map. The initial depth map defines initial depth values for the respective pixels of the monocular image or monocular motion picture frames. The respective pixels of the original image or video frame data are mapped to the initial depth map according to positions of the pixels, in order to obtain corresponding initial depth values. An image data space of the image is subdivided into a plurality of sub-image spaces, and the initial depth value of each of the pixels of the image is filtered according to the initial depth values of the pixels located in the same sub-image space, in order to obtain depth information for the pixels.

Claims

1. A method for generating depth maps, comprising of the following steps:

obtaining, using a computer, an image comprising a plurality of pixels;

obtaining, using the computer, an initial depth map, wherein the initial depth map defines initial depth values for the plurality of pixels;

mapping, using the computer, the plurality of pixels of the image to the initial depth map according to positions of the plurality of pixels to obtain the corresponding initial depth values of the plurality of pixels; and

subdividing, using the computer, an image data space of the image into a plurality of sub-image spaces, and filtering the initial depth values of the plurality of pixels of the image according to the corresponding initial depth values of the plurality of pixels located in the same sub-image space to obtain depth information for the plurality of pixels.

2. The method of claim 1, wherein the step of subdividing the image data space, and filtering the initial depth values of the plurality of pixels of the image comprises the steps of:

mapping, using the computer, the plurality of pixels to at least one 3D grid based on pixel intensity, wherein the 3D grid comprises multiple cells;

calculating, using the computer, an average depth value for each cell of the 3D grid according to the initial depth values of the plurality of pixels in the cells respectively; and

replacing, using the computer, the initial depth values of the plurality of pixels of the image data by the average depth value of the 3D grid where the pixel is located to obtain the depth information for the plurality of pixels.

3. The method of claim 2, further comprising smoothing, using the computer, the average depth value of each cell of the 3D grid according to corresponding surrounding cells.

4. The method of claim 1, further comprising the steps of:

calculating, using the computer, motion vectors for the plurality of pixels of the image according to the image and a second image; and

adjusting, using the computer, the initial depth values of the plurality of pixels in the initial depth map according to the motion vectors for the plurality of pixels.

5. The method of claim 4, further comprising:

segmenting, using the computer, the image so that the plurality of pixels are grouped into segments; and

calculating, using the computer, an average motion vector for each of the segments according to the motion vectors of the plurality of pixels in each of the segments respectively, and assigning the average motion vector to each of the plurality of pixels in each of the segments.

6. The method of claim 1, further comprising the steps of:

calculating, using the computer, an average motion vector for at least one region of interest (ROI) in the image; and

selecting, using the computer, the initial depth map from a plurality of candidate initial depth maps according to the average motion vector of the ROI.

7. The method of claim 6, further comprising the steps of:

determining, using the computer, a tag for the ROI in the image according to the average motion vector of the ROI; and

selecting, using the computer, a specific candidate initial depth map among several initial depth maps, wherein the specific candidate initial depth map defines the ROI and a corresponding tag representing the depth, and the tag of the ROI in the specific candidate initial depth map conforms to the tag of the ROI in the image.

8. The method of claim 1, further comprising the steps of:

calculating, using the computer, motion vectors for the plurality of pixels of the image according to the image and a second image; and

constructing, using the computer, the initial depth map according to the motion vectors of each of the plurality of pixels of the image respectively.

9. The method of claim 1, further comprising generating, using the computer, the initial depth map according to contrast of each of the plurality of pixels of the image respectively.

10. The method of claim 1, further comprising generating the initial depth map according to blur information of each of the plurality of pixels of the image respectively.

11. The method of claim 1, further comprising generating the initial depth map according to texture gradient of each of the plurality of pixels of the image respectively.

12. A method for generating depth maps, comprising:

obtaining, using the computer, an image comprising a plurality of pixels;

obtaining, using the computer, an initial depth map, wherein the initial depth map defines initial depth values for the plurality of pixels;

mapping, using the computer, the plurality of pixels of the image to the initial depth map according to positions of the plurality of pixels to obtain the corresponding initial depth values of the plurality of pixels; and

filtering, using the computer, the corresponding initial depth values of the plurality of pixels of the image using a bilateral filter to obtain depth information for the plurality of pixels.

13. The method of claim 12, further comprising accelerating the filtering process of the initial depth values of the plurality of pixels of the image using a bilateral grid method.

14. The method of claim 12, further comprising adjusting the initial depth values of the plurality of pixels in the initial depth map according to a motion vector for one pixel of the plurality of pixels between a first image and a second image part of a sequence in a motion picture.

15. A depth map generation system, comprising:

a non-transitory storage unit comprising an image having a plurality of pixels, and an initial depth map defining initial depth values for the plurality of pixels, wherein the image is obtained from video data; and

a processing unit mapping the plurality of pixels of the image to the initial depth map according to positions of the plurality of pixels to obtain the corresponding initial depth values of the plurality of pixels, subdividing an image data space of the image into a plurality of sub-image spaces and filtering the initial depth values of the plurality of pixels of the image according to the corresponding initial depth values of the plurality of pixels located in the same sub-image space to obtain depth information for the plurality of pixels.

16. The system of claim 15, wherein the processing unit further maps the plurality of pixels of the image to at least one 3D grid with multiple cells based on pixel intensity, calculates an average depth value for each cell of the 3D grid according to the initial depth values of each corresponding pixels in the calculated cell respectively, and replaces the initial depth value of each of the plurality of pixels of the image with the average depth value located in the same sub-image space to obtain the depth information for the pixels.

17. The system of claim 16, wherein the filtering operation of the processing unit consists of smoothing the average depth value of each cell of the 3D grid according to corresponding surrounding cells in the 3D grid.

18. The system of claim 15, wherein the storage unit further comprises the video data and the processing unit further calculates motion vectors for the plurality of pixels, groups of the plurality of pixels or segments of the image according to the image of two correlated video frames separated in time, and adjusts the initial depth values of the plurality of pixels in the initial depth map according to the motion vectors for the plurality of pixels, the groups of the plurality of pixels or the segments of the image.

19. The system of claim 18, wherein the processing unit further segments the plurality of pixels of the image, calculates an average motion vector for each of the segments according to the motion vectors of the plurality of pixels in each of the segments, and assigns the average motion vector to each of the plurality of pixels in each of the segments.

20. The system of claim 15, wherein the processing unit further calculates an average motion vector for at least one ROI (region of interest) in the image, and selects the initial depth map from a plurality of candidate initial depth maps according to the average motion vector of the ROI.

21. The system of claim 20, wherein the processing unit further determines a tag for the ROI in the image according to the average motion vector of the ROI, and selects a specific candidate initial depth map as the initial depth map, wherein each of the candidate initial depth maps defines the ROI and a corresponding tag representing a near side or a far side, and the tag of the ROI in the specific candidate depth map conforms to the tag of the ROI in the image.

22. The system of claim 15, wherein the processing unit further calculates a motion vector for each of the plurality of pixels of the image according to the image and a second image, and constructs the initial depth map according to the motion vectors of each of the plurality of pixels of the image respectively.

23. The system of claim 15, wherein the processing unit further generates the initial depth map according to contrast, intensity, blur information, or texture gradient of each of the plurality of pixels of the image respectively.

24. A depth map generation system, comprising:

a non-transitory storage unit comprising an image comprising a plurality of pixels, and a initial depth map defining initial depth values for the plurality of pixels;

a processing unit mapping the plurality of pixels of the image to the initial depth map according to positions of the plurality of pixels to obtain the corresponding initial depth values, and filtering the corresponding initial depth values of the plurality of pixels of the image using a bilateral filter to obtain depth information for the plurality of pixels.

25. The system of claim 24, wherein the processing unit further accelerates the filtering process of the initial depth values of the plurality of pixels of the image using a bilateral grid method.

26. The system of claim 24, wherein the processing unit further adjusts the initial depth values of the plurality of pixels in the initial depth map according to a motion vector for the pixel in the plurality of pixels between the image and second image.

27. A depth map generation program stored in a non-transitory machine-readable storage medium, the program comprising instructions for causing a device to perform:

obtaining image comprising a plurality of pixels;

obtaining an initial depth map, wherein the initial depth map defines initial depth values for the plurality of pixels;

mapping the plurality of pixels of the image to the initial depth map according to positions of the plurality of pixels to obtain corresponding initial depth values of the plurality of pixels; and

subdividing an image data space of the image into a plurality of sub-image spaces, and filtering the initial depth values of the plurality of pixels of the image according to the corresponding initial depth values of the plurality of pixels located in the same sub-image space to obtain depth information for the plurality of pixels.

28. A depth map generation program stored in a non-transitory machine-readable storage medium, the program comprising instructions for causing a device to perform:

obtaining an image comprising a plurality of pixels;

obtaining an initial depth map, wherein the initial depth map defines initial depth values for the plurality of pixels;

mapping the plurality of pixels of the image to the initial depth map according to positions of the plurality of pixels to obtain the corresponding initial depth values of the plurality of pixels; and

filtering the corresponding initial depth values of the plurality of pixels of the image using a bilateral filter to obtain depth information for the plurality of pixels.

29. A method for generating depth maps, comprising:

obtaining an image comprising a plurality of pixels from a storage unit of a computer;

obtaining an initial depth map from the storage unit of the computer, wherein the initial depth map defines initial depth values for the plurality of pixels;

mapping, using a processing unit of the computer, at least some of the plurality of pixels of the image to the initial depth map according to positions of the plurality of pixels to obtain the corresponding initial depth values of the plurality of pixels; and

filtering, using the processing unit of the computer, the corresponding initial depth values of the mapped plurality of pixels to obtain depth information for the plurality of pixels.

30. The method of claim 29, wherein filtering the initial depth values comprises filtering with a bilateral filter using the processing unit of the computer.

31. The method of claim 29, further comprising accelerating the filtering process of the initial depth values of the plurality of pixels of the image using a bilateral grid method.

32. The method of claim 29, further comprising adjusting the initial depth values of the plurality of pixels in the initial depth map according to a motion vector for the pixel of the plurality of pixels between a first image and a second image part of a sequence in a motion picture.