Depth map generation techniques for conversion of 2D video data to 3D video data

• US 8,537,200 B2
• Filed: 01/29/2010
• Issued: 09/17/2013
• Est. Priority Date: 10/23/2009
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Zhang, Rong, Chen, Ying, Karczewicz, Marta
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): Jebari, Mohammed
• Application Number:
  US12/696,499
• Publication Number:
  US 20110096832A1
• Time in Patent Office: 1,327 Days
• Field of Search: None
• US Class Current: 348/42
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion

Abstract

This disclosure describes techniques for generating depth maps for video units, such as video frames or slices video frames. The techniques may be performed by a video encoder in order to convert two-dimensional (2D) video to three-dimensional (3D) video. The techniques may alternatively be performed by a video decoder in order to convert received 2D video to 3D video. The techniques may use a combination of motion and color considerations in the depth map generation process.

Claims

1. A method comprising:

assigning initial depth values to pixels of a video unit based on motion associated with the pixels relative to corresponding pixels of a previous video unit of a video sequence;

identifying whether the initial depth values satisfy a threshold;

assigning adjusted depth values to one or more of the pixels of the video unit for which the initial depth values satisfy the threshold, wherein the adjusted depth values are based on color associated with the pixels;

generating a depth map for the video unit, wherein the depth map comprises an initialized depth map, wherein the initialized depth map comprises the initial depth values for a first subset of the pixels and the adjusted depth values for a second subset of the pixels;

determining whether the video unit corresponds to a scene change relative to the previous video unit;

if the video unit does not correspond to the scene change relative to the previous video unit, determining whether the video unit represents a low level of motion relative to the previous video unit; and

if the video unit does not represent the low level of motion, defining a final depth map based on weighted averages of the depth values of the initialized depth map for the video unit and depth values of a previous depth map for the previous video unit.

2. The method of claim 1, wherein assigning the initial depth values based on motion comprises assigning the initial depth values based on luma values of the pixels of the video unit relative to luma values of co-located pixels of the previous video unit.

3. The method of claim 1, wherein assigning the adjusted depth values based on color comprises assigning the adjusted depth values based at least in part on chroma values of the pixels of the video unit.

4. The method of claim 1,

wherein assigning the initial depth values based on motion comprises assigning the initial depth values based on luma values of the pixels of the video unit relative to luma values of co-located pixels of the previous video unit,

wherein assigning the adjusted depth values based on color comprises assigning the adjusted depth values based at least in part on chroma values of the pixels of the video unit, and

wherein assigning the adjusted depth values includes mapping chroma values within an allowed interval of chroma values to depth values between zero and the threshold.

5. The method of claim 1, further comprising:

if the video unit corresponds to a scene change relative to the previous video unit, selecting the initialized depth map as a final depth map for the video unit;

if the video unit represents the low level of motion, defining the final depth map based on maximums of the depth values of the initialized depth map for the video unit and the depth values of the previous depth map for the previous video unit.

6. The method of claim 5, wherein the method is performed by a video decoder, the method further comprising performing view synthesis based on the final depth map to generate a secondary view for the video unit.

7. The method of claim 5, wherein the method is performed by a video encoder, the method further comprising communicating the final depth map with the video unit.

8. The method of claim 5, wherein the method is performed by a video encoder, the method further comprising:

performing view synthesis based on the final depth map to generate a secondary view for the video unit; and

communicating the secondary view with the video unit.

9. A video coder apparatus comprising a depth map generation unit and a processor capable of:

assigning initial depth values to pixels of a video unit based on motion associated with the pixels relative to corresponding pixels of a previous video unit of a video sequence;

identifying whether the initial depth values satisfy a threshold;

assigning adjusted depth values to one or more of the pixels of the video unit for which the initial depth values satisfy the threshold, wherein the adjusted depth values are based on color associated with the pixels;

generating a depth map for the video unit, wherein the depth map comprises an initialized depth map, wherein the initialized depth map comprises the initial depth values for a first subset of the pixels and the adjusted depth values for a second subset of the pixels;

determining whether the video unit corresponds to a scene change relative to the previous video unit;

if the video unit does not correspond to the scene change relative to the previous video unit, determining whether the video unit represents a low level of motion relative to the previous video unit; and

if the video unit does not represent the low level of motion, defining a final depth map based on weighted averages of the depth values of the initialized depth map for the video unit and depth values of a previous depth map for the previous video unit.

10. The video coder apparatus of claim 9, wherein in assigning the initial depth values based on motion, the depth map generation unit assigns the initial depth values based on luma values of the pixels of the video unit relative to luma values of co-located pixels of the previous video unit.

11. The video coder apparatus of claim 9, wherein assigning adjusted depth values based on color, the depth map generation unit assigns the adjusted depth values based at least in part on chroma values of the pixels of the video unit.

12. The video coder apparatus of claim 9,

wherein assigning the initial depth values based on motion comprises assigning the initial depth values based on luma values of the pixels of the video unit relative to luma values of co-located pixels of the previous video unit,

wherein assigning the adjusted depth values based on color comprises assigning the adjusted depth values based at least in part on chroma values of the pixels of the video unit, and

wherein assigning the adjusted depth values includes mapping chroma values within an allowed interval of chroma values to depth values between zero and the threshold.

13. The video coder apparatus of claim 9, wherein the depth map generation unit:

if the video unit corresponds to a scene change relative to the previous video unit, selects the initialized depth map as a final depth map for the video unit; and

if the video unit represents the low level of motion, defines the final depth map based on maximums of the depth values of the initialized depth map for the video unit and the depth values of the previous depth map for the previous video unit.

14. The video coder apparatus of claim 13, wherein the video coder comprises a video decoder, wherein the video decoder further comprises:

a two-dimension (2D) to three-dimension conversion unit that performs view synthesis based on the final depth map to generate a secondary view for the video unit.

15. The video coder apparatus of claim 13, wherein the video coder comprises a video encoder, wherein video coding apparatus further comprises a transmitter that communicates the final depth map with the video unit.

16. The video coder apparatus of claim 13, wherein the video coder comprises a video encoder, wherein the video encoder further comprises a two-dimension (2D) to three-dimension conversion unit that performs view synthesis based on the final depth map to generate a secondary view for the video unit, and wherein the video coding apparatus further comprises a transmitter that communicates the secondary view with the video unit.

17. The video coder apparatus of claim 9, wherein the apparatus comprises at least one of:

an integrated circuit;

a microprocessor,

a wireless communication device that includes a video encoder, and

a wireless communication device that includes a video decoder.

18. A device comprising:

means for assigning initial depth values to pixels of a video unit based on motion associated with the pixels relative to corresponding pixels of a previous video unit of a video sequence;

means for identifying whether the initial depth values satisfy a threshold;

means for assigning adjusted depth values to one or more of the pixels of the video unit for which the initial depth values satisfy the threshold, wherein the adjusted depth values are based on color associated with the pixels;

means for generating a depth map for the video unit, wherein the depth map comprises an initialized depth map, wherein the initialized depth map comprises the initial depth values for a first subset of the pixels and the adjusted depth values for a second subset of the pixels;

means for determining whether the video unit corresponds to a scene change relative to the previous video unit;

means for determining whether the video unit represents a low level of motion relative to the previous video unit if the video unit does not correspond to the scene change relative to the previous video unit; and

means for defining a final depth map based on weighted averages of the depth values of the initialized depth map for the video unit and depth values of a previous depth map for the previous video unit if the video unit does not represent the low level of motion.

19. The device of claim 18, wherein the means for assigning the initial depth values based on motion comprises means for assigning the initial depth values based on luma values of the pixels of the video unit relative to luma values of co-located pixels of the previous video unit.

20. The device of claim 18, wherein the means for assigning the adjusted depth values based on color comprises means for assigning the adjusted depth values based at least in part on chroma values of the pixels of the video unit.

21. The device of claim 18,

wherein means for assigning the initial depth values based on motion comprises means for assigning the initial depth values based on luma values of the pixels of the video unit relative to luma values of co-located pixels of the previous video unit,

wherein means for assigning the adjusted depth values based on color comprises means for assigning the adjusted depth values based at least in part on chroma values of the pixels of the video unit, and

wherein means for assigning the adjusted depth values includes means for mapping chroma values within an allowed interval of chroma values to depth values between zero and the threshold.

22. The device of claim 18, further comprising:

means for selecting the initialized depth map as a final depth map for the video unit if the video unit corresponds to a scene change relative to the previous video unit; and

means for defining the final depth map based on maximums of the depth values of the initialized depth map for the video unit and the depth values of the previous depth map for the previous video unit if the video unit represents the low level of motion.

23. The device of claim 22, wherein the device comprises a video decoder, the device further comprising:

means for performing view synthesis based on the final depth map to generate a secondary view for the video unit.

24. The device of claim 22, wherein the device comprises a video encoder, the device further comprising means for communicating the final depth map with the video unit.

25. The device of claim 22, wherein the device comprises a video encoder, the device further comprising:

means for performing view synthesis based on the final depth map to generate a secondary view for the video unit; and

means for communicating the secondary view with the video unit.

26. A non-transitory computer-readable storage medium comprising instructions that upon execution by a processor cause the processor to:

assign initial depth values to pixels of a video unit based on motion associated with the pixels relative to corresponding pixels of a previous video unit of a video sequence;

identify whether the initial depth values satisfy a threshold;

assign adjusted depth values to one or more of the pixels of the video unit for which the initial depth values satisfy the threshold, wherein the adjusted depth values are based on color associated with the pixels;

generate a depth map for the video unit, wherein the depth map comprises an initialized depth map, wherein the initialized depth map comprises the initial depth values for a first subset of the pixels and the adjusted depth values for a second subset of the pixels;

determine whether the video unit corresponds to a scene change relative to the previous video unit;

if the video unit does not correspond to the scene change relative to the previous video unit, determine whether the video unit represents a low level of motion relative to the previous video unit; and

if the video unit does not represent the low level of motion, define a final depth map based on weighted averages of the depth values of the initialized depth map for the video unit and depth values of a previous depth map for the previous video unit.

27. The computer-readable storage medium of claim 26, wherein in assigning the initial depth values based on motion, the instructions cause the processor to assign the initial depth values based on luma values of the pixels of the video unit relative to luma values of co-located pixels of the previous video unit.

28. The computer-readable storage medium of claim 26, wherein in assigning the adjusted depth values based on color, the instructions cause the processor to assign the adjusted depth values based at least in part on chroma values of the pixels of the video unit.

29. The computer-readable storage medium of claim 26,

wherein assigning the initial depth values based on motion comprises assigning the initial depth values based on luma values of the pixels of the video unit relative to luma values of co-located pixels of the previous video unit,

wherein assigning the adjusted depth values based on color comprises assigning the adjusted depth values based at least in part on chroma values of the pixels of the video unit, and

wherein assigning the adjusted depth values includes mapping chroma values within an allowed interval of chroma values to depth values between zero and the threshold.

30. The computer-readable storage medium of claim 26, wherein the instructions cause the processor to:

if the video unit corresponds to a scene change relative to the previous video unit, select the initialized depth map as a final depth map for the video unit;

if the video unit represents the low level of motion, define the final depth map based on maximums of the depth values of the initialized depth map for the video unit and the depth values of the previous depth map for the previous video unit.

31. The computer-readable storage medium of claim 30, wherein the processor comprises a video decoder, wherein the instructions cause the processor to:

perform view synthesis based on the final depth map to generate a secondary view for the video unit.

32. The computer-readable storage medium of claim 30, wherein the processor comprises a video encoder, wherein the instructions cause the processor to communicate the final depth map with the video unit.

33. The computer-readable storage medium of claim 30, wherein the processor comprises a video encoder, wherein the instructions cause the processor to:

perform view synthesis based on the final depth map to generate a secondary view for the video unit; and

communicate the secondary view with the video unit.