2D to 3D video conversion

• US 8,659,592 B2
• Filed: 09/24/2009
• Issued: 02/25/2014
• Est. Priority Date: 09/24/2009
• Status: Active Grant

• Current Assignee: Shenzhen TCL New Technology Limited
• Original Assignee: Shenzhen TCL New Technology Limited
• Inventors: Wang, Haohong, Adler, Glenn
• Primary Examiner(s): BAYARD, DJENANE M
• Application Number:
  US12/585,784
• Publication Number:
  US 20110069152A1
• Time in Patent Office: 1,615 Days
• Field of Search: None
• US Class Current: 345/419
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion

Abstract

A method for real-time 2D to 3D video conversion includes receiving a decoded 2D video frame having an original resolution, downscaling the decoded 2D video frame into an associated 2D video frame having a lower resolution, and segmenting objects present in the downscaled 2D video frame into background objects and foreground objects. The method also includes generating a background depth map and a foreground depth map for the downscaled 2D video frame based on the segmented background and foreground objects, and deriving a frame depth map in the original resolution based on the background depth map and the foreground depth map. The method further includes providing a 3D video frame for display at a real-time playback rate. The 3D video frame is generated in the original resolution based on the frame depth map.

Claims

1. A computer-implemented method for real-time 2D to 3D video conversion, comprising:

receiving a decoded 2D video frame having an original resolution;

downscaling the decoded 2D video frame into an associated 2D video frame having a lower resolution;

storing in a video frame database the decoded 2D video frame in association with the downscaled 2D video frame, wherein the decoded 2D video frame is associated with the downscaled 2D video frame by a unique identifier that uniquely identifies the decoded 2D video frame having the original resolution and the associated downscaled 2D video frame having the lower resolution;

segmenting objects present in the downscaled 2D video frame into background objects and foreground objects;

generating a background depth map and a foreground depth map for the downscaled 2D video frame based on the segmented background and foreground objects;

retrieving the decoded 2D video frame having the original resolution based on the unique identifier;

deriving a frame depth map in the original resolution based on the retrieved decoded 2D video frame having the original resolution and both the background depth map and the foreground depth map that are generated in the downscaled lower resolution; and

providing a 3D video frame for display at a real-time playback rate, wherein the 3D video frame is generated in the original resolution based on the frame depth map.

2. The method of claim 1, further comprising:

decoding a received 2D video bitstream to provide the decoded 2D video frame having the original resolution.

3. The method of claim 1, further comprising:

processing the downscaled 2D video frame based on a pixel by pixel order for the segmenting.

4. The method of claim 1, further comprising:

receiving a plurality of decoded 2D video frames, the downscaling including downscaling the plurality of the decoded 2D video frames to provide an associated plurality of downscaled 2D video frames; and

performing the segmenting based on tracking moving objects in each of the plurality of the downscaled 2D video frames.

5. The method of claim 1, further comprising:

receiving a plurality of decoded 2D video frames, the downscaling including downscaling the plurality of the decoded 2D video frames to provide an associated plurality of downscaled 2D video frames; and

performing the segmenting based on tracking moving objects in at least one of the plurality of the downscaled 2D video frames and at least one neighboring downscaled frame, the at least one neighboring downscaled frame preceding or following the at least one downscaled 2D video frame.

6. The method of claim 1, wherein the generating the background depth map comprises:

applying structure analysis to the background objects segmented from the downscaled 2D video frame, wherein the segmented background objects form a background frame in the lower resolution; and

deriving the background depth map based on the structure analysis.

7. The method of claim 6, wherein the structure analysis comprises:

detecting, if present, a vanishing point and one or more vanishing lines of the background frame based on the segmented background objects, wherein the vanishing point represents a most distant point from an observer, and the one or more vanishing lines represent a direction of increasing depth.

8. The method of claim 7, wherein the detecting further comprises, if no vanishing point is detected, determining a default vanishing point as a convergent point on top of the background frame and a default vanishing line as a vertical line passing through the default vanishing point from top to bottom of the background frame.

9. The method of claim 1, further comprising:

associating the generated background depth map with the downscaled 2D video frame by a unique identifier that uniquely identifies the generated background depth map and the associated downscaled 2D video frame.

10. The method of claim 1, wherein the generating the foreground depth map comprises:

applying foreground skeleton depth estimation to the foreground objects segmented from the downscaled 2D video frame; and

deriving the foreground depth map based on the depth estimation.

11. The method of claim 10, wherein the foreground skeleton depth estimation comprises:

decomposing the foreground object into a skeleton and distances of boundary points from the skeleton in a horizontal direction, wherein the skeleton is vertically connected midpoints between two boundary points in the horizontal direction.

12. The method of claim 1, further comprising:

associating the generated foreground depth map with the downscaled 2D video frame by a unique identifier that uniquely identifies the generated foreground depth map and the associated downscaled 2D video frame.

13. The method of claim 1, wherein the deriving the frame depth map comprises:

fusing, in the original resolution, the background depth map and the foreground depth map; and

generating the frame depth map in the original resolution based on the fusing.

14. The method of claim 1, further comprising:

adjusting the derived frame depth map based at least one neighboring decoded 2D video frame with the original resolution and corresponding background and foreground depth maps of the at least one neighboring decoded 2D video frame, wherein the at least one neighboring decoded 2D video frame precedes or follows the decoded 2D video frame having the original resolution.

15. The method of claim 1, further comprising:

storing the generated frame depth map in association with the decoded 2D video frame having the original resolution by a unique identifier that uniquely identifies the generated frame depth map and the associated decoded 2D video frame; and

providing from storage the generated frame depth map for generating the 3D video frame.

16. The method of claim 1, further comprising:

adjusting the generated 3D video frame based at least one neighboring decoded 2D video frame in original resolution and a corresponding frame depth map of the at least one neighboring decoded 2D video frame, wherein the at least one neighboring decoded 2D video frame precedes or follows the decoded 2D video frame having the original resolution.

17. The method of claim 1, wherein providing the 3D video frame further comprises:

applying a depth image based rendering algorithm to the decoded 2D video frame and the frame depth map; and

adjusting the depth image based rendering based on at least one neighboring decoded 2D frame in original resolution and a corresponding frame depth map of the at least one neighboring decoded 2D frame, wherein the at least one neighboring decoded 2D video frame precedes or follows the decoded 2D video frame having the original resolution.

18. A 2D-to-3D video converter coupled to receive a decoded 2D video frame having an original resolution, the 2D-to-3D video converter comprising:

a downscaler to downscale the decoded 2D video frame into an associated 2D video frame having a lower resolution;

a video frame database to store the decoded 2D video frame in association with the downscaled 2D video frame, wherein the decoded 2D video frame is associated with the downscaled 2D video frame by a unique identifier that uniquely identifies the decoded 2D video frame having the original resolution and the associated downscaled 2D video frame having the lower resolution;

a scene object analyzer to segment objects present in the downscaled 2D video frame into background objects and foreground objects;

a first depth map generator to generate a background depth map and a foreground depth map for the downscaled 2D video frame based on the segmented background and foreground objects;

a second depth map generator to retrieve the decoded 2D video frame having the original resolution based on the unique identifier, and derive a frame depth map in the original resolution based on the retrieved decoded 2D video frame having the original resolution and both the background depth map and the foreground depth map that are generated in the downscaled lower resolution; and

a rendering engine to provide a 3D video frame for display at a real-time playback rate on a user device, wherein the 3D video frame is generated in the original resolution based on the frame depth map.

19. The 2D-to-3D video converter of claim 18, wherein the user device is one of a mobile device, a computer, or a video playing device.

20. The 2D-to-3D video converter of claim 18, wherein the converter is further coupled to receive a plurality of decoded 2D video frames, the downscaling including downscaling the plurality of the decoded 2D video frames to provide an associated plurality of downscaled 2D video frames;

and

wherein the scene object analyzer is configured to segment the objects based on tracking moving objects in each of the plurality of the downscaled 2D video frames.

21. The 2D-to-3D video converter of claim 18, wherein the converter is further coupled to receive a plurality of decoded 2D video frames, the downscaling including downscaling the plurality of the decoded 2D video frames to provide an associated plurality of downscaled 2D video frames;

and

wherein the scene object analyzer is configured to segment the objects based on tracking moving objects in at least one of the plurality of the downscaled 2D video frames and at least one neighboring downscaled frame, the at least one neighboring downscaled frame preceding or following the at least one downscaled 2D video frame.

22. The 2D-to-3D video converter of claim 18, wherein the first depth map generator is configured to:

apply structure analysis to the background objects segmented from the downscaled 2D video frame, wherein the segmented background objects form a background frame in the lower resolution; and

generate the background depth map based on the structure analysis.

23. The 2D-to-3D video converter of claim 22, wherein the first depth map generator is further configured to:

detect, if present, a vanishing point and one or more vanishing lines of the background frame based on the segmented background objects, wherein the vanishing point represents a most distant point from an observer, and the one or more vanishing lines represent a direction of increasing depth.

24. The 2D-to-3D video converter of claim 23, wherein the first depth map generator is further configured to:

determine, if no vanishing point is detected, a default vanishing point as a convergent point on top of the background frame and a default vanishing line as a vertical line passing through the default vanishing point from top to bottom of the background frame.

25. The 2D-to-3D video converter of claim 18, further comprising:

a depth map database to

store the generated background depth map and the generated foreground depth map in association with the downscaled 2D video frame by a unique identifier that uniquely identifies the generated background depth map, the generated foreground depth map, and the associated downscaled 2D video frame, and

store the derived frame depth map in association with the decoded 2D video frame having the original resolution by a unique identifier that uniquely identifies the generated frame depth map and the associated decoded 2D video frame.

26. The 2D-to-3D video converter of claim 18, wherein the first depth map generator is configured to:

apply foreground skeleton depth estimation to the foreground objects segmented from the downscaled 2D video frame; and

derive the foreground depth map based on the depth estimation.

27. The 2D-to-3D video converter of claim 26, wherein the first depth map generator is further configured to:

decompose the foreground object into a skeleton and distances of boundary points from the skeleton in a horizontal direction, wherein the skeleton is vertically connected midpoints between two boundary points in the horizontal direction.

28. The 2D-to-3D video converter of claim 18, wherein the second depth map generator is configured to:

fuse, in the original resolution, the background depth map and the foreground depth map; and

derive the frame depth map in the original resolution based on the fusing.

29. The 2D-to-3D video converter of claim 18, wherein the second depth map generator is configured to:

adjust the derived frame depth map based at least one neighboring decoded 2D video frame with the original resolution and corresponding background and foreground depth maps of the at least one neighboring decoded 2D video frame, wherein the at least one neighboring decoded 2D video frame precedes or follows the decoded 2D video frame having the original resolution.

30. The 2D-to-3D video converter of claim 18, wherein the rendering engine is configured to:

adjust the generated 3D video frame based at least one neighboring decoded 2D video frame in original resolution and a corresponding frame depth map of the at least one neighboring decoded 2D video frame, wherein the at least one neighboring decoded 2D video frame precedes or follows the decoded 2D video frame having the original resolution.

31. The 2D-to-3D video converter of claim 18, wherein the rendering engine is configured to:

apply a depth image based rendering algorithm to the frame depth map and the images in the associated decoded 2D video frame in the original resolution to generate the 3D video frame; and

adjust the depth image based rendering based on at least one neighboring decoded 2D frame in original resolution and a corresponding frame depth map of the at least one neighboring decoded 2D frame, wherein the at least one neighboring decoded 2D video frame precedes or follows the decoded 2D video frame having the original resolution.

32. A non-transitory computer readable medium storing instructions that, when executed, cause a computer to perform a method for real-time 2D to 3D video conversion, the method comprising:

receiving a decoded 2D video frame having an original resolution;

downscaling the decoded 2D video frame into an associated 2D video frame having a lower resolution;

storing in a video frame database the decoded 2D video frame in association with the downscaled 2D video frame, wherein the decoded 2D video frame is associated with the downscaled 2D video frame by a unique identifier that uniquely identifies the decoded 2D video frame having the original resolution and the associated downscaled 2D video frame having the lower resolution;

segmenting objects present in the downscaled 2D video frame into background objects and foreground objects;

generating a background depth map and a foreground depth map for the downscaled 2D video frame based on the segmented background and foreground objects;

retrieving the decoded 2D video frame having the original resolution based on the unique identifier;

deriving a frame depth map in the original resolution based on the retrieved decoded 2D video frame having the original resolution and both the background depth map and the foreground depth map that are generated in the downscaled lower resolution; and

providing a 3D video frame for display at a real-time playback rate, wherein the 3D video frame is generated in the original resolution based on the frame depth map.