Systems and methods for performing video conversion based on non-linear stretch information

• US 8,786,673 B2
• Filed: 01/07/2011
• Issued: 07/22/2014
• Est. Priority Date: 01/07/2011
• Status: Active Grant

• Current Assignee: Cyberlink Corporation
• Original Assignee: Cyberlink Corporation
• Inventors: Cheng, Chih-Yu, Huang, Ho Chao, Tseng, Wei-Hsin
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): PARIKH, DAKSHESH D
• Application Number:
  US12/986,524
• Publication Number:
  US 20120176471A1
• Time in Patent Office: 1,292 Days
• Field of Search: 382/154, 382/164, 382/285, 382/173, 382/118, 382/165, 382/275, 382/291, 382/181, 382/254, 382/298, 382/171, 382/174, 382/203, 382/225, 382/282, 382/288
• US Class Current: 348/43
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: G06T 3/04 Context-preserving transformations, e.g. by using an importance map panospheric to cylindrical image transformations G06T3/12
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 13/264 using the relative movement of objects in two video frames or fields
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

A method implemented in a computing system for converting two-dimensional (2D) video to three-dimensional (3D) format comprises sampling the 2D video, wherein the sampling is performed non-linearly in one or more directions. The method further comprises determining depth information of one or more objects within the 2D video based on sampling information and transforming the 2D video to a 3D-compatible format according to the sampling and the depth information.

Claims

1. A method implemented in a computing system for converting two-dimensional (2D) video to three-dimensional (3D) video, comprising:

sampling the 2D video according to a sampling density, wherein the sampling is performed non-linearly in one or more spatial directions, and wherein at least part of the sampling density is adjusted according to a saliency region;

determining depth information of one or more objects within the 2D video based on sampling information; and

transforming the 2D video to a 3D-compatible format according to the sampling and the depth information.

2. The method of claim 1, further comprising outputting a 3D-compatible video to a display device.

3. The method of claim 1, wherein 3D-compatible format comprises one of:

a side-by-side format, a checkerboard format, a line interlaced format, a 2D-plus-depth format, anaglyph format, and a page-flipping format.

4. The method of claim 3, wherein the side-by-side format comprises one of:

an over/under format and a left/right format.

5. The method of claim 1, wherein a sampling density for performing the non-linear sampling is adjusted according to a projection type.

6. The method of claim 5, wherein the projection type comprises one or more of:

a spherical projection, a cylindrical projection, a spheroid projection, a quadratic projection, and a paraboloid projection.

7. The method of claim 1, wherein the depth increases as the sampling density increases.

8. The method of claim 1, wherein the depth increases as the sampling density decreases.

9. The method of claim 1, wherein the saliency region is defined based on one or more of the following attributes associated with the 2D video:

motion estimation data;

facial features;

degree of focus;

a predefined central area;

a brightness map;

an order of objects;

a color temperature of region; and

a contrast map.

10. The method of claim 9, wherein facial features comprise:

location within the 2D video of one or more facial regions;

skin color;

face position;

facial size; and

facial shapes.

11. The method of claim 9, wherein contrast map are defined according to one or more of:

gradient analysis;

color contrast;

edge analysis;

frequency domain spectrum analysis of the 2D video;

first order edge detection; and

second order edge detection.

12. The method of claim 11, wherein edge analysis is performed based on Sobel analysis, Canny analysis, or Prewitt analysis.

13. The method of claim 9, wherein determining the order of objects is performed based on one or more of the following:

a vanishing point technique based on lines;

sizes of objects;

motion of the objects;

a presence of human features;

placement of objects;

an object'"'"'s shadow; and

occlusive relationships between objects.

14. The method of claim 1, wherein the non-linear sampling is performed according to an aspect ratio of the 2D video and an aspect ratio of a rendering target, wherein the aspect ratio of the rendering target is based on the aspect ratio of the display device.

15. A video system for converting two-dimensional (2D) video to three-dimensional (3D) format video, comprising:

a sampling module configured to perform non-linear sampling of the 2D video in one or more spatial directions according to a sampling density;

a depth processor configured to determine depth information relating to objects in the 2D video based on sampling information;

a saliency region identifier configured to identify one or more regions of interest with a frame, wherein the sampling module is further configured to adjust the sampling density for the sampling based on a saliency region; and

a graphics renderer for rendering a 3D video according to the sampling density and the depth information.

16. The system of claim 15, wherein the saliency region identifier identifies one or more regions of interest with the frame based on the order of objects.

17. The system of claim 15, wherein the sampling module adjusts the sampling density based on spherical projection or cylindrical projection such that the sampling density is maximized at approximately the middle of a high saliency region.

18. A method implemented in a computing system for converting two-dimensional (2D) video to three-dimensional (3D) video, comprising:

defining a high saliency region;

adjusting a sampling density according to the high saliency region and the depth information, wherein the sampling density is utilized for performing non-linear sampling in one or more spatial directions;

determining depth information relating to objects within the high saliency region based on sampling information;

sampling the 2D video according to the adjusted sampling density;

transforming the 2D video to a 3D-compatible format according to the sampling and depth information; and

outputting the transformed 2D video to a display device.

19. The method of claim 18, wherein 3D-compatible format comprises one of:

a side-by-side format, an over/under format, a checkerboard format, a line interlaced format, a 2D-plus-depth format, anaglyph format, and a page-flipping format.

20. The method of claim 18, wherein the high saliency region is defined based on one or more of the following attributes:

motion estimation data associated with the objects in the frame;

facial features relating to the objects;

degree of focus and blurring in the frame;

a predefine central area;

a brightness map;

an order associated with the objects;

a color temperature of region; and

a contrast map.

21. The method of claim 18, wherein adjusting a sampling density according to the high saliency region comprises adjusting the sampling density according to one or more of:

a spherical projection, a cylindrical projection, a spheroid projection, a quadratic projection, and a paraboloid projection.

22. The method of claim 21, wherein a mean associated with spherical projection or cylindrical projection corresponds to a position within the high saliency region.

23. The method of claim 22, wherein the mean associated with spherical projection or cylindrical projection corresponds to a center of the high saliency region.

24. The method of claim 21, further comprising:

determining an order of objects within the high saliency region;

positioning the mean based on a location of an object having a lowest depth within the high saliency region.

25. The method of claim 18, wherein adjusting a sampling density according to the high saliency region and the depth information comprises decreasing the sampling density as depth relating to objects decreases.