3D image generating method, 3D animation generating method, and both 3D image generating module and 3D animation generating module thereof

• US 8,711,141 B2
• Filed: 08/28/2011
• Issued: 04/29/2014
• Est. Priority Date: 08/28/2011
• Status: Active Grant

• Current Assignee: Arcsoft Corporation Limited
• Original Assignee: ArcSoft, Inc.
• Inventors: Lin, Jian-Hua, Wang, Jin, Xie, YanTao
• Primary Examiner(s): Zarka, David
• Assistant Examiner(s): Gill, Andrew J
• Application Number:
  US13/219,709
• Publication Number:
  US 20130050185A1
• Time in Patent Office: 975 Days
• Field of Search: 345/419
• US Class Current: 345/419
• CPC Class Codes: G06T 13/20 3D [Three Dimensional] animation
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 7/41 based on statistical description of texture
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion

Abstract

For transforming a 2D image into a 3D image, or for transforming a 2D animation into a 3D animation, depths of the 2D image or the 2D animation are required and corrected by enhancing edges. Moreover, a 3D model is generated according to a corrected 2D depth map, and the 3D model is then filmed by virtual cameras to generate the 3D image or the 3D animation. While generating the 3D animation, key frames are selected for reducing processing time, and depths of unselected frames between the key frames are calculated by using interpolation.

Claims

1. A 3D image generating method comprising:

estimating depths of a 2D image to generate a 2D depth map having estimated depths;

correcting the estimated depths of the 2D depth map to generate a corrected 2D depth map, by merging blocks neighboring to a marked figure line on the 2D depth map;

creating a 3D model by connecting lines between pixels on the corrected 2D depth map having a depth difference smaller than a lower bound depth difference;

wherein creating the 3D model by connecting lines between pixels on the corrected 2D depth map comprises;

(a) physically connecting first and second pixel sets of a first rectangular unit to form a diagonal if a depth difference of the first and second pixel sets is smaller than the lower bound depth difference and is smaller than a lower bound depth difference of third and fourth pixel sets of the first rectangular unit; and

(b) physically connecting the first and third pixel sets, and the second and third pixel sets to form a triangle if a depth difference of the first and third pixel sets and a depth difference of the second and third pixel sets are both smaller than the lower bound depth difference;

determining at least one location;

virtually positioning at least one virtual camera at the at least one location; and

virtually recording the 3D model using the at least one virtual camera virtually positioned at the at least one location to generate a 3D image.

2. The method of claim 1 wherein estimating the depths on the 2D image to generate the 2D depth map comprises:

estimating the depths of the 2D image according to coverage between objects, relative sizes of objects, texture gradients, line transparency, air transparency, and/or object recognition.

3. The method of claim 1 wherein correcting the estimated depths of the 2D depth map to generate the corrected 2D depth map comprises:

segmenting the 2D depth map into a plurality of blocks according to a mean-shift algorithm;

generating the marked figure line on the 2D depth map according to a figure line drawn on the 2D image;

merging blocks neighboring to the marked figure line to generate merged blocks and defining a foreground region and a background region for each of the merged blocks;

expanding the marked figure line on the 2D depth map; and

recalculating depth in a plurality of marginal blocks of the 2D depth map using Bayesian Matting algorithm.

4. The method of claim 3 wherein merging the blocks neighboring to the marked figure line to generate the merged blocks and defining the foreground region and the background region for each of the merged blocks comprises:

scanning each block of the 2D depth map, and searching for one neighboring block having a smallest depth difference;

merging the scanned block with the neighboring block having the smallest depth difference to generate a merged block;

scanning the merged block to determine whether the merged block includes at least one marked block; and

classifying blocks of the merged block into a foreground region and a background region until all blocks of the merged block are classified.

5. The method of claim 3 wherein expanding the marked figure line on the 2D depth map comprises:

scanning at least one of the plurality of marginal blocks along the marked figure line on the 2D depth map to generate at least one scanned marginal block;

fetching foreground regions and background regions next to the at least one scanned marginal block;

determining a primary color for each foreground region, and determining a proportion of the primary color for a background region corresponding to the foreground region; and

expanding the at least one scanned marginal block according to determined proportions.

6. The method of claim 5 wherein expanding the at least one scanned marginal block according to the determined proportion comprises:

expanding the at least one scanned marginal block till a circle is formed in each merged block, the circle being centered at one of the at least one scanned marginal block of the merged block, and having a radius equivalent to half a side length of the merged block.

7. The method of claim 6 wherein the proportion is between 30% and 80%.

8. The method of claim 5 wherein expanding the at least one scanned marginal block according to the determined proportion comprises:

expanding the at least one scanned marginal block till a circle is formed in each merged block, the circle being centered at one of the at least one marginal block of the merged block, and having a radius equivalent to a quarter of a side length of the merged block.

9. The method of claim 8 wherein the proportion is between 10% and 30%.

10. The method of claim 5 wherein expanding the at least one scanned marginal blocks according to the determined proportion comprises:

expanding the at least one scanned marginal block till a circle is formed in each merged block, the circle being centered at one of the at least one scanned marginal blocks of the merged block, and having a radius equivalent to a pixel unit length.

11. The method of claim 10 wherein the proportion is below 10%.

12. The method of claim 1 wherein creating the 3D model by connecting lines between pixels on the corrected 2D depth map further comprises:

(c) eliminating connected lines and keeping a boundary of the triangle.

13. The method of claim 1 wherein determining the at least one location comprises:

determining the at least one location according to a resolution of the virtual camera, a size of the 3D model, a number of the at least one virtual camera, at least one virtual distance between the at the one virtual camera and the 3D model, and at least one virtual visual angle from the at least one virtual camera to the 3D model.

14. A 3D animation generating method comprising:

selecting a plurality of frames from a 2D animation according to average pixel differences each being an average pixel difference between two frames of the 2D animation;

estimating depths of each selected frame to generate a 2D depth map having estimated depths;

correcting the estimated depths of the 2D depth map to generate a corrected 2D depth map, by merging blocks neighboring to a marked figure line on the 2D depth map;

generating a 2D depth map for each unselected frame according to two corrected 2D depth maps corresponding to two selected frames next to the unselected frame through interpolation;

creating a 3D model by connecting lines between pixels on the corrected 2D depth map and by connecting lines between pixels on the 2D depth map of the unselected frame, the pixels on the corrected 2D depth map having a depth difference smaller than a lower bound depth difference, the pixels on the 2D depth map of the unselected frame having a depth difference smaller than the lower bound depth difference;

wherein creating the 3D model by connecting lines between pixels on the corrected 2D depth map and by connecting lines between pixels on the 2D depth map of the unselected frame comprises;

(a) physically connecting first and second pixel sets of a first rectangular unit on the corrected 2D depth map or on the 2D depth map of the unselected frame to form a diagonal if a depth difference of the first and second pixel sets is smaller than the lower bound depth difference and is smaller than a lower bound depth difference of third and fourth pixel sets of the first rectangular unit; and

(b) physically connecting the first and third pixel sets, and the second and third pixel sets to form a triangle if a depth difference of the first and third pixel sets and a depth difference of the second and third pixel sets are both smaller than the lower bound depth difference;

determining at least one location;

virtually positioning at least one virtual camera at the at least one location; and

virtually recording the 3D model using the at least one virtual camera virtually positioned at the at least one location to generate a 3D animation.

15. The method of claim 14 wherein selecting the plurality of frames from the 2D animation according to the average pixel differences each being the average pixel difference between the two frames of the 2D animation comprising:

selecting the plurality of frames from the 2D animation while the average pixel differences is larger than a critical pixel difference, wherein any frame between two consecutive selected frames has a pixel difference smaller than the critical difference with each of the consecutive selected frames.

16. The method of claim 14 wherein estimating the depths of each the selected frame to generate the 2D depth map having estimated depths comprises:

estimating the depths of each the selected frame according to coverage between objects, relative sizes of objects, texture gradients, line transparency, air transparency, and object recognition.

17. The method of claim 14 wherein correcting the estimated depths of the 2D depth map to generate the corrected 2D depth map comprises:

segmenting the 2D depth map into a plurality of blocks according to a mean-shift algorithm;

generating the marked figure line on the 2D depth map according to a figure line drawn on the 2D image;

merging blocks neighboring to the marked figure line to generate merged blocks and defining a foreground region and a background region for each of the merged blocks; and

expanding the marked figure line on the 2D depth map.

18. The method of claim 17 wherein expanding the marked figure line on the 2D depth map comprises:

scanning marginal blocks along the marked figure line on the 2D depth map;

fetching foreground regions and background regions next to the marginal blocks;

determining a primary color for each foreground region, and determining a proportion of the primary color for a background region corresponding to the foreground region; and

expanding the marginal blocks according to determined proportions.

19. The method of claim 18 wherein expanding the marginal blocks according to the determined proportion comprises:

expanding the marginal blocks till a circle is formed in each merged block, the circle being centered at one of marginal blocks of the merged block, and having a radius equivalent to half a side length of the merged block.

20. The method of claim 19 wherein the proportion is between 30% and 80%.

21. The method of claim 18 wherein expanding the marginal blocks according to the determined proportion comprises:

expanding the marginal blocks till a circle is formed in each merged block, the circle being centered at one of marginal blocks of the merged block, and having a radius equivalent to a quarter of a side length of the merged block.

22. The method of claim 21 wherein the proportion is between 10% and 30%.

23. The method of claim 18 wherein expanding the marginal blocks according to the determined proportion comprises:

expanding the marginal blocks till a circle is formed in each merged block, the circle being centered at one of marginal blocks of the merged block, and having a radius equivalent to a pixel unit length.

24. The method of claim 23 wherein the proportion is below 10%.

25. The method of claim 17 wherein merging the blocks neighboring to the marked figure line to generate the merged blocks and defining the foreground region and the background region for each of the merged blocks comprises:

scanning each block of the 2D depth map, and searching for one neighboring block having a smallest depth difference;

merging the scanned block with the neighboring block having the smallest depth difference to generate a merged block;

scanning the merged block to determine whether the merged block includes at least one marked block; and

classifying blocks of the merged block into a foreground region and a background region until all blocks of the merged block are classified.

26. The method of claim 14 wherein generating the 2D depth map for each unselected frame according to both the corrected 2D depth maps corresponding to both the selected frames next to the unselected frame through interpolation comprises:

determining a depth of a first pixel on the 2D depth map for each the unselected frame according to;

P_i=((n−

i)²·

I_k+i²·

I_k+1)/((n−

i)²+i²)

wherein P_i indicates the first pixel;

I_k indicates a second pixel on a first one of the key frames;

I_k+1 indicates a third pixel on a second one of the key frames;

the first, second, and third pixels indicate a same pixel location on respective frames;

i indicates an amount of frames between the first frame and the non-key frame; and

n indicates an amount of frames between the first and second key frames.

27. The method of claim 14 wherein creating the 3D model by connecting lines between pixels on the corrected 2D depth map and by connecting lines between pixels on the 2D depth map of the unselected frame further comprises:

(c) eliminating connected lines and keeping a boundary of the triangle.

28. The method of claim 14 wherein determining the at least one location comprises:

determining the at least one location according to a resolution of the virtual camera, a size of the 3D model, a number of the at least one virtual camera, at least one virtual distance between the at the one virtual camera and the 3D model, and at least one virtual visual angle from the at least one virtual camera to the 3D model.