Example-based two-dimensional to three-dimensional image conversion method, computer readable medium therefor, and system

• US 8,411,932 B2
• Filed: 01/09/2009
• Issued: 04/02/2013
• Est. Priority Date: 07/18/2008
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Liu, Kai-Che, Chen, Wen-Chao, Chen, Tsu-Han, Wu, Qi, Jan, Fu-Chiang, Wu, Cheng-Feng
• Primary Examiner(s): Chen, Wenpeng
• Application Number:
  US12/351,187
• Publication Number:
  US 20100014781A1
• Time in Patent Office: 1,544 Days
• Field of Search: 382/154, 382/294, 382/174, 345/422
• US Class Current: 382/154
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 2213/003 Aspects relating to the "2D+depth" image format

Abstract

An example-based 2D to 3D image conversion method, a computer readable medium therefore, and a system are provided. The embodiments are based on an image database with depth information or with which depth information can be generated. With respect to a 2D image to be converted into 3D content, a matched background image is found from the database. In addition, graph-based segmentation and comparison techniques are employed to detect the foreground of the 2D image so that the relative depth map can be generated from the foreground and background information. Therefore, the 3D content can be provided with the 2D image plus the depth information. Thus, users can rapidly obtain the 3D content from the 2D image automatically and the rendering of the 3D content can be achieved.

Claims

1. An image conversion method of converting an input two-dimensional image into three-dimensional image content, the method comprising the steps of:

obtaining from an image database, a matched background image, which most closely approximates content of the input two-dimensional image;

performing a rectification operation to obtain a rectified background image according to the matched background image and the input two-dimensional image, wherein the rectified background image and the input two-dimensional image have a same image-capturing plane;

performing a segmentation operation on the input two-dimensional image to obtain a segmented image, which comprises a plurality of patches;

performing a foreground and background detection operation to separate the patches of the segmented image into a foreground patch group and a background patch group according to the input two-dimensional image, the rectified background image and the segmented image; and

performing a depth estimation operation to set a depth value corresponding to each of the patches of the background patch group and the foreground patch group and to obtain a depth map of the input two-dimensional image according to depth information provided by the rectified background image,

wherein the three-dimensional image content is based on the input two-dimensional image and the depth map.

2. The method according to claim 1, further comprising the step of:

performing an object detection operation on the foreground patch group to determine whether there is any patch of the foreground patch group satisfying at least one object feature, performing a boundary smoothing process on each patch satisfying the at least one object feature, and thus adjusting the foreground patch group into an adjusted foreground patch group for processing in the step of performing the depth estimation operation.

3. The method according to claim 1, wherein the step of performing the foreground and background detection operation comprises:

performing a subtraction operation according to the input two-dimensional image and the rectified background image; and

performing a foreground and background classification according to a result of the subtraction operation and the segmented image to classify the patches of the segmented image into the foreground patch group and the background patch group.

4. The method according to claim 1, wherein the segmentation operation is performed before or after the step of performing the rectification operation and the step of obtaining the matched background image.

5. The method according to claim 1, wherein the segmentation operation is performed in parallel with the step of performing the rectification operation and the step of obtaining the matched background image.

6. A non-transitory computer readable medium having computer executable data, which comprises codes of executing each step recited in the image conversion method of claim 1.

7. The medium according to claim 6, wherein the image conversion method further comprises the step of:

determining whether there is any patch of the foreground patch group satisfying at least one object feature, performing a boundary smoothing process on each patch satisfying the object feature, and thus adjusting the foreground patch group for processing in the depth estimation operation.

8. An image conversion method of converting an input two-dimensional image into three-dimensional image content, the method comprising the steps of:

selecting one of a plurality of example background images of an image database as a matched background image;

performing a rectification operation to rectify the matched background image, so that the rectified matched background image and the input two-dimensional image are located on the same reference plane, to obtain a rectified background image according to the matched background image and the input two-dimensional image;

performing a segmentation operation on the input two-dimensional image to obtain a segmented image, which comprises a plurality of patches;

performing a foreground and background detection operation to separate the patches of the segmented image into a foreground patch group and a background patch group according to the input two-dimensional image, the rectified background image, and the segmented image; and

performing a depth estimation operation to set a depth value corresponding to each of the patches of the background patch group and the foreground patch group and to obtain a depth map of the input two-dimensional image according to depth information provided by the rectified background image,

wherein the three-dimensional image content is based on the input two-dimensional image and the depth map.

9. The method according to claim 8, wherein the image database comprises the depth information of the example background images.

10. The method according to claim 9, wherein the depth information of one of the example background images comprises a ground-vertical boundary between a ground and a vertical object in the example background image.

11. The method according to claim 9, wherein the depth information of the example background images are given in advance.

12. The method according to claim 8, further comprising a method of generating the depth information of the example background images, wherein the method of generating the depth information comprises the steps of:

performing a vertical object and ground classification on the example background images, the step comprising;

separating the example background image into a plurality of patches and labeling each of the patches of the example background image as a vertical object or a ground object; and

for each of the example background images, determining a ground-vertical boundary for all the patches of the example background image according to the patches labeled as the vertical object and the patches labeled as the ground object, wherein the depth information of the example background image comprises the ground-vertical boundary.

13. The method according to claim 12, wherein the step of performing the vertical object and ground classification further comprises:

generating a classifier with the example background images serving as a training dataset by way of supervised learning, wherein an input of the classifier is one of the patches of each of the example background images in the training dataset, and an output of the classifier is one of labeling results of the vertical object or the ground object.

14. The method according to claim 12, wherein the step of determining the ground-vertical boundary comprises:

for each of the example background images, determining at least one boundary between a vertical object and a ground object for all the patches labeled as the vertical object and all the patches labeled as the ground object; and

for each of the example background images, determining the ground-vertical boundary for the example background image according to the at least one boundary of the example background image.

15. The method according to claim 8, wherein the depth information provided by the rectified background image comprises a ground-vertical boundary of the rectified background image, and the step of performing the depth estimation operation comprises:

setting the depth value corresponding to each of the patches of the background patch group according to the ground-vertical boundary of the rectified background image; and

setting the depth value corresponding to each of the patches of the foreground patch group,

wherein the depth map comprises the depth value corresponding to each of the patches of the background patch group and the foreground patch group.

16. The method according to claim 8, wherein the step of performing the foreground and background detection operation comprises:

performing a subtraction operation according to the input two-dimensional image and the rectified background image; and

performing a foreground and background classification according to a result of the subtraction operation and the segmented image to classify the patches of the segmented image into the foreground patch group and the background patch group.

17. The method according to claim 16, wherein in the step of performing the foreground and background detection operation, the subtraction operation calculates a pixel-based difference of the input two-dimensional image and the rectified background image, wherein the result of the subtraction operation represents difference values between corresponding pixel data of the input two-dimensional image and the rectified background image.

18. The method according to claim 17, wherein in the step of performing the foreground and background detection operation, a pixel position is classified as the background if the result of the subtraction operation at the pixel position is smaller than a threshold value, or otherwise the pixel position is classified as the foreground.

19. The method according to claim 18, wherein in the step of performing the foreground and background detection operation and when the foreground and background classification is being performed, one of the patches of the segmented image is classified as the foreground if a ratio of the number of pixels in the patch, classified as the foreground according to the subtraction operation, to the total number of pixels in the patch is greater than a ratio threshold value.

20. The method according to claim 16, wherein in the step of performing the foreground and background detection operation, the subtraction operation is a patch-based correlation subtraction operation according to a correlation window of one of the input two-dimensional image and the rectified background image, and a patch in a search window of the other one of the input two-dimensional image and the rectified background image, wherein the search window is greater than the correlation window, wherein results obtained after the correlation subtraction operation is performed on all the patches of one of the input two-dimensional image and the rectified background image represent differences between the input two-dimensional image and the rectified background image.

21. The method according to claim 20, wherein in the step of performing the foreground and background detection operation, the correlation window is classified as the foreground if the results of the correlation subtraction operation is smaller than a threshold value.

22. The method according to claim 16, wherein the step of performing the foreground and background classification comprises:

setting a ratio of the number of pixels classified as the foreground to the number of pixels classified as the background for each of the patches of the segmented image according to the result of the subtraction operation and the segmented image to obtain foreground information and thus to classify each of the patches of the segmented image into the foreground or the background.

23. The method according to claim 16, wherein in the step of performing the foreground and background detection operation, the subtraction operation is based on pixel or patch selectively.

24. The method according to claim 23, wherein in the step of performing the foreground and background detection operation, the result of the subtraction operation is compared with a threshold value to classify each of the patches of the segmented image into the foreground or the background.

25. The method according to claim 16, wherein the step of performing the foreground and background classification comprises:

merging the patches of the segmented image, classified as the foreground, into a patch belonging to the foreground patch group.

26. The method according to claim 16, wherein in the step of performing the foreground and background detection operation further comprises, before the step of performing the subtraction operation:

performing a color calibration on the input two-dimensional image and the rectified background image,

wherein the subtraction operation is performed according to the input two-dimensional image and the rectified background image, which have been color corrected.

27. The method according to claim 8, wherein one of the example background images of the image database is selected as the matched background image according to a texton histogram similarity and the number of corresponding feature point pairs, wherein the matched background image has a higher similarity and a greatest number of corresponding feature point pairs than those among the other example background images.

28. The method according to claim 8, wherein, the step of performing the rectification operation comprises:

comparing the matched background image with the input two-dimensional image to find a plurality of corresponding feature points; and

rectifying the matched background image, such that the rectified matched background image and the input two-dimensional image are located on the same reference plane, to obtain the rectified background image according to the corresponding feature points.

29. The method according to claim 8, wherein the example background images of the image database comprise an image shot by a user.

30. The method according to claim 8, wherein the depth information of the background images is obtained by a depth information estimation when the depth estimation operation is to be performed.

31. The method according to claim 30, wherein the depth information estimation of the rectified background image is obtained based on a vanishing point, an object size, or an object shielding relationship.

32. An image conversion method of converting an input two-dimensional image into three-dimensional image content, the method comprising the steps of:

selecting one of a plurality of example background images of an image database as a matched background image, wherein content of the matched background image, among the other the example background images, most closely approximates content of the input two-dimensional image;

performing a rectification operation to rectify the matched background image, such that the rectified matched background image and the input two-dimensional image are located on the same reference plane, to obtain a rectified background image according to the matched background image and the input two-dimensional image;

performing a segmentation operation on the input two-dimensional image to obtain a segmented image, which comprises a plurality of patches, wherein each of the patches respectively represents correspondingly homogeneous pixels included in the input two-dimensional image;

performing a foreground and background detection operation according to the input two-dimensional image, the rectified background image and the segmented image, wherein the step of performing the foreground and background detection operation comprises;

performing a subtraction operation on the input two-dimensional image and the rectified background image; and

performing a foreground and background classification according to a result of the subtraction operation and the segmented image in order to classify the patches of the segmented image as a foreground or a background, and separating the patches of the segmented image into a foreground patch group and a background patch group;

performing an object detection operation on the foreground patch group to determine whether there is any patch of the foreground patch group satisfying at least one object feature, and performing a boundary smoothing process on each patch satisfying the at least one object feature to obtain an adjusted foreground patch group, which is based on the patches which are processed by the boundary smoothing process; and

performing a depth estimation operation according to depth information provided by the rectified background image in order to assign a depth value corresponding to each of the patches of the background patch group and the adjusted foreground patch group and to obtain a depth map of the input two-dimensional image accordingly,

wherein the three-dimensional image content is based on the input two-dimensional image and the depth map.

33. The method according to claim 32, wherein the step of performing the object detection operation comprises:

performing a dedicated object detection operation to determine whether there is any patch of the foreground patch group satisfying a dedicated object feature, and performing the boundary smoothing process on a boundary of each patch satisfying the dedicated object feature.

34. The method according to claim 32, wherein the step of performing the object detection operation comprises:

performing a human feature detection operation to determine whether there is any patch of the foreground patch group satisfying a human feature, and performing the boundary smoothing process on a boundary of each patch satisfying the human feature.

35. The method according to claim 34, wherein in the step of performing the object detection operation, the human feature detection comprises a human body detection, and the human feature is a human-like contour.

36. The method according to claim 34, wherein in the step of performing the object detection operation, the human feature detection comprises a human face detection operation, and the human feature is a human face feature.

37. The method according to claim 34, wherein in the step of performing the object detection operation, the boundary smoothing process re-corrects a shape of the patch satisfying the human feature, based on principle component analysis (PCA).

38. The method according to claim 34, wherein in the step of performing the object detection operation, the boundary smoothing process comprises:

training a principle component analysis model to obtain prior information according to known contour data of the human feature in order to depict a contour of the human feature by the principle component analysis model, wherein the contour of the human feature is regarded as a lower-order principle component, and a zigzag contour is regarded as a high-order component.

39. The method according to claim 38, wherein in the step of performing the object detection operation, the boundary smoothing process comprises:

performing an edge detection operation on each patch satisfying the human feature, finding a control point according to the prior information, and performing a control point position adjustment according to the trained principle component analysis model to obtain a smoothed result.

40. An image conversion system for converting an input two-dimensional image into three-dimensional image content, the system comprising:

an image matching unit for determining a matched background image according to an image database, wherein the matched background image most closely approximates the input two-dimensional image in content,;

a rectifying unit for obtaining a rectified background image, which is on a same image-capturing plane as that of the input two-dimensional image, according to the matched background image and the input two-dimensional image;

a segmenting unit for performing a segmentation operation on the input two-dimensional image to obtain a segmented image, which comprises a plurality of patches;

a foreground and background detecting unit for separating the patches of the segmented image into a foreground patch group and a background patch group by performing a subtraction operation on the input two-dimensional image and the rectified background image according to the input two-dimensional image, the rectified background image and the segmented image; and

a depth estimating unit for setting a depth value corresponding to each of the patches of the background patch group and the foreground patch group to obtain a depth map of the input two-dimensional image according to depth information provided by the rectified background image,

wherein the three-dimensional image content is based on the input two-dimensional image and the depth map.

41. The system according to claim 40, further comprising:

an object detecting unit for determining whether there is any patch of the foreground patch group satisfying at least one object feature, and performing a boundary smoothing process on each patch satisfying the object feature, and thus adjusting the foreground patch group to obtain an adjusted foreground patch group to be processed by the depth estimating unit.

42. The system according to claim 40, wherein the image conversion system is a field programmable gate array.

43. An image conversion system for converting a two-dimensional image into three-dimensional image content, the system comprising:

a memory device; and

a microprocessor, coupled to the memory device, for reading commands stored in the memory device to;

determine a matched background image, which most closely approximates content of the two-dimensional image, according to a digital image database;

perform a rectification operation according to the matched background image and the two-dimensional image to obtain a rectified background image, wherein the rectified background image and the two-dimensional image are on a same image-capturing plane;

perform a segmentation operation on the two-dimensional image to obtain a segmented image, wherein the segmented image comprises a plurality of patches, and each of the patches respectively represents a portion of pixels in the two-dimensional image correspondingly having similarity;

perform a subtraction operation on the two-dimensional image and the rectified background image according to the two-dimensional image, the rectified background image and the segmented image, and to perform a foreground and background detection operation to separate the patches of the segmented image into a foreground patch group and a background patch group; and

perform a depth estimation operation to set a depth value corresponding to each of the patches of the background patch group and the foreground patch group to obtain a depth map of the two-dimensional image according to depth information provided by the rectified background image,

wherein the three-dimensional image content is based on the two-dimensional image and the depth map.

44. The system according to claim 43, wherein the microprocessor is further for:

determining whether there is any patch of the foreground patch group satisfying at least one object feature, performing a boundary smoothing process on each patch satisfying the object feature, and thus adjusting the foreground patch group for processing of the depth estimation operation.

45. The system according to claim 43 being an embedded system.

46. The system according to claim 43 being a microprocessor.