Depth map generation module for foreground object and method thereof

• US 8,478,024 B2
• Filed: 11/01/2011
• Issued: 07/02/2013
• Est. Priority Date: 12/31/2010
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Liu, Kai Che, Huang, Wei Jia, Wu, Chun Te, Wang, Chin Yuan
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): RICE, ELISA M
• Application Number:
  US13/286,501
• Publication Number:
  US 20120170832A1
• Time in Patent Office: 609 Days
• Field of Search: 382/103, 382/107, 382/232, 382/236, 382/238, 382/239, 382/154, 348/51, 375/240.03, 375/240.12
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/55 from multiple images

Abstract

The present invention discloses a depth map generation module for a foreground object and the method thereof. The depth map generation method for a foreground object comprises the following steps: receiving an image sequence data, wherein the image sequence data includes a plurality of image frames; selecting at least one key image frame from the image sequence data; providing at least one depth indicative information and a contour of a first segment in the at least one key image frame; and performing a signal processing steps by a microprocessor.

Claims

1. A depth map generation module for a foreground object configured to receive an image sequence data corresponding to a same scene, so as to generate a depth indicative information of a plurality of image frames in the image sequence data, wherein the depth map generation module for a foreground object comprises:

a microprocessor, comprising;

a sequence unit configured to receive the image sequence data and to selectively adjust the order of the image sequence data according to the operation mode of the depth map generation module for a foreground object, so as to generate a transformed image sequence data, wherein the transformed image sequence data comprises at least one key image frame and a first non-key image frame;

a data providing unit configured to provide at least one first depth indicative information and a contour of a first segment in the at least one key image frame;

a segment motion vector generation unit configured to generate a segment motion vector, a deformed key image frame, and a deformed contour of the first segment according to a color information of the key image frame, a color information of the first non-key image frame and a contour of a first segment in the key image frame;

a contour generation unit configured to generate a contour of a second segment in the first non-key image frame according to the segment motion vector, the contour of the first segment, a color information of the deformed key image frame, the contour of the deformed first segment, a color information of the key image frame, and a color information of the first non-key image frame, wherein the first and the second segments correspond to a same object of the key image frame; and

a depth motion unit configured to transfer the at least one first depth indicative information in the at least one key image frame to the first non-key image frame, so as to generate at least one second depth indicative information in the first non-key image frame; and

a storage unit coupled with the microprocessor, configured to store the data processed by the microprocessor.

2. The depth map generation module for a foreground object of claim 1, wherein the microprocessor further comprises a depth repair unit configured to compensate the second depth indicative information of the first non-key image frame according to the contour of the second segment, the color information of the key image frame, and the color information of the first non-key image frame, so as to generate at least one third depth indicative information of the second segment in the first non-key image frame.

3. The depth map generation module for a foreground object of claim 1, wherein when the depth map generation module for a foreground object is operated under a forward mode, the key image frame is a first image frame of the image sequence data, and the first non-key image frame is a second image frame of the image sequence data.

4. The depth map generation module for a foreground object of claim 1, wherein when the depth map generation module for a foreground object is operated under a reverse mode, the key image frame is a last image frame of the image sequence data, and the first non-key image frame is a second-to-last image frame of the image sequence data.

5. The depth map generation module for a foreground object of claim 1, wherein the contour generation unit comprises:

a windowing and first sampling unit configured to establish a plurality of first windows along the contour of the first segment in the key image frame, and to sample the contour and the color information in the first windows;

a window motion unit configured to transfer the first windows to the first non-key image frame according to the information of the segment motion vector of the first segment in the key image frame of the first windows, so as to establish a plurality of second windows;

a second sampling unit configured to form a set of foreground contour information according to the contour information sampled by the second windows on the contour of the deformed first segment, and to form a set of foreground and background color information according to the foreground and background color information sampled by the second windows on the contour of the deformed first segment and referencing the foreground and background color information sampled by the first windows; and

a profile generation unit configured to perform a weighted computation on the sampled contour and color information in the second windows, so as to generate the contour of the second segment.

6. The depth map generation module for a foreground object of claim 5, wherein when performing the weighted computation, the profile generation unit determines a proportion of a foreground contour in the second segment according to the foreground and background color information.

7. The depth map generation module for a foreground object of claim 5, wherein when performing the weighted computation, the profile generation unit determines a proportion of the foreground contour or the color in the second segment according to the information of the segment motion vector.

8. The depth map generation module for a foreground object of claim 1, wherein the segment motion vector generation unit comprises:

a search unit configured to search coordinates of common feature points of the contour of the first segment in the key image frame and the first non-key image frame;

an affine transformation unit configured to execute an affine transformation according to the difference of the coordinates of the common feature points, so as to generate the deformed key image frame, the contour of the deformed first segment, and a motion vector;

a first vector calculation unit configured to calculate the deformed key image frame and the non-key image frame, so as to obtain a relative motion vector regarding the deformed key image frame and the non-key image frame; and

a second vector calculation unit configured to receive the contour of the deformed first segment and the relative motion vector, and to add up the motion vector and the relative motion vector of each pixel in the contour of the deformed first segment, so as to generate the segment motion vector of each pixel in the contour of the deformed first segment;

wherein the segment motion vector defines the transfer between the first segment and the second segment.

9. The depth map generation module for a foreground object of claim 1, wherein the at least one depth indicative information of a third segment in a second non-key image frame close to the first non-key image frame is generated according to the contour of the second segment, the first non-key image frame, and the second non-key image frame, and the second and third segments correspond to a same object in the key image frame.

10. The depth map generation module for a foreground object of claim 1, wherein when the depth map generation module for a foreground object is operated under a bidirectional mode, the transformed image sequence data comprises a first key image frame and a second key image frame, wherein the first key image frame is the first image frame in the image sequence data, the second key image frame is the last image frame in the image sequence data, and the first non-key image frame is the second image frame in the image sequence data.

11. The depth map generation module for a foreground object of claim 10, further comprising a depth interpolation unit, wherein when the depth map generation module for a foreground object is operated under the bidirectional mode, the data providing unit provides the at least one depth indicative information of the first segment in the first key image frame and the at least one depth indicative information of a fourth segment in the second key image frame, the fourth and the first segment correspond to a first object in the key image frame, and the depth interpolation unit uses linear interpolation to generate at least one fourth depth indicative information according to the information calculated by the depth indicative information of the first segment and the information calculated by the depth indicative information of the fourth segment.

12. A method for generating a depth map for a foreground object in an image sequence data corresponding to a same scene, comprising:

receiving the image sequence data, wherein the image sequence data comprises a plurality of image frames, and each image frame comprises at least one object;

selecting at least one key image frame and a first non-key image frame from the image sequence data according to an operation mode of a depth map generation module for a foreground object, so as to generate a transformed image sequence data;

providing at least one first depth indicative information and a contour of a first segment in the at least one key image frame; and

executing the following steps via a microprocessor;

generating a segment motion vector, a deformed key image frame, and a contour of the deformed first segment according to a color information of the key image frame, a color information of the first non-key image frame, and a contour of the first segment in the key image frame;

generating a contour of a second segment in the first non-key image frame, wherein the first and second segments correspond to a same object in the key image frame, according to the segment motion vector, the contour of the first segment, the color information of the deformed key image frame, the contour of the deformed first segment, the color information of the key image frame, and the color information of the first non-key image frame; and

transferring at least one first depth indicative information of the at least one key image frame to the first non-key image frame according to the segment motion vector, so as to generate at least one second depth indicative information of the first non-key image frame.

13. The method of claim 12, wherein the microprocessor executing process further comprises a step of:

compensating the second depth indicative information in the first non-key image frame according to the contour of the second segment, the color information of the key image frame, and the color information of the first non-key image frame, so as to generate at least one third depth indicative information of the second segment in the first non-key image frame.

14. The method of claim 12, wherein when the depth map generation module for a foreground object is operated under a forward mode, the key image frame is the first image frame of the image sequence data, and the first non-key image frame is the second image frame of the image sequence data.

15. The method of claim 12, wherein when the depth map generation module for a foreground object is operated under a reverse mode, the key image frame is the last image frame of the image sequence data, and the first non-key image frame is a second-to-last image frame of the image sequence data.

16. The method of claim 12, wherein the step of generating a segment motion vector further comprises:

locating coordinates of the common feature points inside the contour of the first segment in the key image frame and the first non-key image frame;

executing an affine transformation according to differences between the coordinates of the common feature points inside the contour of the first segment in the key image frame and the first non-key image frame, so as to generate the deformed key image frame, the contour of the deformed first segment, and a motion vector;

calculating the deformed key image frame and the non-key image frame to obtain a relative motion vector regarding the deformed key image frame and the non-key image frame; and

receiving the relative motion vector and the contour of the deformed first segment, and adding up the motion vector and the relative motion vector of each pixel in the contour of the deformed first segment, so as to generate a segment motion vector of each pixel inside the contour of the deformed first segment;

wherein the segment motion vector defines the translation between the first segment and the second segment.

17. The method of claim 12, wherein the step of generating the contour of the second segment in the first non-key image frame comprises:

establishing a plurality of first windows along the contour of the first segment in the key image frame;

sampling the contour and the color information in the first windows;

transferring the first windows to the first non-key image frame according to the information of the segment motion vector of the first segment in the key image frame, so as to establish a plurality of second windows;

forming a set of foreground contour and color information and background color information according to the contour and color information sampled by the second windows on the contour of the deformed first segment, and referencing the foreground and background color information sampled by the first windows; and

performing a weighted computation on the contour and color information sampled by the second windows, so as to generate the contour of the second segment.

18. The method of claim 17, wherein the performing step further comprises:

determining a proportion of the foreground contour or color in the second segment according to the foreground and background color information.

19. The method of claim 17, wherein the performing step further comprises:

determining a proportion of the foreground contour or color in the second segment according to the information of the segment motion vector.

20. The method of claim 12, wherein the at least one depth indicative information of a third segment in a second non-key image frame close to the first non-key image frame is generated according to the contour of the second segment, the first non-key image frame, and the second non-key image frame, and the second and third segments correspond to a same object in the key image frame.

21. The method of claim 12, wherein when the depth map generation module for a foreground object is operated under a bidirectional mode, the transformed image sequence data further comprises another key image frame, the key image frame is the first image frame in the image sequence data, the another key image frame is the last image frame in the image sequence data, and the first non-key image frame is the second image frame in the image sequence data.

22. The method of claim 21, wherein the microprocessor executing process further comprises the steps of:

providing the at least one depth indicative information of the first segment in the key image frame;

providing the at least one depth indicative information of a fourth segment in the another key image frame, wherein the fourth segment and the first segment correspond to a same object in the key image frame; and

generating at least one fourth depth indicative information according to the information calculated by the depth indicative information of the first segment and the information calculated by the depth indicative information of the fourth segment by linear interpolation.

23. The method of claim 12, wherein the microprocessor executing process further comprises the steps of:

receiving the image sequence data, so as to generate the depth indicative information of a background object; and

integrating the at least one second depth indicative information and the depth indicative information of the background object, so as to generate the depth map of the image sequence data.

24. The method of claim 12, wherein the microprocessor executing process further comprises a step of:

receiving the image sequence data, so as to generate the depth indicative information of the image sequence data; and

integrating the at least one second depth indicative information and the depth indicative information of the image sequence data, so as to generate the depth map of the image sequence data.