Three-dimensional video creating device and three-dimensional video creating method

• US 8,908,011 B2
• Filed: 09/16/2011
• Issued: 12/09/2014
• Est. Priority Date: 09/17/2010
• Status: Active Grant

• Current Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Yamada, Hitoshi, Li, Bo, Ong, Chin Phek, Shen, Sheng Mei
• Primary Examiner(s): Philippe, Gims
• Assistant Examiner(s): Fereja, Samuel D
• Application Number:
  US13/509,809
• Publication Number:
  US 20120229609A1
• Time in Patent Office: 1,180 Days
• Field of Search: None
• US Class Current: 348/46
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: H04N 13/264 using the relative movement of objects in two video frames or fields

Abstract

A three-dimensional video creating device (100) includes: a selection unit (123) which selects, from among frames constituting the 2D video, frames each of which has a common area whose proportion to the frame is greater than or equal to a predetermined value, as candidate three-dimensional partner frames that are candidate frames each constituting a three-dimensional image together with a target frame included in the frames constituting the 2D video; a determination unit (124) which determines, from among the candidate three-dimensional partner frames, a three-dimensional partner frame, based on the first criteria; and a three-dimensional pair creation unit (125) which creates a three-dimensional pair constituting the three-dimensional image corresponding to the target frame, using the target frame and the three-dimensional partner frame.

Claims

1. A three-dimensional video creating device which creates a three-dimensional (3D) video from a two-dimensional (2D) video, said three-dimensional video creating device comprising:

a receiving unit configured to receive a 2D video;

a selection unit configured to select, from among frames constituting the 2D video, frames each of which has a common area whose proportion to the frame is greater than or equal to a predetermined value, as candidate three-dimensional partner frames that are candidate frames each constituting a three-dimensional image together with a target frame included in the frames constituting the 2D video, each of the common areas being an area of an image common to the target frame;

a determination unit configured to determine, from among the candidate three-dimensional partner frames, a three-dimensional partner frame that is a frame which, together with the target frame, constitutes a three-dimensional image, based on predetermined criteria;

a three-dimensional pair creation unit configured to create a three-dimensional pair constituting the three-dimensional image corresponding to the target frame, using the target frame and the three-dimensional partner frame; and

a conversion unit configured to generate, from the target frame and the candidate three-dimensional partner frames selected by said selection unit, a transformation matrix for warping the candidate three-dimensional partner frames, and convert the three-dimensional pair by applying the generated transformation matrix to the three-dimensional pair.

2. The three-dimensional video creating device according to claim 1, further comprising

a stabilization unit configured to stabilize the frames constituting the 2D video by correcting a fluctuation between one of the frames and another one of the frames,

wherein said selection unit is configured to select, from among the frames constituting the 2D video and stabilized by said stabilization unit, candidate three-dimensional partner frames.

3. The three-dimensional video creating device according to claim 1, further comprising:

a storage unit in which the three-dimensional pair converted by said conversion unit is stored; and

an output unit configured to output the converted three-dimensional pair stored in said storage unit,

wherein said output unit is configured to output the three-dimensional pair which corresponds to each of the frames constituting the 2D video and is converted by said conversion unit, to create a 3D video from the 2D video.

4. The three-dimensional video creating device according to claim 1,

wherein said selection unit is configured to select, from among the frames constituting the 2D video, frames captured for the same scene as the target frame, as the candidate three-dimensional partner frames.

5. The three-dimensional video creating device according to claim 4,

wherein, when each of a proportion of the common area to the target frame and a proportion of each of the common areas to a corresponding one of the candidate three-dimensional partner frames is greater than or equal to the predetermined value, said selection unit is configured to determine that the corresponding candidate three-dimensional partner frame is a frame captured for the same scene as the target frame, and

each of the common areas is calculated based on corresponding feature points among feature points on the target frame and feature points on each of the candidate three-dimensional partner frames.

6. The three-dimensional video creating device according to claim 4,

wherein said selection unit is configured to select, from among the frames constituting the 2D video, adjacent frames temporally close to the target frame, as the candidate three-dimensional partner frames.

7. The three-dimensional video creating device according to claim 1,

wherein the predetermined criteria include a vertical difference criterion, a horizontal parallax criterion, a smoothing criterion, and a distortion reduction criterion,

the vertical difference criterion specifies that the three-dimensional partner frame is a frame whose difference in vertical position between the common area in the target frame and the common area in the three-dimensional partner frame is substantially zero,

the horizontal parallax criterion specifies that the three-dimensional partner frame is a frame whose difference in horizontal position between the common area in the target frame and the common area in the three-dimensional partner frame is within a predetermined range,

the smoothing criterion specifies that the three-dimensional partner frame is a frame for which displacement of a common area between temporally successive three-dimensional pairs is smoothed, and

the distortion reduction criterion specifies that the three-dimensional partner frame is a frame which has a common area whose proportion to the respective frames is greater than or equal to the predetermined value in the same manner as the target frame.

8. The three-dimensional video creating device according to claim 7,

wherein said determination unit is configured to determine the three-dimensional partner frame by judging whether or not each of the candidate three-dimensional partner frames satisfies the predetermined criteria using feature points on a frame which become detectable by performing image processing.

9. The three-dimensional video creating device according to claim 1,

wherein said conversion unit is configured to generate the transformation matrix based on corresponding feature points among feature points on the target frame and feature points on the three-dimensional partner frame.

10. The three-dimensional video creating device according to claim 1,

wherein said conversion unit is configured to:

calculate a fundamental matrix using the target frame and the candidate three-dimensional partner frames;

calculate the skew transformation matrix based on the fundamental matrix so that a vertical difference between the target frame and each of the candidate three-dimensional partner frames in the three-dimensional pair is minimized;

calculate the projective transformation matrix so that the common area in the three-dimensional partner frame converted using the projective transformation matrix maintains the same orthogonality and aspect ratio as the common area in the three-dimensional partner frame before the conversion; and

calculate the translation transformation matrix so that a parallax between the target frame and the three-dimensional partner frame to which the projective transformation matrix and the skew transformation matrix are applied becomes the same as a parallax between a target frame prior to the target frame and a three-dimensional partner frame; and

generate the transformation matrix by combining the projective transformation matrix, the skew transformation matrix, and the translation transformation matrix that are calculated, and apply the transformation matrix to the three-dimensional pair.

11. The three-dimensional video creating device according to claim 2,

wherein said stabilization unit is configured to correct the fluctuation between the one frame and the other frame, by using a projective transformation matrix calculated based on corresponding feature points among feature points on a predetermined frame and feature points on each of adjacent frames temporally close to the predetermined frame.

12. The three-dimensional video creating device according to claim 2,

wherein said stabilization unit includes:

a detection unit configured to detect corresponding feature points among feature points on a predetermined frame and feature points on each of adjacent frames adjacent to the predetermined frame; and

a calculation unit configured to calculate a projective transformation matrix for warping the predetermined frame so that the feature points on the predetermined frame and the weighted feature points on a corresponding one of the adjacent frames have the same coordinate value, and

said stabilization unit is configured to stabilize the frames constituting the 2D video, by applying the projective transformation matrix to the predetermined frame.

13. The three-dimensional video creating device according to claim 12,

wherein the adjacent frames are temporally close to the predetermined frame.

14. The three-dimensional video creating device according to claim 12,

wherein said calculation unit is configured to:

calculate a weight for each of the adjacent frames using a weight function; and

when the corresponding adjacent frame is temporally closest to the predetermined frame, calculate a weighted value closer to 1 using the weight function, and when the corresponding adjacent frame is temporally far from the predetermined frame, calculate a weighted value smaller than 1 using the weight function.

15. The three-dimensional video creating device according to claim 2,

wherein said receiving unit is configured to receive a 2D video captured by a user operating an image capturing apparatus, and

said three-dimensional video creating device further comprising:

an analysis unit configured to analyze motion of the image capturing apparatus to generate information for guiding the user to capture a stable 2D video;

a feedback unit configured to generate feedback information for giving a feedback to the user, based on the information generated by said analysis unit; and

a display unit configured to display the generated feedback information and the 2D video.

16. A three-dimensional video creating method for creating a three-dimensional (3D) video from a two-dimensional (2D) video, said three-dimensional video creating method comprising:

receiving a 2D video;

selecting, from among frames constituting the 2D video, frames each of which has a common area whose proportion to the frame is greater than or equal to a predetermined value, as candidate three-dimensional partner frames that are candidate frames each constituting a three-dimensional image together with a target frame included in the frames constituting the 2D video, each of the common areas being an area of an image common to the target frame;

determining, from among the candidate three-dimensional partner frames, a three-dimensional partner frame that is a frame which, together with the target frame, constitutes a three-dimensional image, based on predetermined criteria;

creating a three-dimensional pair constituting the three-dimensional image corresponding to the target frame, using the target frame and the three-dimensional partner frame; and

generating, from the target frame and the candidate three-dimensional partner frames selected in said selecting, a transformation matrix for warping the candidate three-dimensional partner frames, and converting the three-dimensional pair by applying the generated transformation matrix to the three-dimensional pair.