Methods, systems, and computer program products for creating three-dimensional video sequences

• US 9,185,388 B2
• Filed: 11/03/2011
• Issued: 11/10/2015
• Est. Priority Date: 11/03/2010
• Status: Active Grant

• Current Assignee: 3DMedia Corporation
• Original Assignee: 3DMedia Corporation
• Inventors: Markas, Tassos, McNamer, Michael
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): ABIMBOLA, KEHINDE
• Application Number:
  US13/288,209
• Publication Number:
  US 20120105602A1
• Time in Patent Office: 1,468 Days
• Field of Search: 340/50
• US Class Current: 1/1
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof

Abstract

Methods and systems for creating three-dimensional video sequences of a scene are disclosed. An example method can include receiving multiple frames of a scene. The method may include selecting a target frame from among the multiple frames; selecting a first subset of frames, N, from among the multiple frames that are associated with the target frame that is representative of a large stereo baseline; and analyzing the first frame subset to identify two images for forming a stereoscopic pair of frames. Further, the method includes extracting depth data of static objects in the stereoscopic pair. The method includes selecting a second subset of frames that are associated with the target frame that is representative of a smaller stereo baseline than that represented by N; and utilizing the second frame subset to calculate depth of moving objects. The method includes generating a three-dimensional video frame based on the depth data.

Claims

1. A method for creating a three-dimensional video sequence of a scene, the method comprising:

utilizing a two-dimensional video sequence of a scene, the two-dimensional video sequence including a plurality of frames;

selecting a target frame from among the plurality of frames;

selecting a first subset of frames, N, from among the plurality of frames that are associated with the target frame;

identifying and classifying static and moving objects utilizing the first subset of frames;

extracting depth data of static objects utilizing the first subset of frames;

selecting a second subset of frames, n, from among the plurality of frames that are associated with the target frame, representative of a stereo displacement that is substantially different than the displacement represented by N;

utilizing the second subset of frames to calculate depth of moving objects at the time of the target frame; and

combining the depth values of static and moving objects to generate a three-dimensional video frame.

2. The method of claim 1, further comprising one or more steps of identifying suitable frames, registration, stabilization, color correction, transformation, and depth adjustment.

3. The method of claim 1, further comprising generating one or more additional frames and frame viewpoints using one of existing raster data and depth information.

4. The method of claim 1, further comprising using a micro stereo base technique for generating image representations of close and moving objects of the scene.

5. The method of claim 1, further comprising using a macro stereo base technique for generating image representations of a background and non-moving objects of the scene.

6. The method of claim 1, further comprising using an image capture device for capturing the two-dimensional video sequence.

7. The method of claim 1, further comprising using the steps of claim 1 for generating a plurality of three-dimensional video frames.

8. The method of claim 1, further comprising dividing the captured two-dimensional video sequence into segments divided by utilizing one of a scene change detection and camera pose information.

9. The method of claim 8, further comprising creating a stereoscopic video sequence from each segment.

10. The method of claim 9, further comprising equalizing the depths and other three-dimensional parameters and combining the individual stereoscopic segments to form a single three-dimensional video stream.

11. The method of claim 1, further comprising analyzing captured frames to measure object displacements between two or more frames via motion vectors and identifying and classifying static and moving objects within the current target frame.

12. The method of claim 1, further comprising:

analyzing captured frames; and

identifying a position of key static objects to estimate the position of the camera.

13. The method of claim 1, further comprising measuring the motion vectors of the moving objects and estimating their relative position on the x, y space.

14. The method of claim 1, further comprising:

measuring sizes of moving objects; and

estimating their relative position on the z-space based on the rate of increase or decrease of their sizes.

15. The method of claim 1, further comprising estimating the depth of moving objects by identifying the depth of key points of moving objects relative to the depth of neighboring static objects with known depths and extrapolating based on temporal or structural relationships.

16. The method of claim 1, further comprising estimating the depth of moving objects by determining whether they are moving behind or in front of static objects and utilizing the depth of static objects to estimate the depth of a moving object based on the moving object'"'"'s trajectory in time.

17. The method of claim 1, further comprising:

segmenting the static portions of the scene into static objects with known positions in three-dimensional space and utilizing prediction techniques to estimate the depth of new information entering into the captured video sequence associated with the same or related static objects.

18. The method of claim 1, further comprising:

measuring sizes of static objects; and

estimating changes in camera focal length and adjusting the depth of the scene based on the relative zoom factor of the camera based on their relative rate of increase or decrease over time.

19. A system for creating a three-dimensional video sequence of a scene, the system comprising:

a memory having stored therein computer program code;

a computer processor that executes the computer program code;

a video generator configured to;

receive a two-dimensional video sequence of a scene, the two-dimensional video sequence including a plurality of frames;

select a target frame from among the plurality of frames;

select a first subset of frames, N, from among the plurality of frames that are associated with the target frame;

extract depth data of static objects utilizing the first subset of frames;

select a target frame from among the plurality of frames;

select a second subset of frames, n, from among the plurality of frames that are associated with the target frame, representative of a stereo displacement that is substantially different than the displacement represented by N;

utilize the second subset of frames to calculate depth of moving objects at the time of the target frame; and

combine the depth values of static and moving objects to generate a three-dimensional video frame.

20. The system of claim 19, wherein the video generator is configured to implement one or more functions of identifying suitable frames, registration, stabilization, color correction, transformation, and depth adjustment.

21. The system of claim 19, wherein the video generator is configured to generate one or more additional frames and frame viewpoints using one of existing raster data and depth information.

22. The system of claim 19, wherein the video generator is configured to use a micro stereo base technique for generating image representations of close and moving objects of the scene.

23. The system of claim 19, wherein the video generator is configured to use a macro stereo base technique for generating image representations of a background and non-moving objects of the scene.

24. The system of claim 19, comprising an image sensor and a lens configured to capture the two-dimensional video sequence.

25. The system of claim 19, wherein the video generator is configured to generate a plurality of three-dimensional video frames using the functions implemented by the video generator recited in claim 9.

26. The system of claim 25, comprising a display configured to display the plurality of three-dimensional video frames in a sequence.

27. A computer program product for creating three-dimensional video sequence, the computer program product comprising:

a non-transitory computer readable storage medium having computer readable program code embodied therewith, the non-transitory computer readable program code comprising;

computer readable code configured to receive a two-dimensional video sequence of a scene, the two-dimensional video sequence including a plurality of frames;

computer readable code configured to select a target frame from among the plurality of frames;

computer readable code configured to select a first subset of frames, N, from among the plurality of frames that are associated with the target frame;

computer readable code configured to identify and classify static and moving objects using the first subset of frames;

computer readable code configured to extract depth data of static objects utilizing the first subset of frames;

computer readable code configured to select a second subset of frames, n, from among the plurality of frames that are associated with the target frame, representative of a stereo displacement that is substantially different than the displacement represented by N;

computer readable code configured to utilize the second subset of frames to calculate depth of moving objects at the time of the target frame; and

computer readable code configured to combine the static and moving objects to generate a three-dimensional video frame.

28. The computer program product of claim 27, wherein the computer readable program code comprises computer readable code to implement one or more steps of identifying suitable frames, registration, stabilization, color correction, transformation, and depth adjustment.

29. The computer program product of claim 27, wherein the computer readable program code comprises computer readable code to generate one or more additional frames and frame viewpoints using one of existing raster data and depth information.

30. The computer program product of claim 27, wherein the computer readable program code comprises computer readable code to use a micro stereo base technique for generating image representations of close and moving objects of the scene.

31. The computer program product of claim 27, wherein the computer readable program code comprises computer readable code to use a macro stereo base technique for generating image representations of a background and non-moving objects of the scene.

32. The computer program product of claim 27, wherein the computer readable program code comprises computer readable code to use an image capture device for capturing the two-dimensional video sequence.

33. The computer program product of claim 27, wherein the computer readable program code comprises computer readable code to generate a plurality of three-dimensional video frames.

34. The computer program product of claim 33, wherein the computer readable program code comprises computer readable code to control a display to display the plurality of three-dimensional video frames in a sequence.