METHODS, SYSTEMS, AND COMPUTER PROGRAM PRODUCTS FOR CREATING THREE-DIMENSIONAL VIDEO SEQUENCES

• US 20160065948A1
• Filed: 11/09/2015
• Published: 03/03/2016
• Est. Priority Date: 11/03/2010
• Status: Abandoned Application

• Current Assignee: 3DMedia Corporation
• Original Assignee: 3DMedia Corporation
• Inventors: McNamer, Michael, Markas, Tassos
• Application Number:
  US14/935,899
• Publication Number:
  US 20160065948A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof

Abstract

Methods and systems for creating three-dimensional video sequences of a scene are disclosed. An example method can include receiving multiple frames of a scene. The method may include selecting a target frame from among the multiple frames; selecting a first subset of frames, N, from among the multiple frames that are associated with the target frame that is representative of a large stereo baseline; and analyzing the first frame subset to identify two images for forming a stereoscopic pair of frames. Further, the method includes extracting depth data of static objects in the stereoscopic pair. The method includes selecting a second subset of frames that are associated with the target frame that is representative of a smaller stereo baseline than that represented by N; and utilizing the second frame subset to calculate depth of moving objects. The method includes generating a three-dimensional video frame based on the depth data.

Claims

1. A method for creating a three-dimensional video sequence of a scene, the method comprising:

using at least one processor and memory for;

receiving a two-dimensional video sequence of a scene, the two-dimensional video sequence including a plurality of frames;

selecting a target frame from among the plurality of frames;

selecting a first subset of frames, N, from among the plurality of frames that are associated with the target frame, representative of a targeted stereo displacement of the capture device from the target frame position;

analyzing the first subset of frames to identify two images for use in forming a stereoscopic pair of frames with a predetermined spatial difference;

identifying and classifying static and moving objects between the stereoscopic pair of frames;

extracting depth data of static objects in the stereoscopic pair of frames;

selecting a second subset of frames, n, from among the plurality of frames that are associated with the target frame, representative of a stereo displacement that is substantially smaller than the displacement represented by N;

utilizing the second subset of frames to calculate depth of moving objects;

combining the depth values of static and moving objects based on an absolute depth of static objects and the relative depth of moving objects; and

generating a three-dimensional video frame consisting of the target frame and a depth-generated view corresponding to the target frame based on the depth data.

2. The method of claim 1, further comprising one or more steps of identifying suitable frames, registration, stabilization, color correction, transformation, and depth adjustment.

3. The method of claim 1, further comprising generating one or more additional frames and frame viewpoints using one of existing raster data and depth information.

4. The method of claim 1, further comprising using a micro stereo base technique for generating image representations of close and moving objects of the scene.

5. The method of claim 1, further comprising using a macro stereo base technique for generating image representations of a background and non-moving objects of the scene.

6. The method of claim 1, further comprising using an image capture device for capturing the two-dimensional video sequence.

7. The method of claim 1, further comprising using the steps of claim 1 for generating a plurality of three-dimensional video frames.

8. The method of claim 1, further comprising dividing the captured two-dimensional video sequence into segments divided by utilizing one of a scene change detection and camera pose information.

9. The method of claim 8, further comprising creating a stereoscopic video sequence from each segment.

10. The method of claim 9, further comprising equalizing the depths and other three-dimensional parameters and combining the individual stereoscopic segments to form a single three-dimensional video stream.

11. The method of claim 1, further comprising analyzing captured frames to measure object displacements between two or more frames via motion vectors and identifying and classifying static and moving objects within the current target frame.

12. The method of claim 1, further comprising:

analyzing captured frames; and

identifying a position of key static objects to estimate the position of the camera.

13. The method of claim 1, further comprising measuring the motion vectors of the moving objects and estimating their relative position on the x, y space.

14. The method of claim 1, further comprising:

measuring sizes of moving objects; and

estimating their relative position on the z-space based on the rate of increase or decrease of their sizes.

15. The method of claim 1, further comprising estimating the depth of moving objects by identifying the depth of key points of moving objects relative to the depth of neighboring static objects with known depths and extrapolating based on temporal or structural relationships.

16. The method of claim 1, further comprising estimating the depth of moving objects by determining whether they are moving behind or in front of static objects and utilizing the depth of static objects to estimate the depth of a moving object based on the moving object'"'"'s trajectory in time.

17. The method of claim 1, further comprising:

segmenting the static portions of the scene into static objects with known positions in three-dimensional space and utilizing prediction techniques to estimate the depth of new information entering into the captured video sequence associated with the same or related static objects.

18. The method of claim 1, further comprising:

measuring sizes of static objects; and

estimating changes in camera focal length and adjusting the depth of the scene based on the relative zoom factor of the camera based on their relative rate of increase or decrease over time.

19. A system for creating a three-dimensional video sequence of a scene, the system comprising:

a memory having stored therein computer program code;

a computer processor that executes the computer program code;

a video generator configured to;

receive a two-dimensional video sequence of a scene, the two-dimensional video sequence including a plurality of frames;

select a target frame from among the plurality of frames;

select a first subset of frames, N, from among the plurality of frames that are associated with the target frame, representative of a targeted stereo displacement of the capture device from the target frame position;

analyze the first subset of frames to identify two images for use in forming a stereoscopic pair of frames with a predetermined spatial difference;

extract depth data of static objects in the stereoscopic pair of frames;

select a second subset of frames, n, from among the plurality of frames that are associated with the target frame, representative of a stereo displacement that is substantially smaller than the displacement represented by N;

utilize the second subset of frames to calculate depth of moving objects;

combine the depth values of static and moving objects based on the absolute depth of static objects and the relative depth of moving objects; and

generate a three-dimensional video frame consisting of the target frame and a depth-generated view corresponding to the target frame based on the depth data.

20. The system of claim 19, wherein the video generator is configured to implement one or more functions of identifying suitable frames, registration, stabilization, color correction, transformation, and depth adjustment.