METHOD AND SYSTEMS FOR MULTI-VIEW HIGH-SPEED MOTION CAPTURE

• US 20160140733A1
• Filed: 11/13/2014
• Published: 05/19/2016
• Est. Priority Date: 11/13/2014
• Status: Abandoned Application

• Current Assignee: Futurewei Technologies Incorporated (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Futurewei Technologies Incorporated (Huawei Investment & Holding Co., Ltd.)
• Inventors: GU, Jinwei, JIANG, Wei
• Application Number:
  US14/540,394
• Publication Number:
  US 20160140733A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/167 Synchronising or controlling image signals
• CPC Class Codes: H04N 13/172 image signals comprising non-image signal components, e.g. headers or format information
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0096 Synchronisation or controlling aspects
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/2627 for providing spin image effect, 3D stop motion effect or temporal freeze effect 2D image animation in general G06T13/80

Abstract

Systems and methods that facilitate an efficient and effective multi-view hybrid system for high-speed motion analysis are presented. The multi-view hybrid system can include a mix of relatively few high-speed cameras (e.g., 1000 fps, etc.) and a greater number of regular-speed cameras (e.g., 30 fps, etc.) that are utilized to provide more high speed views than high speed cameras included in the system. The new multi-view hybrid approach exploits spatial-temporal coherence in sampled images from both the regular-speed cameras and high-speed cameras to increase the number of high speed motion viewpoints. In one embodiment, the high-speed motion captured from one or a few high-speed cameras is “transferred” to other viewpoints to create a set of virtual high-speed camera viewpoints that provide the ability to obtain high-speed motion information (e.g., synthesized image frame sequences) from multiple viewpoints in addition to viewpoints of the physical high speed cameras in the system.

Claims

1. A device comprising:

a processing circuit that receives input associated with a three dimension (3D) scene, wherein the input includes information associated with a first frame sequence at a first frame rate corresponding to a first camera and information associated with a second frame sequence at a second frame rate corresponding to a second camera;

the first frame rate is faster than the second frame rate, wherein temporally synchronized image frames recorded in the first frame sequence and the second frame sequence at given timestamps are aligned, a three dimensional (3D) model for a dynamic scene at the first frame rate is created, and synthesized image frame sequences associated with a plurality of additional viewpoints are generated at the first frame rate; and

a memory that stores information for the processing circuit.

2. The device of claim 1, wherein the first frame sequence and the second frame sequence are calibrated.

3. The device of claim 2, wherein a calibration pre-process is conducted to determine intrinsic parameters and extrinsic parameters associated with the first camera and the second camera.

4. The device of claim 1, wherein the 3D model is based upon 3D key frame reconstruction, constrained 3D motion estimation, and 3D key frame interpolation.

5. The device of claim 4, wherein 3D key frame reconstruction includes a visual-hull based method to estimate a 3D mesh model of the target objects as the 3D model of a target scene.

6. The device of claim 4, wherein a kinematic-based 3D motion model is used to estimate a set of motion parameters from the 3D key frames.

7. The device of claim 4, wherein approximate optimal kinematic-based 3D motion parameters are used to establish an intermediate interpolated 3D frame sequence based on consecutive 3D key frame sequences and the interpolated 3D frame sequence gives a dynamic 3D model at the first rate for a dynamic scene from a virtual camera viewpoint at the first rate.

8. The device of claim 4, wherein the calibrated frame sequences captured from multiple viewpoints are used to enhance video quality, including denoising and correlation among multiple views, and spatial-temporal coherence within and among multiple video streams are used to increase signal-to-noise ratio.

9. A method comprising:

calibrating a first video frame sequence at a first frame rate from a first camera and a second video frame sequence at a second frame rate from a second camera, wherein the first frame rate is faster than the second frame rate;

creating a three dimension (3D) model based upon 3D key frame reconstruction, constrained 3D motion estimation, and 3D key frame interpolation; and

generating a plurality of synthesized image frame sequences at the first frame rate, wherein the plurality of synthesized image frame sequences are associated with a plurality of viewpoints different from a viewpoint associated with the first camera.

10. The method of claim 9, enabling an alternative in which the calibrating uses a plurality of intrinsic parameters and a plurality of extrinsic parameters and produces a calibrated first video frame sequence and a calibrated second video frame sequence.

11. The method of claim 10, enabling an alternative in which the calibrating includes at least one of the following:

correcting lens distortion for the first video frame sequence and the second video frame sequence based upon the plurality of intrinsic parameters; and

enabling geometric calibration by warping the first video frame sequence and the second video frame sequence to align with a reference sequence based upon the plurality of intrinsic parameters and the plurality of extrinsic parameters.

12. The method of claim 10, enabling an alternative in which the plurality of intrinsic parameters and the plurality of extrinsic parameters include at least one of the following:

an intrinsic camera parameter for the first camera and an intrinsic camera parameter for the second camera;

lens distortion coefficients for the first camera and the second camera; and

an extrinsic 6 degree-of-freedom position and pose for the first camera and the second camera.

13. The method of claim 9, enabling an alternative in which the 3D key frame reconstruction includes:

reconstructing a sequence of 3D key frames based upon a calibrated second video frame sequence;

ascertaining a kinematic motion parameter based upon the sequence of 3D key frames, an intrinsic parameter, and an extrinsic parameter; and

establishing a sequence of interpolated 3D frames for a virtual camera viewpoint at the first frame rate based upon the sequence of 3D key frames and the kinematic motion parameter.

14. The method of claim 9, enabling an alternative in which the generating a plurality of synthesized image frame sequences further includes:

creating a textured 3D model sequence by texture mapping a calibrated regular frame rate image sequence to a high frame rate dynamic 3D model; and

constructing a synthesized image frame sequence for a virtual camera viewpoint at the first rate based upon the textured 3D model sequence.

15. The method of claim 9, enabling an alternative in which the generating a plurality of synthesized image frame sequences further includes utilizing spatial-temporal coherence in multi-view video denoising.

16. A system comprising:

a first camera that captures images at a first speed;

a second camera that captures images at a second speed, the second speed is slower than the first speed;

a multi-view system for simulating a plurality of viewpoints at the first speed in addition to a viewpoint of the first camera, wherein image results of the first camera capture and the second camera capture are utilized to sample respective temporal appearance discrepancies, and spatial-temporal coherence in the sampled image results are utilized to synthesis video streams at the first speed from a viewpoint different than the viewpoint of the first camera.

17. The system of claim 16, wherein the first camera is one of a first plurality of cameras that capture images at the first rate and the second camera is one of a second plurality of cameras that capture images at the second rate, wherein the number of cameras included in the first plurality of cameras is less than the number of cameras included in the second plurality of cameras.

18. The system of claim 16, wherein the first frame rate is in or above the range of hundreds of frames per second the second rate is in the range of tens of frames per second.

19. The system of claim 16, wherein positions and poses of the first camera and second camera can change with respect to a scene but remain fixed relative to one another during a scene capture

20. The system of claim 16, wherein positions and poses of the first camera and second camera change with respect to one another after a first scene capture and before a second scene capture