System and method for removing camera rotation from a panoramic video

• US 9,277,122 B1
• Filed: 08/13/2015
• Issued: 03/01/2016
• Est. Priority Date: 08/13/2015
• Status: Expired due to Fees

• Current Assignee: Legend3D, Inc.
• Original Assignee: Legend3D, Inc.
• Inventors: Imura, Ken, Cummins, Ryan
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Aynalem, Nathnael
• Application Number:
  US14/825,746
• Time in Patent Office: 201 Days
• Field of Search: 348/36, 348/37
• US Class Current: 1/1
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Removes undesirable camera rotation from a panoramic video. Video frames are mapped to a sphere and then projected to a forward-looking planar video. Camera tracking or other techniques are used to derive camera position and orientation in each frame. From the camera orientation, a correcting rotation is derived and is applied to the sphere onto which the panoramic video was mapped. Re-projecting video from the rotation-corrected sphere reduces or eliminates camera rotation artifacts from the panoramic video. Rotation may be re-introduced and other visual effects may be added to the stabilized video. May be utilized in 2D virtual reality and augmented reality displays. Depth may be accepted by the system for objects in the video and utilized to create 3D stereoscopic virtual reality and augmented reality displays.

Claims

1. A method to remove camera rotation from a panoramic video comprising

obtaining a panoramic video of a scene in a three-dimensional space, wherein each panoramic frame of said panoramic video is mapped onto a sphere to form a spherically mapped panoramic frame;

accepting reference axes that define said three-dimensional space;

projecting each of said spherically mapped panoramic frames onto a front plane to create a forward-looking planar video;

calculating a virtual camera rotation and a virtual camera translation, relative to said reference axes, for each planar frame of said forward-looking planar video;

wherein said calculating said virtual camera rotation and said virtual camera translation, relative to said reference axes, for each planar frame of said forward-looking planar video comprises

identifying and locating a plurality of features in one or more key frames of said forward-looking planar video, wherein each of said plurality of features has a fixed position in said three-dimensional space;

tracking the pixel location of each of said plurality of features across the planar frames of said forward-looking planar video to create a feature track for each of said plurality of features;

calculating a 3D geometry model from said feature tracks that comprises

the position of each of said plurality of features in said three-dimensional space;

the virtual camera rotation and the virtual camera translation, relative to said reference axes, for each planar frame of said forward-looking planar video;

wherein said 3D geometry model minimizes the error between

the projection of said position of each of said plurality of features onto a view plane defined by said virtual camera rotation and said virtual camera translation for each planar frame of said forward-looking planar video, and

said pixel location of each of said plurality of features in each of said feature tracks;

obtaining said virtual camera rotation and said virtual camera translation from said 3D geometry model;

applying said virtual camera rotation to said sphere in each of said spherically mapped panoramic frames to generate a rotation-corrected panoramic frame for each of said spherically mapped panoramic frames;

combining said rotation-corrected panoramic frames to form a rotation-corrected panoramic video.

2. The method of claim 1, wherein said obtaining said panoramic video comprises

obtaining a plurality of videos of said scene, each captured by a corresponding camera from a plurality of cameras, wherein all of said plurality of videos are captured over a common time interval;

selecting a sequence of frame sample times within said common time interval;

for each of said frame sample times and

for each of said plurality of cameras

mapping the camera frame that was captured from said camera approximately at said frame sample times onto said sphere, to form a spherically mapped camera frame;

combining said spherically mapped camera frames to form said spherically mapped panoramic frame for said frame sample times.

3. The method of claim 2, wherein

said obtaining a plurality of videos of said scene comprises

attaching said plurality of cameras to a camera mount;

placing said camera mount in said scene;

capturing said plurality of videos from said plurality of cameras over said common time interval.

4. The method of claim 3, wherein said camera mount is coupled to a drone.

5. The method of claim 3, further comprising

accepting a definition of a reference sphere surrounding said camera mount;

calculating a spherical region for each of said plurality of cameras as the intersection of the field of view of each of said plurality of cameras with the surface of said reference sphere;

wherein said mapping the camera frame that was captured from said camera approximately at said frame sample times onto said sphere comprises

projecting said camera frame onto said spherical region for the corresponding camera.

6. The method of claim 1, further comprising

generating a virtual reality environment from said 3D geometry model;

obtaining a position and orientation of a user in said virtual reality environment;

generating a view image for said user in said virtual reality environment by

calculating an image plane for said user in said virtual environment based on said position and orientation of said user;

projecting each of said plurality of features from said position of each of said plurality of features onto said image plane.

7. The method of claim 1, further comprising

accepting input depths to assign a plurality of features in said forward-looking planar video that comprises the position of each of said plurality of features in said three-dimensional space;

generating a 2D presentation or 3D stereoscopic presentation to display in a virtual reality or augmented reality viewing device.

16. The method of claim 2, further comprising:

blending said spherically mapped camera frames.

17. The method of claim 3, further comprising:

translating and rotating said camera mount in said scene over said common time interval.

8. A method for removing camera rotation from a panoramic video comprising

attaching a plurality of cameras to a camera mount;

placing said camera mount in a scene in a three-dimensional space;

capturing a plurality of videos from said plurality of cameras over a common time interval;

translating and rotating said camera mount in said scene over said common time interval;

defining reference axes for said three-dimensional space;

defining a reference sphere surrounding said camera mount;

calculating a spherical region for each of said plurality of cameras as the intersection of the field of view of each of said plurality of cameras with the surface of said reference sphere;

selecting a sequence of frame sample times within said common time interval;

for each of said frame sample times and

for each of said plurality of cameras

projecting the camera frame that was captured from said camera approximately at said frame sample times onto said spherical region for the corresponding camera, to form a spherically mapped camera frame;

blending said spherically mapped camera frames;

combining said spherically mapped camera frames to form a spherically mapped panoramic frame for said frame sample times;

projecting each of said spherically mapped panoramic frames onto a front plane to create a forward-looking planar video;

obtaining depths by either or both of

identifying and locating a plurality of features in one or more key frames of said forward-looking planar video, wherein each of said plurality of features has a fixed position in said three-dimensional space;

tracking the pixel location of each of said plurality of features across the planar frames of said forward-looking planar video to create a feature track for each of said plurality of features;

calculating a 3D geometry model from said feature tracks that comprises the position of each of said plurality of features in said three-dimensional space;

the virtual camera rotation and the virtual camera translation, relative to said reference axes, for each planar frame of said forward-looking planar video;

wherein said 3D geometry model minimizes the error between the projection of said position of each of said plurality of features onto a view plane defined by said virtual camera rotation and said virtual camera translation for each planar frame of said forward-looking planar video; and

said pixel location of each of said plurality of features in each of said feature tracks;

obtaining said virtual camera rotation and said virtual camera translation from said 3D geometry model;

or

accepting input depths to assign a plurality of features in said forward-looking planar video that comprises the position of each of said plurality of features in said three-dimensional space;

generating a 2D presentation or 3D stereoscopic presentation to display in a virtual reality or augmented reality viewing device;

applying said virtual camera rotation to said sphere in each of said spherically mapped panoramic frames to generate a rotation-corrected panoramic frame for each of said spherically mapped panoramic frames;

combining said rotation-corrected panoramic frames to form a rotation-corrected panoramic video;

generating a virtual reality environment from said 3D geometry model;

obtaining a position and orientation of a user in said virtual reality environment;

generating a view image for said user in said virtual reality environment by calculating an image plane for said user in said virtual environment based on said position and orientation of said user;

projecting each of said plurality of features from said position of each of said plurality of features onto said image plane.

9. A system to remove camera rotation from a panoramic video comprising

obtain a panoramic video of a scene in a three-dimensional space, wherein each panoramic frame of said panoramic video is mapped onto a sphere to form a spherically mapped panoramic frame;

accept reference axes that define said three-dimensional space;

project each of said spherically mapped panoramic frames onto a front plane to create a forward-looking planar video;

calculate a virtual camera rotation and a virtual camera translation, relative to said reference axes, for each planar frame of said forward-looking planar video;

wherein said calculate said virtual camera rotation and said virtual camera translation, relative to said reference axes, for each planar frame of said forward-looking planar video comprises

identify and locating a plurality of features in one or more key frames of said forward-looking planar video, wherein each of said plurality of features has a fixed position in said three-dimensional space;

track the pixel location of each of said plurality of features across the planar frames of said forward-looking planar video to create a feature track for each of said plurality of features;

calculate a 3D geometry model from said feature tracks that comprises

the position of each of said plurality of features in said three-dimensional space;

the virtual camera rotation and the virtual camera translation, relative to said reference axes, for each planar frame of said forward-looking planar video;

wherein said 3D geometry model minimizes the error between

the projection of said position of each of said plurality of features onto a view plane defined by said virtual camera rotation and said virtual camera translation for each planar frame of said forward-looking planar video, and

said pixel location of each of said plurality of features in each of said feature tracks;

obtain said virtual camera rotation and said virtual camera translation from said 3D geometry model;

apply said virtual camera rotation to said sphere in each of said spherically mapped panoramic frames to generate a rotation-corrected panoramic frame for each of said spherically mapped panoramic frames;

combine said rotation-corrected panoramic frames to form a rotation-corrected panoramic video.

10. The system of claim 9, wherein said obtain said panoramic video comprises

obtain a plurality of videos of said scene, each captured by a corresponding camera from a plurality of cameras, wherein all of said plurality of videos are captured over a common time interval;

select a sequence of frame sample times within said common time interval;

for each of said frame sample times and

for each of said plurality of cameras

map the camera frame that was captured from said camera approximately at said frame sample times onto said sphere, to form a spherically mapped camera frame;

combine said spherically mapped camera frames to form said spherically mapped panoramic frame for said frame sample times.

11. The system of claim 10, wherein

said obtain said plurality of videos of said scene comprises

attach said plurality of cameras to a camera mount;

place said camera mount in said scene;

capture said plurality of videos from said plurality of cameras over said common time interval.

12. The system of claim 11, wherein said camera mount is coupled to a drone.

13. The system of claim 11, further comprising

accept a definition of a reference sphere surrounding said camera mount;

calculate a spherical region for each of said plurality of cameras as the intersection of the field of view of each of said plurality of cameras with the surface of said reference sphere;

wherein said mapping the camera frame that was captured from said camera approximately at said frame sample times onto said sphere comprises

project said camera frame onto said spherical region for the corresponding camera.

14. The system of claim 9, further comprising

generate a virtual reality environment from said 3D geometry model;

obtain a position and orientation of a user in said virtual reality environment;

generate a view image for said user in said virtual reality environment through

calculate an image plane for said user in said virtual environment based on said position and orientation of said user;

project each of said plurality of features from said position of each of said plurality of features onto said image plane.

15. The system of claim 9, further comprising

accept input depths to assign a plurality of features in said forward-looking planar video that comprises the position of each of said plurality of features in said three-dimensional space;

generate a 2D presentation or 3D stereoscopic presentation to display in a virtual reality or augmented reality viewing device.

18. The system of claim 10, further comprising:

blend said spherically mapped camera frames.

19. The system of claim 11, further comprising:

translate and rotate said camera mount in said scene over said common time interval.