Stitching frames into a panoramic frame

• US 10,460,459 B2
• Filed: 12/12/2016
• Issued: 10/29/2019
• Est. Priority Date: 01/03/2016
• Status: Expired due to Fees

• Current Assignee: HumanEyes Technologies Limited
• Original Assignee: HumanEyes Technologies Limited
• Inventors: Bar, Anton, Tulbovich, Yitzchak, Fine, Shmuel
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): Nirjhar, Nasim N
• Application Number:
  US15/560,495
• Publication Number:
  US 20180063513A1
• Time in Patent Office: 1,051 Days
• CPC Class Codes: G03B 35/08 by simultaneous recording
• CPC Class Codes: G03B 35/10 having single camera with stereoscopic-base-defining system
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: G06T 7/38 Registration of image sequences
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

A method of stitching frames captured at multiple viewing angles comprising: receiving a set of frames captured by imagers, identifying in each of pairs of neighboring frames, each neighboring frame is captured about the same time as the other from an imager having an overlapping field of view with another. For each overlapping area a center connecting axis connecting between centers of respective the pair of neighboring frames is identified, a motion gradient for each frame of the pair of neighboring frames is calculated, pixels of the pair of neighboring frames in the at least one overlapping area are adapted based on corresponding values from the motion gradient along the center connecting axis, the pair of adapted neighboring frames into a panoramic frame which at least partly imaging an environment surrounding the common center is stitched.

Claims

1. A method of stitching a plurality of frames captured at multiple viewing angles by a plurality of imagers mounted around a common center into a panoramic frame imaging, at least partly, an environment surrounding said common center, comprising:

receiving a set of a plurality of frames, each of the plurality of frames is captured a different viewing angle from other frames of the plurality of frames by one of a plurality of imagers mounted around a common center region to image an environment surrounding said common center region;

providing a plurality of calibration models each is calculated based on one or more intrinsic parameters of one of said plurality of imagers;

identifying in each of a plurality of pairs of neighboring frames from said plurality of frames at least one overlapping area using at least some of the plurality of calibration models, each said pair of neighboring frames comprises frames captured about the same time by two imagers from said plurality of imagers, said two imagers having overlapping fields of view;

for each said at least one overlapping area;

identifying a center connecting axis connecting between centers of respective said pair of neighboring frames;

calculating a motion gradient for each frame of said pair of neighboring frames;

adapting pixels of said pair of neighboring frames in said at least one overlapping area based on corresponding values from said motion gradient along said center connecting axis; and

stitching said pair of adapted neighboring frames into a panoramic frame which at least partly imaging an environment surrounding said common center;

wherein each of the one or more intrinsic parameters is selected from a group consisting of a principal point parameter, a focal length parameter, and a fisheye distortion parameter

wherein at least one of said plurality of calibration models comprises a fish-eye correction matrix and an extrinsic camera calibration matrix which map coordinates of pixels to corrected coordinates defined according to symmetry axes of an image plane of respective said imager;

wherein, in each of said plurality of frames, said at least one overlapping area is calculated based on a combined matrix generated according to said fish-eye correction matrix and said extrinsic camera calibration matrix are combined into a combined matrix used for calculating said at least one overlapping area overlap between each two consequent frames by projecting them onto a sphere so that each pixel becomes a 3D vector and correcting the 3D vectors according to the combined, full calibration matrix.

2. The method of claim 1, wherein said panoramic frame is a left eye panoramic frame and said plurality of imagers are intertwined with a plurality of additional imagers capturing a plurality of additional frames which are combined into a right eye panoramic frame;

further comprising combining said right eye panoramic frame with said left eye panoramic frame for creating a stereoscopic frame.

3. The method of claim 2, wherein members of said plurality of imagers and said plurality of additional frames are alternately arranged along a virtual circle encircling said common center.

4. The method of claim 1, wherein said panoramic frame is a right eye panoramic frame and said plurality of imagers are intertwined with a plurality of additional imagers capturing a plurality of additional frames which are combined into a left eye panoramic frame;

further comprising combining said left eye panoramic frame with said right eye panoramic frame for creating a stereoscopic frame.

5. The method of claim 1, wherein said plurality of imagers are arranged along a virtual circle encircling said common center;

wherein each one of said plurality of imagers is mounted such that an optical axis thereof is tilted in relation to an axis passing through said common center and a tangential point of an origin of a field of view of a respective said imager.

6. The method of claim 1, wherein said panoramic frame is a 360 degrees spherical panorama frame imaging said environment around said common center.

7. The method of claim 6, wherein said panoramic frame is used as one of two of frames comprising a stereoscopic frame.

8. The method of claim 1, wherein said method is repeated for adapting respective said at least one overlapping area in each frame of each of a plurality of sequentially captured sets.

9. The method of claim 1, further comprising estimating a projection of a visual representation of said plurality of frames onto a sphere skeleton.

10. The method of claim 1, wherein each one of said plurality of imagers is a camera comprising a fish eye lens.

11. The method of claim 1, wherein each one of said plurality of imagers is calibrated using a Camera-Rig Calibration.

12. The method of claim 1, wherein said adapting comprises shifting pixels located in said overlapping area towards a lateral edge of a respective said frame along said respective said axis.

13. The method of claim 12, further comprising correcting a plurality of blank pixels formed as an outcome of said shifting by interpolation.

14. A system of stitching a plurality of frames captured at multiple viewing angles by a plurality of imagers mounted around a common center into a panoramic frame imaging, at least partly, an environment surrounding said common center, comprising:

an interface adapted to receive a set of a plurality of frames, each of the plurality of frames is captured about the same time as other frames of the plurality of frames at a different viewing angle from the other frames of the plurality of frames by a plurality of imagers mounted around a common center region to image an environment surrounding said common center region;

a code store adapted to store a code;

a processor adapted to execute said code for;

providing access to a plurality of calibration models each is calculated based on one or more intrinsic parameters of one of said plurality of imagers;

identifying in each of a plurality of pairs of neighboring frames from said plurality of frames at least one overlapping area using at least some of the plurality of calibration models, each said pair of neighboring frames comprises frames captured about the same time by two imagers from said plurality of imagers, said two imagers having overlapping fields of view;

for each said at least one overlapping area;

identifying a center connecting axis connecting between centers of respective said pair of neighboring frames;

calculating a motion gradient for each frame of said pair of neighboring frames;

adapting pixels of said pair of neighboring frames in said at least one overlapping area based on corresponding values from said motion gradient along said center connecting axis; and

stitching said pair of adapted neighboring frames into a panoramic frame which at least partly imaging an environment surrounding said common center;

wherein each of the one or more intrinsic parameters is selected from a group consisting of a principal point parameter, a focal length parameter, and a fisheye distortion parameter;

wherein at least one of said plurality of calibration models comprises a fish-eye correction matrix and an extrinsic camera calibration matrix which map coordinates of pixels to corrected coordinates defined according to symmetry axes of an image plane of respective said imager;

wherein, in each of said plurality of frames, said at least one overlapping area is calculated based on a combined matrix generated according to said fish-eye correction matrix and said extrinsic camera calibration matrix are combined into a combined matrix used for calculating said at least one overlapping area overlap between each two consequent frames by projecting them onto a sphere so that each pixel becomes a 3D vector and correcting the 3D vectors according to the combined, full calibration matrix.

15. The system of claim 14, wherein said plurality of imagers are mounted on the lateral sides of a housing of a virtual reality imaging device.

16. A software program product for stitching a plurality of frames captured at multiple viewing angles by a plurality of imagers mounted around a common center into a panoramic frame imaging, at least partly, an environment surrounding said common center, comprising:

a non-transitory computer readable storage medium;

first program instructions for receiving a set of a plurality of frames, each of the plurality of frames is captured at a different viewing angle from other frames by a plurality of imagers mounted around a common center region to image an environment surrounding said common center region;

second program instructions for providing a plurality of calibration models each is calculated based on one or more intrinsic parameters of one of said plurality of imagers;

third program instructions for identifying in each of a plurality of pairs of neighboring frames from said plurality of frames at least one overlapping area using at least some of the plurality of calibration models, each said pair of neighboring frames comprises frames captured about the same time by two imagers from said plurality of imagers, said two imagers having overlapping fields of view;

fourth program instructions for performing the following for each said at least one overlapping area;

identifying a center connecting axis connecting between centers of respective said pair of neighboring frames;

calculating a motion gradient for each frame of said pair of neighboring frames;

adapting pixels of said pair of neighboring frames in said at least one overlapping area based on corresponding values from said motion gradient along said center connecting axis; and

stitching said pair of adapted neighboring frames into a panoramic frame which at least partly imaging an environment surrounding said common center;

wherein each of the one or more intrinsic parameters is selected from a group consisting of a principal point parameter, a focal length parameter, and a fisheye distortion parameter;

wherein said first, second, third, and fourth program instructions are executed by at least one computerized processor from said non-transitory computer readable storage medium;

wherein at least one of said plurality of calibration models comprises a fish-eye correction matrix and an extrinsic camera calibration matrix which map coordinates of pixels to corrected coordinates defined according to symmetry axes of an image plane of respective said imager;

wherein, in each of said plurality of frames, said at least one overlapping area is calculated based on a combined matrix generated according to said fish-eye correction matrix and said extrinsic camera calibration matrix are combined into a combined matrix used for calculating said at least one overlapping area overlap between each two consequent frames by projecting them onto a sphere so that each pixel becomes a 3D vector and correcting the 3D vectors according to the combined, full calibration matrix.