STITCHING FRAMES INTO A PANORAMIC FRAME

• US 20180063513A1
• Filed: 12/12/2016
• Published: 03/01/2018
• Est. Priority Date: 01/03/2016
• Status: Active Grant

• Current Assignee: HumanEyes Technologies Limited
• Original Assignee: HumanEyes Technologies Limited
• Inventors: BAR, Anton, TULBOVICH, Yitzchak, FINE, Shmuel
• Granted Patent: US 10,460,459 B2
• Time in Patent Office: Days
• CPC Class Codes: G03B 35/08 by simultaneous recording
• CPC Class Codes: G03B 35/10 having single camera with stereoscopic-base-defining system
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: G06T 7/38 Registration of image sequences
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

A method of stitching frames captured at multiple viewing angles comprising: receiving a set of frames captured by imagers, identifying in each of pairs of neighboring frames, each neighboring frame is captured about the same time as the other from an imager having an overlapping field of view with another. For each overlapping area a center connecting axis connecting between centers of respective the pair of neighboring frames is identified, a motion gradient for each frame of the pair of neighboring frames is calculated, pixels of the pair of neighboring frames in the at least one overlapping area are adapted based on corresponding values from the motion gradient along the center connecting axis, the pair of adapted neighboring frames into a panoramic frame which at least partly imaging an environment surrounding the common center is stitched.

Claims

1. A method of stitching a plurality of frames captured at multiple viewing angles by a plurality of imagers mounted around a common center into a panoramic frame imaging, at least partly, an environment surrounding said common center, comprising:

receiving a set of a plurality of frames captured at multiple viewing angles by a plurality of imagers mounted around a common center region to image an environment surrounding said common center region;

identifying in each of a plurality of pairs of neighboring frames from said plurality of frames at least one overlapping area, each said pair of neighboring frames comprises frames captured about the same time by two imagers from said plurality of imagers, said two imagers having overlapping fields of view;

for each said at least one overlapping area;

identifying a center connecting axis connecting between centers of respective said pair of neighboring frames;

calculating a motion gradient for each frame of said pair of neighboring frames, values of said motion gradient increase towards a lateral edge of a respective said frame;

adapting pixels of said pair of neighboring frames in said at least one overlapping area based on corresponding values from said motion gradient along said center connecting axis; and

stitching said pair of adapted neighboring frames into a panoramic frame which at least partly imaging an environment surrounding said common center.

2. The method of claim 1, wherein said panoramic frame is a left eye panoramic frame and said plurality of imagers are intertwined with a plurality of additional imagers capturing a plurality of additional frames which are combined into a right eye panoramic frame;

further comprising combining said right eye panoramic frame with said left eye panoramic frame for creating a stereoscopic frame.

3. The method of claim 2, wherein members of said plurality of imagers and said plurality of additional frames are alternately arranged along a virtual circle encircling said common center.

4. The method of claim 1, wherein said panoramic frame is a right eye panoramic frame and said plurality of imagers are intertwined with a plurality of additional imagers capturing a plurality of additional frames which are combined into a left eye panoramic frame;

further comprising combining said left eye panoramic frame with said right eye panoramic frame for creating a stereoscopic frame.

5. The method of claim 1, wherein said plurality of imagers are arranged along a virtual circle encircling said common center;

wherein each one of said plurality of imagers is mounted such that an optical axis thereof is tilted in relation to an axis passing through said common center and a tangential point of an origin of a field of view of a respective said imager.

6. The method of claim 1, wherein said panoramic frame is a 360 degrees spherical panorama frame imaging said environment around said common center.

7. The method of claim 6, wherein said panoramic frame is used as one of two of frames comprising a stereoscopic frame.

8. The method of claim 1, wherein said method is repeated for adapting respective said at least one overlapping area in each frame of each of a plurality of sequentially captured sets.

9. The method of claim 1, further comprising estimating a projection of a visual representation of said plurality of frames onto a sphere skeleton.

10. The method of claim 1, wherein said at least one overlapping area is identified using parameters from at least one calibration model defined for at least one of said plurality of imagers.

11. The method of claim 10, wherein said at least one calibration model is defined for each one of said plurality of imagers according to at least one member of a group consisting of a principal point parameter, a focal length parameter, and a fisheye distortion parameter.

12. The method of claim 10, wherein said at least one calibration model comprises a fish-eye correction matrix and an extrinsic camera calibration matrix which map coordinates of pixels to corrected coordinates defined according to symmetry axes of an image plane of respective said imager.

13. The method of claim 12, wherein, in each of said plurality of frames, said at least one overlapping area is calculated based on a combined matrix generated according to said fish-eye correction matrix and said extrinsic camera calibration matrix are combined into a combined matrix used for calculating said at least one overlapping area overlap between each two consequent frames by projecting them onto a sphere so that each pixel becomes a 3D vector and correcting the 3D vectors according to the combined, full calibration matrix.

14. The method of claim 1, wherein each one of said plurality of imagers is a camera comprising a fish eye lens.

15. The method of claim 1, wherein each one of said plurality of imagers is calibrated using a Camera-Rig Calibration.

16. The method of claim 1, wherein said adapting comprises shifting pixels located in said overlapping area towards said lateral edge along said respective said axis.

17. The method of claim 16, further comprising correcting a plurality of blank pixels formed as an outcome of said shifting by interpolation.

18. A system of stitching a plurality of frames captured at multiple viewing angles by a plurality of imagers mounted around a common center into a panoramic frame imaging, at least partly, an environment surrounding said common center, comprising:

an interface adapted to receive a set of a plurality of frames captured about the same time at multiple viewing angles by a plurality of imagers mounted around a common center region to image an environment surrounding said common center region;

a code store adapted to store a code;

a processor adapted to execute said code for;

identifying in each of a plurality of pairs of neighboring frames from said plurality of frames at least one overlapping area, each said pair of neighboring frames comprises frames captured about the same time by two imagers from said plurality of imagers, said two imagers having overlapping fields of view;

for each said at least one overlapping area;

identifying a center connecting axis connecting between centers of respective said pair of neighboring frames;

calculating a motion gradient for each frame of said pair of neighboring frames, values of said motion gradient increase towards a lateral edge of a respective said frame;

adapting pixels of said pair of neighboring frames in said at least one overlapping area based on corresponding values from said motion gradient along said center connecting axis; and

stitching said pair of adapted neighboring frames into a panoramic frame which at least partly imaging an environment surrounding said common center.

19. The system of claim 18, wherein said plurality of imagers are mounted on the lateral sides of a housing of a virtual reality imaging device.

20. A software program product for stitching a plurality of frames captured at multiple viewing angles by a plurality of imagers mounted around a common center into a panoramic frame imaging, at least partly, an environment surrounding said common center, comprising:

a non-transitory computer readable storage medium;

first program instructions for receiving a set of a plurality of frames captured at multiple viewing angles by a plurality of imagers mounted around a common center region to image an environment surrounding said common center region;

second program instructions for identifying in each of a plurality of pairs of neighboring frames from said plurality of frames at least one overlapping area, each said pair of neighboring frames comprises frames captured about the same time by two imagers from said plurality of imagers, said two imagers having overlapping fields of view;

third program instructions for performing the following for each said at least one overlapping area;

identifying a center connecting axis connecting between centers of respective said pair of neighboring frames;

calculating a motion gradient for each frame of said pair of neighboring frames, values of said motion gradient increase towards a lateral edge of a respective said frame;

adapting pixels of said pair of neighboring frames in said at least one overlapping area based on corresponding values from said motion gradient along said center connecting axis; and

stitching said pair of adapted neighboring frames into a panoramic frame which at least partly imaging an environment surrounding said common center;

wherein said first, second and third program instructions are executed by at least one computerized processor from said non-transitory computer readable storage medium.