ADAPTIVE STITCHING OF FRAMES IN THE PROCESS OF CREATING A PANORAMIC FRAME

• US 20190019299A1
• Filed: 12/12/2016
• Published: 01/17/2019
• Est. Priority Date: 01/03/2016
• Status: Abandoned Application

• Current Assignee: HumanEyes Technologies Limited
• Original Assignee: HumanEyes Technologies Limited
• Inventors: BAR, Anton, TULBOVICH, Yitzchak, FINE, Shmuel
• Application Number:
  US16/067,832
• Publication Number:
  US 20190019299A1
• Time in Patent Office: Days
• CPC Class Codes: G03B 35/08 by simultaneous recording
• CPC Class Codes: G03B 35/10 having single camera with stereoscopic-base-defining system
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: G06T 7/38 Registration of image sequences
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

A method of creating a panoramic frame by combining frames along an irregular line identified in an overlapping depicted in the frames. The method comprises identifying an overlap between content documented in a first frame portion of a first frame and content documented in a second frame portion of a second frame, segmenting a plurality of objects in the first frame portion, identifying boundaries of the plurality of objects, identifying an irregular line path connecting between two opposing sides of the first frame portion along at least some of the boundaries of at least some of the plurality of objects, and stitching the first frame and second frame along the irregular line path in the process of creating a panoramic frame.

Claims

1. A method of creating a panoramic frame by combining frames along an irregular line identified in an overlapping depicted in the frames, comprising:

identifying an overlap between content documented in a first frame portion of a first frame and content documented in a second frame portion of a second frame;

wherein said first frame and said second frame are captured by two neighboring imagers from a plurality of imagers housed in a housing of an imaging device, said two neighboring imagers having overlapping fields of view;

segmenting a plurality of objects in said first frame portion;

identifying boundaries of said plurality of objects;

identifying an irregular line path connecting between two opposing sides of said first frame portion along at least some of said boundaries of at least some of said plurality of objects; and

stitching said first frame and second frame along said irregular line path in the process of creating a panoramic frame.

2. The method of claim 1, further comprising detecting in said first frame a plurality of feature points each having at least one predefined feature point characteristic;

wherein said plurality of objects are segmented according to said plurality of feature points.

3. The method of claim 2, wherein said plurality of objects are identified on a watershed image generated from said first portion based on said plurality of feature points.

4. The method of claim 2, wherein said plurality of feature points are identified using an accelerated segment test (FAST) process.

5. The method of claim 1, wherein said overlap is identified by conducting a feature descriptor matching process.

7. The method of claim 1, wherein said two neighboring imagers are mounted around a common center together with other imagers.

8. The method of claim 7, wherein an optical axis of a first of said two neighboring imagers is tilted in relation to an axis passing through said common center and a respective tangential point of an origin point of a field of view of said first neighboring imager on a virtual circle passing via all origin points of the fields of view of said plurality of imagers.

9. The method of claim 8, further comprising:

adjusting said irregular line path according to a new irregular line path calculated similarly to said irregular line path for a pair of new frames captured after said first frame and second frame, and

using said adjusted irregular line path for stitching said new images.

10. The method of claim 9, wherein said adjusting and said using are performed when a difference between said irregular line path and said new irregular line path is greater than a threshold.

11. The method of claim 1, wherein said panoramic frame is one of a plurality of frames of a virtual reality (VR) file.

12. The method of claim 1, wherein said panoramic frame is a left eye panoramic frame and said plurality of imagers are intertwined with a plurality of additional imagers capturing a plurality of additional frames which are combined into a right eye panoramic frame;

further comprising combining said right eye panoramic frame with said left eye panoramic frame for creating a stereoscopic frame.

13. The method of claim 12, wherein members of said plurality of imagers and said plurality of additional frames are alternately arranged along a virtual circle encircling a common center.

14. The method of claim 1, wherein said panoramic frame is a right eye panoramic frame and said plurality of imagers are intertwined with a plurality of additional imagers capturing a plurality of additional frames which are combined into a left eye panoramic frame;

further comprising combining said left eye panoramic frame with said right eye panoramic frame for creating a stereoscopic frame.

15. The method of claim 1, wherein said plurality of imagers are arranged along a virtual circle encircling a common center;

wherein each one of said plurality of imagers is mounted such that an optical axis thereof is tilted in relation to an axis passing through said common center and a tangential point of an origin of a field of view of a respective said imager.

6. (canceled)

16. A system of creating a panoramic frame by combining frames along an irregular line identified in an overlapping depicted in the frames, comprising:

an interface adapted for receiving first and second frames from two of a plurality of frames captured by a plurality of imagers of an imaging device;

a code store adapted for storing a code;

a processor adapted for executing said code, wherein said code comprising;

code instructions for identifying an overlap between content documented in a first frame portion of a first frame and content documented in a second frame portion of a second frame;

wherein said first frame and said second frame are captured by two neighboring imagers from a plurality of imagers housed in a housing of an imaging device, said two neighboring imagers having overlapping fields of view;

code instructions for segmenting a plurality of objects in said first frame portion;

code instructions for identifying boundaries of said plurality of objects;

code instructions for identifying an irregular line path connecting between two opposing sides of said first frame portion along at least some of said boundaries of at least some of said plurality of objects; and

code instructions for stitching said first frame and second frame along said irregular line path in the process of creating a panoramic frame.

17. A software program product for creating a panoramic frame by combining frames along an irregular line identified in an overlapping depicted in the frames, comprising:

a non-transitory computer readable storage medium;

first program instructions for identifying an overlap between content documented in a first frame portion of a first frame and content documented in a second frame portion of a second frame;

wherein said first frame and said second frame are captured by two neighboring imagers from a plurality of imagers housed in a housing of an imaging device, said two neighboring imagers having overlapping fields of view;

second program instructions for segmenting a plurality of objects in said first frame portion;

third program instructions for identifying boundaries of said plurality of objects;

fourth program instructions for identifying an irregular line path connecting between two opposing sides of said first frame portion along at least some of said boundaries of at least some of said plurality of objects; and

fifth program instructions for stitching said first frame and second frame along said irregular line path in the process of creating a panoramic frame;

wherein said first, second, third, fourth and fifth program instructions are executed by at least one computerized processor from said non-transitory computer readable storage medium.