Image stitching

• US 9,930,238 B2
• Filed: 08/21/2014
• Issued: 03/27/2018
• Est. Priority Date: 08/21/2013
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Jaunt Inc. (Verizon Communications Inc.)
• Inventors: Annau, Thomas M., Van Hoff, Arthur, Kopeinigg, Daniel
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Duley, Janese
• Application Number:
  US14/465,581
• Publication Number:
  US 20150054913A1
• Time in Patent Office: 1,314 Days
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06Q 30/0241 Advertisements
• CPC Class Codes: G06Q 30/0246 Traffic
• CPC Class Codes: G06Q 30/0251 Targeted advertisements
• CPC Class Codes: G06Q 30/0263 based upon Internet or website rating
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0273 Determination of fees for advertising
• CPC Class Codes: G06Q 30/0277 Online advertisement
• CPC Class Codes: G06Q 30/0284 Time or distance, e.g. usage of parking meters or taximeters
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G11B 27/11 by using information not detectable on the record carrier
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: G11B 31/006 with video camera or receiver
• CPC Class Codes: H04N 13/106 : Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/111 : Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/117 : the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/189 : Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/204 : using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/243 : using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 : wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/282 : for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/698 : for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

The disclosure includes a system and method for stitching image frames. The system includes a processor and a memory storing instructions that, when executed, cause the system to: receive image frames that are captured by two or more camera modules at a particular time; interpolate a virtual camera between a first set of camera modules from the two or more camera modules; determine a set of disparity maps between the first set of camera modules; generate, based on the set of disparity maps, a virtual camera image associated with the particular time for the virtual camera from a set of image frames captured by the first set of camera modules at the particular time; and construct a left panoramic image and a right panoramic image associated with the particular time from the image frames captured by the two or more camera modules and the virtual camera image.

Claims

1. A method comprising:

receiving a first set of image frames that are captured by a first set of camera modules at a particular time, wherein (1) the first set of camera modules includes a first camera module and a second camera module that have an overlapping field of view, (2) the first set of image frames includes a first image captured by the first camera module and a second image captured by the second camera module, and (3) the first image includes a first sub-image that overlaps with a second sub-image of the second image on the overlapping field of view;

interpolating a first virtual camera between the first camera module and the second camera module;

determining a first set of disparity maps between the first set of camera modules by;

determining a first disparity map that maps disparity of pixels from the first sub-image to the second sub-image;

determining a second disparity map that maps disparity of pixels from the second sub-image to the first sub-image; and

for each corresponding pixel location in the overlapping field of view;

determining similarity scores between a first pixel of the first sub-image at a first pixel location and second pixels of the second sub-image at second pixel locations, wherein a corresponding distance between the first pixel location and each of the second pixel locations is equal to a different disparity value;

determining a highest similarity score from the similarity scores;

determining a disparity value associated with the highest similarity score; and

assigning the disparity value associated with the highest similarity score to the first pixel location in the first disparity map;

generating, by a processor-based computing device programmed to perform the generating, a first virtual camera image associated with the particular time for the first virtual camera from the first set of image frames, the first virtual camera image being generated based on the first set of disparity maps and a position of the first virtual camera relative to the first camera module and the second camera module; and

constructing a left panoramic image and a right panoramic image associated with the particular time from the first set of image frames captured by the first set of camera modules and the first virtual camera image of the first virtual camera.

2. The method of claim 1, wherein:

the first image and the second image differ in their displacement along an epipolar line that connects projection centers of the first camera module and the second camera module; and

further comprising determining the similarity scores for pixels along the epipolar line.

3. The method of claim 1, wherein generating the first virtual camera image for the first virtual camera comprises:

determining the position of the first virtual camera relative to the first and second camera modules;

generating a first shifted sub-image from the first sub-image based on the first disparity map and the position of the first virtual camera;

generating a second shifted sub-image from the second sub-image based on the second disparity map and the position of the first virtual camera; and

combining the first shifted sub-image and the second shifted sub-image to generate the first virtual camera image for the first virtual camera.

4. The method of claim 1, wherein determining the first disparity map that maps disparity of pixels from the first sub-image to the second sub-image further comprises:

determining that a disparity collision occurs when third pixels of the first sub-image map to a fourth pixel of the second sub-image;

determining the similarity scores between the third pixels of the first sub-image and the fourth pixel of the second sub-image;

assigning the disparity value for a collision pixel of the third pixels with a highest similarity score of the similarity scores to the pixel location for the fourth pixel in the first disparity map; and

leaving corresponding map entries blank for other collision pixels of the third pixels with lower similarity scores of the similarity scores.

5. The method of claim 1, wherein determining the second disparity map that maps disparity of pixels from the second sub-image to the first sub-image comprises:

for each corresponding pixel location in the overlapping field of view;

determining similarity scores between a third pixel of the second sub-image at a third pixel location and fourth pixels of the first sub-image at fourth pixel locations, wherein the corresponding distance between the third pixel location and each of the fourth pixel locations is equal to the different disparity value;

determining the highest similarity score from the similarity scores;

determining the disparity value associated with the highest similarity score; and

assigning the disparity value associated with the highest similarity score to the third pixel location in the second disparity map.

6. The method of claim 1, further comprising:

interpolating a second virtual camera between the first virtual camera and the first camera module from the first set of camera modules;

determining a second set of disparity maps associated with the first virtual camera and the first camera module;

generating, based on the second set of disparity maps, a second virtual camera image associated with the particular time for the second virtual camera from an image frame of the first camera module and the first virtual camera image of the first virtual camera; and

wherein the left panoramic image and the right panoramic image associated with the particular time are constructed from the first set of image frames captured by the first set of camera modules, the first virtual camera image of the first virtual camera, and the second virtual camera image of the second virtual camera.

7. The method of claim 1, further comprising:

interpolating a third virtual camera between a second set of camera modules;

determining a third set of disparity maps associated with the second set of camera modules;

generating, based on the third set of disparity maps, a third virtual camera image associated with the particular time for the third virtual camera from a second set of image frames that are captured by the second set of camera modules at the particular time; and

wherein the left panoramic image and the right panoramic image associated with the particular time are constructed from the second set of image frames captured by the second set of camera modules, the first virtual camera image of the first virtual camera, and the third virtual camera image of the third virtual camera.

8. The method of claim 1, further comprising:

constructing a left camera map and a right camera map based on configurations of the first set of camera modules and the first virtual camera;

wherein the left panoramic image is constructed from the first set of image frames and the first virtual camera image based on the left camera map; and

wherein the right panoramic image is constructed from the first set of image frames and the first virtual camera image based on the right camera map.

9. A system comprising:

one or more processors; and

one or more non-transitory tangible computer readable mediums communicatively coupled to the one or more processors and storing executable instructions executable by the one or more processors to perform operations comprising;

receiving a first set of image frames that are captured by a first set of camera modules at a particular time, wherein (1) the first set of camera modules includes a first camera module and a second camera module that have an overlapping field of view, (2) the first set of image frames includes a first image captured by the first camera module and a second image captured by the second camera module, and (3) the first image includes a first sub-image that overlaps with a second sub-image of the second image on the overlapping field of view;

interpolating a first virtual camera between the first camera module and the second camera module;

determining a first set of disparity maps between the first set of camera modules by;

determining a first disparity map that maps disparity of pixels from the first sub-image to the second sub-image;

determining a second disparity map that maps disparity of pixels from the second sub-image to the first sub-image; and

for each corresponding pixel location in the overlapping field of view;

determining similarity scores between a first pixel of the first sub-image at a first pixel location and second pixels of the second sub-image at second pixel locations, wherein a corresponding distance between the first pixel location and each of the second pixel locations is equal to a different disparity value;

determining a highest similarity score from the similarity scores;

determining a disparity value associated with the highest similarity score; and

assigning the disparity value associated with the highest similarity score to the first pixel location in the first disparity map;

generating, based on the first set of disparity maps and a position of the first virtual camera relative to the first set of camera modules, a first virtual camera image associated with the particular time for the first virtual camera from the first set of image frames, the first virtual camera image being generated based on the first set of disparity maps and a position of the first virtual camera relative to the first camera module and the second camera module; and

constructing a left panoramic image and a right panoramic image associated with the particular time from the first set of image frames captured by the first set of camera modules and the first virtual camera image of the first virtual camera.

10. The system of claim 9, wherein:

the first image and the second image differ in their displacement along an epipolar line that connects projection centers of the first camera module and the second camera module.

11. The system of claim 9, wherein the instructions executable by the one or more processors generate the first virtual camera image for the first virtual camera by:

determining the position of the first virtual camera relative to the first and second camera modules;

generating a first shifted sub-image from the first sub-image based on the first disparity map and the position of the first virtual camera;

generating a second shifted sub-image from the second sub-image based on the second disparity map and the position of the first virtual camera; and

combining the first shifted sub-image and the second shifted sub-image to generate the first virtual camera image for the first virtual camera.

12. The system of claim 9, wherein the instructions executable by the one or more processors further determine the first disparity map that maps disparity of pixels from the first sub-image to the second sub-image by:

determining that a disparity collision occurs when third pixels of the first sub-image map to a fourth pixel of the second sub-image;

determining the similarity scores between the third pixels of the first sub-image and the fourth pixel of the second sub-image;

assigning the disparity value for a collision pixel of the third pixels with a highest similarity score of the similarity scores to the pixel location for the fourth pixel in the first disparity map; and

leaving corresponding map entries blank for other collision pixels of the third pixels with lower similarity scores of the similarity scores.

13. The system of claim 9, wherein the instructions executable by the one or more processors determine the second disparity map that maps disparity of pixels from the second sub-image to the first sub-image by:

for each corresponding pixel location in the overlapping field of view;

determining similarity scores between a third pixel of the second sub-image at a third pixel location and fourth pixels of the first sub-image at fourth pixel locations, wherein the corresponding distance between the third pixel location and each of the fourth pixel locations is equal to the different disparity value;

determining the highest similarity score from the similarity scores;

determining the disparity value associated with the highest similarity score; and

assigning the disparity value associated with the highest similarity score to the third pixel location in the second disparity map.

14. The system of claim 9, wherein the instructions executable by the one or more processors perform operations further comprising:

interpolating a second virtual camera between the first virtual camera and the first camera module from the first set of camera modules;

determining a second set of disparity maps associated with the first virtual camera and the first camera module;

generating, based on the second set of disparity maps, a second virtual camera image associated with the particular time for the second virtual camera from an image frame of the first camera module and the first virtual camera image of the first virtual camera; and

wherein the left panoramic image and the right panoramic image associated with the particular time are constructed from the first set of image frames captured by the first set of camera modules, the first virtual camera image of the first virtual camera, and the second virtual camera image of the second virtual camera.

15. The system of claim 9, wherein the instructions executable by the one or more processors perform operations further comprising:

interpolating a third virtual camera between a second set of camera modules;

determining a third set of disparity maps associated with the second set of camera modules;

generating, based on the third set of disparity maps, a third virtual camera image associated with the particular time for the third virtual camera from a second set of image frames captured by the second set of camera modules at the particular time; and

wherein the left panoramic image and the right panoramic image associated with the particular time are constructed from the second set of image frames captured by the second set of camera modules, the first virtual camera image of the first virtual camera, and the third virtual camera image of the third virtual camera.

16. A computer program product comprising a non-transitory computer-usable medium including a computer-readable program, wherein the computer-readable program when executed on a computer causes the computer to:

receive a first set of image frames that are captured by a first set of camera modules at a particular time, wherein (1) the first set of camera modules includes a first camera module and a second camera module that have an overlapping field of view, (2) the first set of image frames includes a first image captured by the first camera module and a second image captured by the second camera module, and (3) the first image includes a first sub-image that overlaps with a second sub-image of the second image on the overlapping field of view;

interpolate a first virtual camera between the first camera module and the second camera module;

determine a first set of disparity maps between the first set of camera modules by;

determining a first disparity map that maps disparity of pixels from the first sub-image to the second sub-image;

determining a second disparity map that maps disparity of pixels from the second sub-image to the first sub-image; and

for each corresponding pixel location in the overlapping field of view;

determining similarity scores between a first pixel of the first sub-image at a first pixel location and second pixels of the second sub-image at second pixel locations, wherein a corresponding distance between the first pixel location and each of the second pixel locations is equal to a different disparity value;

determining a highest similarity score from the similarity scores;

determining a disparity value associated with the highest similarity score; and

assigning the disparity value associated with the highest similarity score to the first pixel location in the first disparity map;

generate, based on the first set of disparity maps and a position of the first virtual camera relative to the first set of camera modules, a first virtual camera image associated with the particular time for the first virtual camera from the first set of image frames, the first virtual camera image being generated based on the first set of disparity maps and a position of the first virtual camera relative to the first camera module and the second camera module; and

construct a left panoramic image and a right panoramic image associated with the particular time from the set of image frames captured by the first set of camera modules and the first virtual camera image of the first virtual camera.

17. The computer program product of claim 16, wherein:

the first image and the second image differ in their displacement along an epipolar line that connects projection centers of the first camera module and the second camera module.

18. The computer program product of claim 16, wherein generating the first virtual camera image for the first virtual camera comprises:

determining the position of the first virtual camera relative to the first and second camera modules;

generating a first shifted sub-image from the first sub-image based on the first disparity map and the position of the first virtual camera;

generating a second shifted sub-image from the second sub-image based on the second disparity map and the position of the first virtual camera; and

combining the first shifted sub-image and the second shifted sub-image to generate the first virtual camera image for the first virtual camera.

19. The computer program product of claim 16, wherein determining the first disparity map that maps disparity of pixels from the first sub-image to the second sub-image further comprises:

determining that a disparity collision occurs when third pixels of the first sub-image map to a fourth pixel of the second sub-image;

determining the similarity scores between the third pixels of the first sub-image and the fourth pixel of the second sub-image;

assigning the disparity value for a collision pixel of the third pixels with a highest similarity score of the similarity scores to the pixel location for the fourth pixel in the first disparity map; and

leaving corresponding map entries blank for other collision pixels of the third pixels with lower similarity scores of the similarity scores.

20. The computer program product of claim 16, wherein determining the second disparity map that maps disparity of pixels from the second sub-image to the first sub-image comprises:

for each corresponding pixel location in the overlapping field of view;

determining similarity scores between a third pixel of the second sub-image at the third pixel location and fourth pixels of the first sub-image at fourth pixel locations, wherein the corresponding distance between the pixel location and each of the second pixel locations is equal to the different disparity value;

determining the highest similarity score from the similarity scores;

determining the disparity value associated with the highest similarity score; and

assigning the disparity value associated with the highest similarity score to the third pixel location in the second disparity map.

21. The computer program product of claim 16, wherein the computer-readable program when executed on the computer further causes the computer to:

interpolate a second virtual camera between the first virtual camera and the first camera module from the first set of camera modules;

determine a second set of disparity maps associated with the first virtual camera and the first camera module;

generate, based on the second set of disparity maps, a second virtual camera image associated with the particular time for the second virtual camera from an image frame of the first camera module and the first virtual camera image of the first virtual camera; and

wherein the left panoramic image and the right panoramic image associated with the particular time are constructed from the first set of image frames captured by the first set of camera modules, the first virtual camera image of the first virtual camera, and the second virtual camera image of the second virtual camera.

22. The computer program product of claim 16, wherein the computer-readable program when executed on the computer further causes the computer to:

interpolate a third virtual camera between a second set of camera modules;

determine a third set of disparity maps associated with the second set of camera module;

generate, based on the third set of disparity maps, a third virtual camera image associated with the particular time for the third virtual camera from a second set of image frames that are captured by the second set of camera modules at the particular time; and

wherein the left panoramic image and the right panoramic image associated with the particular time are constructed from the second set of image frames captured by the second set of camera modules, the first virtual camera image of the first virtual camera, and the third virtual camera image of the third virtual camera.