IMAGE STITCHING

• US 20150054913A1
• Filed: 08/21/2014
• Published: 02/26/2015
• Est. Priority Date: 08/21/2013
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Jaunt Inc. (Verizon Communications Inc.)
• Inventors: ANNAU, Thomas M., VAN HOFF, Arthur, KOPEINIGG, Daniel
• Granted Patent: US 9,930,238 B2
• Time in Patent Office: Days
• US Class Current: 348/36
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06Q 30/0241 Advertisements
• CPC Class Codes: G06Q 30/0246 Traffic
• CPC Class Codes: G06Q 30/0251 Targeted advertisements
• CPC Class Codes: G06Q 30/0263 based upon Internet or website rating
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0273 Determination of fees for advertising
• CPC Class Codes: G06Q 30/0277 Online advertisement
• CPC Class Codes: G06Q 30/0284 Time or distance, e.g. usage of parking meters or taximeters
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G11B 27/11 by using information not detectable on the record carrier
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: G11B 31/006 with video camera or receiver
• CPC Class Codes: H04N 13/106 : Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/111 : Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/117 : the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/189 : Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/204 : using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/243 : using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 : wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/282 : for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/698 : for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

The disclosure includes a system and method for stitching image frames. The system includes a processor and a memory storing instructions that, when executed, cause the system to: receive image frames that are captured by two or more camera modules at a particular time; interpolate a virtual camera between a first set of camera modules from the two or more camera modules; determine a set of disparity maps between the first set of camera modules; generate, based on the set of disparity maps, a virtual camera image associated with the particular time for the virtual camera from a set of image frames captured by the first set of camera modules at the particular time; and construct a left panoramic image and a right panoramic image associated with the particular time from the image frames captured by the two or more camera modules and the virtual camera image.

Claims

1. A method comprising:

receiving image frames that are captured by two or more camera modules of a camera array at a particular time;

interpolating a first virtual camera between a first set of camera modules from the two or more camera modules;

determining a first set of disparity maps between the first set of camera modules;

generating, by a processor-based computing device programmed to perform the generating, a first virtual camera image associated with the particular time for the first virtual camera from a first set of image frames that are captured by the first set of camera modules at the particular time, the first virtual camera image being generated based on the first set of disparity maps; and

constructing a left panoramic image and a right panoramic image associated with the particular time from the image frames captured by the two or more camera modules and the first virtual camera image of the first virtual camera.

2. The method of claim 1, wherein:

the first set of camera modules includes a first camera module and a second camera module that have an overlapping field of view;

the first set of image frames captured by the first set of camera modules includes a first image captured by the first camera module and a second image captured by the second camera module;

the first image includes a first sub-image that overlaps with a second sub-image of the second image on the overlapping field of view; and

determining the first set of disparity maps between the first set of camera modules comprises;

determining a first disparity map that maps disparity of pixels from the first sub-image to the second sub-image; and

determining a second disparity map that maps disparity of pixels from the second sub-image to the first sub-image.

3. The method of claim 2, wherein generating the first virtual camera image for the first virtual camera comprises:

determining a position of the first virtual camera relative to the first and second camera modules;

generating a first shifted sub-image from the first sub-image based on the first disparity map and the position of the first virtual camera;

generating a second shifted sub-image from the second sub-image based on the second disparity map and the position of the first virtual camera; and

combining the first shifted sub-image and the second shifted sub-image to generate the first virtual camera image for the first virtual camera.

4. The method of claim 2, wherein determining the first disparity map that maps disparity of pixels from the first sub-image to the second sub-image comprises:

for each corresponding pixel location in the overlapping field of view;

determining similarity scores between a first pixel of the first sub-image at the pixel location and second pixels of the second sub-image at second pixel locations, wherein a corresponding distance between the pixel location and each of the second pixel locations is equal to a different disparity value;

determining a highest similarity score from the similarity scores;

determining a disparity value associated with the highest similarity score; and

assigning the disparity value associated with the highest similarity score to the pixel location in the first disparity map.

5. The method of claim 2, wherein determining the second disparity map that maps disparity of pixels from the second sub-image to the first sub-image comprises:

for each corresponding pixel location in the overlapping field of view;

determining similarity scores between a first pixel of the second sub-image at the pixel location and second pixels of the first sub-image at second pixel locations, wherein a corresponding distance between the pixel location and each of the second pixel locations is equal to a different disparity value;

determining a highest similarity score from the similarity scores;

determining a disparity value associated with the highest similarity score; and

assigning the disparity value associated with the highest similarity score to the pixel location in the second disparity map.

6. The method of claim 1, further comprising:

interpolating a second virtual camera between the first virtual camera and a first camera module from the first set of camera modules;

determining a second set of disparity maps associated with the first virtual camera and the first camera module;

generating, based on the second set of disparity maps, a second virtual camera image associated with the particular time for the second virtual camera from an image frame of the first camera module and the first virtual camera image of the first virtual camera module; and

wherein the left panoramic image and the right panoramic image associated with the particular time are constructed from the image frames captured by the two or more camera modules, the first virtual camera image of the first virtual camera, and the second virtual camera image of the second virtual camera.

7. The method of claim 1, further comprising:

interpolating a third virtual camera between a second set of camera modules from the two or more camera modules;

determining a third set of disparity maps associated with the second set of camera modules;

generating, based on the third set of disparity maps, a third virtual camera image associated with the particular time for the third virtual camera from a second set of image frames that are captured by the second set of camera modules at the particular time; and

wherein the left panoramic image and the right panoramic image associated with the particular time are constructed from the image frames captured by the two or more camera modules, the first virtual camera image of the first virtual camera, and the third virtual camera image of the third virtual camera.

8. The method of claim 1, further comprising:

constructing a left camera map and a right camera map based on configurations of the two or more camera modules and the first virtual camera;

wherein the left panoramic image is constructed from the image frames and the first virtual camera image based on the left camera map; and

wherein the right panoramic image is constructed from the image frames and the first virtual camera image based on the right camera map.

9. A system comprising:

one or more processors;

one or more non-transitory tangible computer readable mediums communicatively coupled to the one or more processors and storing executable instructions executable by the one or more processors to perform operations comprising;

receiving image frames that are captured by two or more camera modules of a camera array at a particular time;

interpolating a first virtual camera between a first set of camera modules from the two or more camera modules;

determining a first set of disparity maps between the first set of camera modules;

generating, based on the first set of disparity maps, a first virtual camera image associated with the particular time for the first virtual camera from a first set of image frames that are captured by the first set of camera modules at the particular time; and

constructing a left panoramic image and a right panoramic image associated with the particular time from the image frames captured by the two or more camera modules and the first virtual camera image of the first virtual camera.

10. The system of claim 9, wherein:

the first set of camera modules includes a first camera module and a second camera module that have an overlapping field of view;

the first set of image frames captured by the first set of camera modules includes a first image captured by the first camera module and a second image captured by the second camera module;

the first image includes a first sub-image that overlaps with a second sub-image of the second image on the overlapping field of view; and

the instructions executable by the one or more processors determine the first set of disparity maps between the first set of camera modules by;

determining a first disparity map that maps disparity of pixels from the first sub-image to the second sub-image; and

determining a second disparity map that maps disparity of pixels from the second sub-image to the first sub-image.

11. The system of claim 10, wherein the instructions executable by the one or more processors generate the first virtual camera image for the first virtual camera by:

determining a position of the first virtual camera relative to the first and second camera modules;

generating a first shifted sub-image from the first sub-image based on the first disparity map and the position of the first virtual camera;

generating a second shifted sub-image from the second sub-image based on the second disparity map and the position of the first virtual camera; and

combining the first shifted sub-image and the second shifted sub-image to generate the first virtual camera image for the first virtual camera.

12. The system of claim 10, wherein the instructions executable by the one or more processors determine the first disparity map that maps disparity of pixels from the first sub-image to the second sub-image by:

for each corresponding pixel location in the overlapping field of view;

determining similarity scores between a first pixel of the first sub-image at the pixel location and second pixels of the second sub-image at second pixel locations, wherein a corresponding distance between the pixel location and each of the second pixel locations is equal to a different disparity value;

determining a highest similarity score from the similarity scores;

determining a disparity value associated with the highest similarity score; and

assigning the disparity value associated with the highest similarity score to the pixel location in the first disparity map.

13. The system of claim 10, wherein the instructions executable by the one or more processors determine the second disparity map that maps disparity of pixels from the second sub-image to the first sub-image by:

for each corresponding pixel location in the overlapping field of view;

determining similarity scores between a first pixel of the second sub-image at the pixel location and second pixels of the first sub-image at second pixel locations, wherein a corresponding distance between the pixel location and each of the second pixel locations is equal to a different disparity value;

determining a highest similarity score from the similarity scores;

determining a disparity value associated with the highest similarity score; and

assigning the disparity value associated with the highest similarity score to the pixel location in the second disparity map.

14. The system of claim 9, wherein the instructions executable by the one or more processors perform operations further comprising:

interpolating a second virtual camera between the first virtual camera and a first camera module from the first set of camera modules;

determining a second set of disparity maps associated with the first virtual camera and the first camera module;

generating, based on the second set of disparity maps, a second virtual camera image associated with the particular time for the second virtual camera from an image frame of the first camera module and the first virtual camera image of the first virtual camera module; and

wherein the left panoramic image and the right panoramic image associated with the particular time are constructed from the image frames captured by the two or more camera modules, the first virtual camera image of the first virtual camera, and the second virtual camera image of the second virtual camera.

15. The system of claim 9, wherein the instructions executable by the one or more processors perform operations further comprising:

interpolating a third virtual camera between a second set of camera modules from the two or more camera modules;

determining a third set of disparity maps associated with the second set of camera modules;

generating, based on the third set of disparity maps, a third virtual camera image associated with the particular time for the third virtual camera from a second set of image frames captured by the second set of camera modules at the particular time; and

wherein the left panoramic image and the right panoramic image associated with the particular time are constructed from the image frames captured by the two or more camera modules, the first virtual camera image of the first virtual camera, and the third virtual camera image of the third virtual camera.

16. A computer program product comprising a non-transitory computer-usable medium including a computer-readable program, wherein the computer-readable program when executed on a computer causes the computer to:

receive image frames that are captured by two or more camera modules of a camera array at a particular time;

interpolate a first virtual camera between a first set of camera modules from the two or more camera modules;

determine a first set of disparity maps between the first set of camera modules;

generate, based on the first set of disparity maps, a first virtual camera image associated with the particular time for the first virtual camera from a first set of image frames that are captured by the first set of camera modules at the particular time; and

construct a left panoramic image and a right panoramic image associated with the particular time from the image frames captured by the two or more camera modules and the first virtual camera image of the first virtual camera.

17. The computer program product of claim 16, wherein:

the first set of camera modules includes a first camera module and a second camera module that have an overlapping field of view;

the first set of image frames captured by the first set of camera modules includes a first image captured by the first camera module and a second image captured by the second camera module;

the first image includes a first sub-image that overlaps with a second sub-image of the second image on the overlapping field of view; and

determining the first set of disparity maps between the first set of camera modules comprises;

determining a first disparity map that maps disparity of pixels from the first sub-image to the second sub-image; and

determining a second disparity map that maps disparity of pixels from the second sub-image to the first sub-image.

18. The computer program product of claim 17, wherein generating the first virtual camera image for the first virtual camera comprises:

determining a position of the first virtual camera relative to the first and second camera modules;

generating a first shifted sub-image from the first sub-image based on the first disparity map and the position of the first virtual camera;

generating a second shifted sub-image from the second sub-image based on the second disparity map and the position of the first virtual camera; and

combining the first shifted sub-image and the second shifted sub-image to generate the first virtual camera image for the first virtual camera.

19. The computer program product of claim 17, wherein determining the first disparity map that maps disparity of pixels from the first sub-image to the second sub-image comprises:

for each corresponding pixel location in the overlapping field of view;

determining similarity scores between a first pixel of the first sub-image at the pixel location and second pixels of the second sub-image at second pixel locations, wherein a corresponding distance between the pixel location and each of the second pixel locations is equal to a different disparity value;

determining a highest similarity score from the similarity scores;

determining a disparity value associated with the highest similarity score; and

assigning the disparity value associated with the highest similarity score to the pixel location in the first disparity map.

20. The computer program product of claim 17, wherein determining the second disparity map that maps disparity of pixels from the second sub-image to the first sub-image comprises:

for each corresponding pixel location in the overlapping field of view;

determining similarity scores between a first pixel of the second sub-image at the pixel location and second pixels of the first sub-image at second pixel locations, wherein a corresponding distance between the pixel location and each of the second pixel locations is equal to a different disparity value;

determining a highest similarity score from the similarity scores;

determining a disparity value associated with the highest similarity score; and

assigning the disparity value associated with the highest similarity score to the pixel location in the second disparity map.

21. The computer program product of claim 16, wherein the computer-readable program when executed on the computer further causes the computer to:

interpolate a second virtual camera between the first virtual camera and a first camera module from the first set of camera modules;

determine a second set of disparity maps associated with the first virtual camera and the first camera module;

generate, based on the second set of disparity maps, a second virtual camera image associated with the particular time for the second virtual camera from an image frame of the first camera module and the first virtual camera image of the first virtual camera module; and

wherein the left panoramic image and the right panoramic image associated with the particular time are constructed from the image frames captured by the two or more camera modules, the first virtual camera image of the first virtual camera, and the second virtual camera image of the second virtual camera.

22. The computer program product of claim 16, wherein the computer-readable program when executed on the computer further causes the computer to:

interpolate a third virtual camera between a second set of camera modules from the two or more camera modules;

determine a third set of disparity maps associated with the second set of camera modules;

generate, based on the third set of disparity maps, a third virtual camera image associated with the particular time for the third virtual camera from a second set of image frames that are captured by the second set of camera modules at the particular time; and

wherein the left panoramic image and the right panoramic image associated with the particular time are constructed from the image frames captured by the two or more camera modules, the first virtual camera image of the first virtual camera, and the third virtual camera image of the third virtual camera.