Aggregating images and audio data to generate virtual reality content

• US 10,334,220 B2
• Filed: 08/21/2014
• Issued: 06/25/2019
• Est. Priority Date: 08/21/2013
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Jaunt Inc. (Verizon Communications Inc.)
• Inventors: Van Hoff, Arthur, Annau, Thomas M., Christensen, Jens
• Primary Examiner(s): Dang, Hung Q
• Application Number:
  US14/465,575
• Publication Number:
  US 20150055937A1
• Time in Patent Office: 1,769 Days
• Field of Search: 386223-224, 386239-248, 386341
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06Q 30/0241 Advertisements
• CPC Class Codes: G06Q 30/0246 Traffic
• CPC Class Codes: G06Q 30/0251 Targeted advertisements
• CPC Class Codes: G06Q 30/0263 based upon Internet or website rating
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0273 Determination of fees for advertising
• CPC Class Codes: G06Q 30/0277 Online advertisement
• CPC Class Codes: G06Q 30/0284 Time or distance, e.g. usage of parking meters or taximeters
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G11B 27/11 by using information not detectable on the record carrier
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: G11B 31/006 with video camera or receiver
• CPC Class Codes: H04N 13/106 : Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/111 : Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/117 : the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/189 : Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/204 : using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/243 : using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 : wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/282 : for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/698 : for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

The disclosure includes a system and method for aggregating image frames and audio data to generate virtual reality content. The system includes a processor and a memory storing instructions that, when executed, cause the system to: receive video data describing image frames from a camera array; receive audio data from a microphone array; aggregate the image frames to generate a stream of three-dimensional (3D) video data, the stream of 3D video data including a stream of left panoramic images and a stream of right panoramic images; generate a stream of 3D audio data from the audio data; and generate virtual reality content that includes the stream of 3D video data and the stream of 3D audio data.

Claims

1. A computer-implemented method comprising:

receiving video data describing image frames from camera modules;

receiving audio data from a microphone array;

determining a first matching camera module by;

determining a set of camera modules that have a point in their respective fields of view;

determining a left viewing direction from a left eye position to the point;

determining a set of viewing directions to the point for the set of camera modules; and

selecting the first matching camera module based on the first matching camera module having a first viewing direction that is most parallel to the left viewing direction, wherein the first viewing direction is determined to be the most parallel to the left viewing direction based on the first viewing direction forming a smallest angle with the left viewing direction as compared to other angles formed between the left viewing direction and other viewing directions associated with other camera modules from the set of camera modules;

constructing a left camera map that associates a first pixel location in a left panoramic image to the first matching camera module, wherein the first pixel location corresponds to the point in a panorama from the left viewing direction;

generating, based on the left camera map, a stream of left panoramic images;

constructing a right camera map that associates a second pixel location in a right panoramic image to a second matching camera module, wherein the second pixel location corresponds to the point in the panorama from a right viewing direction and the second matching camera module is selected based on having a second field of view that includes a second viewing direction that is most parallel to the second viewing direction as compared to the other viewing directions associated with the other camera modules;

generating, based on the right camera map, a stream of right panoramic images;

generating a stream of three dimensional (3D) video data from the stream of left panoramic images and the stream of right panoramic images;

generating a stream of 3D audio data from the audio data; and

generating augmented reality content that includes the stream of 3D video data and the stream of 3D audio data.

2. The method of claim 1, wherein the first matching camera module is part of a set of first matching camera modules, the second matching camera module is part of a set of second matching camera modules, and generating the stream of 3D video data comprises:

stitching first image frames captured by the set of first matching camera modules at a particular time to form a corresponding left panoramic image in the stream of left panoramic images; and

stitching second image frames captured by the set of second matching camera modules at a particular time to form a corresponding right panoramic image in the stream of right panoramic images.

3. The method of claim 1, wherein:

for a pixel with a yaw value and a pitch value in the panorama;

the left camera map matches the pixel in the panorama to a first corresponding pixel in a first image plane of the first matching camera module; and

the right camera map matches the pixel in the panorama to a second corresponding pixel in a second image plane of the second matching camera module.

4. The method of claim 3, wherein the pixel is formed by blending first pixel values for the first corresponding pixel with second pixel values for the second corresponding pixel.

5. The method of claim 1, wherein generating the stream of 3D video data comprises:

determining a current viewing direction associated with a user; and

generating the stream of left panoramic images and the stream of right panoramic images based on the current viewing direction.

6. The method of claim 5, wherein:

the left panoramic images have a higher resolution in the current viewing direction of the user than a second viewing direction opposite to the current viewing direction; and

the right panoramic images have a higher resolution in the current viewing direction of the user than the second viewing direction opposite to the current viewing direction.

7. The method of claim 1, further comprising:

correcting color deficiencies in the left panoramic images and the right panoramic images; and

correcting stitching errors in the left panoramic images and the right panoramic images.

8. A system comprising:

one or more processors;

one or more non-transitory tangible computer-readable mediums communicatively coupled to the one or more processors and storing executable instructions executable by the one or more processors to perform operations comprising;

receiving video data describing image frames from camera modules;

receiving audio data from a microphone array;

determining a first matching camera module by;

determining a set of camera modules that have a point in their respective fields of view;

determining a left viewing direction from a left eye position to the point;

determining a set of viewing directions to the point for the set of camera modules; and

selecting the first matching camera module based on the first matching camera module having a first viewing direction that is most parallel to the left viewing direction, wherein the first viewing direction is determined to be the most parallel to the left viewing direction based on the first viewing direction forming a smallest angle with the left viewing direction as compared to other angles formed between the left viewing direction and other viewing directions associated with other camera modules from the set of camera modules;

constructing a left camera map that associates a first pixel in a left panoramic image to the first matching camera module, wherein a first pixel location associated with the first pixel corresponds to the point in a panorama from the left viewing direction, the first pixel has a yaw value and a pitch value in the panorama, and the first pixel is matched to a first corresponding pixel in an image plane of the first matching camera module;

generating, based on the left camera map, a stream of left panoramic images;

constructing a right camera map that associates a second pixel location in a right panoramic image to a second matching camera module, wherein the second pixel location corresponds to the point in the panorama from a right viewing direction and the second matching camera module is selected based on having a second field of view that includes a second viewing direction that is most parallel to the second viewing direction as compared to the other viewing directions associated with the other camera modules;

generating, based on the right camera map, a stream of right panoramic images;

generating a stream of 3D audio data from the audio data; and

generating augmented reality content that includes the stream of 3D video data and the stream of 3D audio data.

9. The system of claim 8, wherein the first matching camera module is part of a set of first matching camera modules, the second matching camera module is part of a set of second matching camera modules, and the instructions executable by the one or more processors generate the stream of 3-D video data by:

stitching first image frames captured by the first set of matching camera modules at a particular time to form a corresponding left panoramic image in the steam of left panoramic images; and

stitching second image frames captured by the set of second matching camera modules at a particular time to form a corresponding right panoramic image in the stream of right panoramic images.

10. The system of claim 8, wherein the first pixel is on a border between two images and the first pixel is transformed into a blended pixel by separating the first pixel into a first high-frequency part and a first low-frequency part and combining the first low-frequency part with a second low-frequency part corresponding to a second pixel.

11. The system of claim 8, wherein the instructions executable by the one or more processors aggregates the image frames to generate the stream of 3D video data by:

determining a current viewing direction associated with a user; and

generating the stream of left panoramic images and the stream of right panoramic images based on the current viewing direction.

12. The system of claim 11, wherein:

the left panoramic images have a higher resolution in the current viewing direction of the user than a second viewing direction opposite to the current viewing direction; and

the right panoramic images have a higher resolution in the current viewing direction of the user than the second viewing direction opposite to the current viewing direction.

13. The system of claim 8, wherein the instructions executable by the one or more processors perform operations further comprising:

correcting color deficiencies in the left panoramic images and the right panoramic images; and

correcting stitching errors in the left panoramic images and the right panoramic images.

14. A computer program product comprising a non-transitory computer-usable medium including a computer-readable program, wherein the computer-readable program when executed on a computer causes the computer to:

receive video data describing image frames from camera modules;

receive audio data from a microphone array;

determining a first matching camera module by;

determining a set of camera modules that have a point in their respective fields of view;

determining a left viewing direction from a left eye position to the point;

determining a set of viewing directions to the point for the set of camera modules; and

selecting the first matching camera module based on the first matching camera module having a first viewing direction that is most parallel to the left viewing direction, wherein the first viewing direction is determined to be the most parallel to the left viewing direction based on the first viewing direction forming a smallest angle with the left viewing direction as compared to other angles formed between the left viewing direction and other viewing directions associated with other camera modules from the set of camera modules;

construct a left camera map that associates a first pixel location in a left panoramic image to the first matching camera module, wherein the first pixel location corresponds to the point in a panorama from the left viewing direction;

generate, based on the left camera map, a stream of left panoramic images;

construct a right camera map that associates a second pixel location in a right panoramic image to a second matching camera module, wherein the second pixel location corresponds to the point in the panorama from a right viewing direction and the second matching camera module is selected based on having a second field of view that includes a second viewing direction that is most parallel to the second viewing direction as compared to the other viewing directions associated with the other camera modules;

generate, based on the right camera map, a stream of right panoramic images;

generate a stream of three dimensional (3D) video data from the stream of left panoramic images and the stream of right panoramic images;

generate a stream of 3D audio data from the audio data; and

generate augmented reality content that includes the stream of 3D video data and the stream of 3D audio data.

15. The computer program product of claim 14, wherein the first matching camera module is part of a set of first matching camera modules, the second matching camera module is part of a set of second matching camera modules, and generating the stream of 3D video data comprises:

stitching first image frames captured by the first set of matching camera modules at a particular time to form a corresponding left panoramic image in the stream of left panoramic images; and

stitching second image frames captured by the second set of matching camera modules at a particular time to form a corresponding right panoramic image in the stream of right panoramic images.

16. The computer program product of claim 14, wherein:

for a pixel with a yaw value and a pitch value in the panorama;

the left camera map matches the pixel in the panorama to a first corresponding pixel in a first image plane of the first matching camera module; and

the right camera map matches the pixel in the panorama to a second corresponding pixel in a second image plane of the second matching camera module.

17. The computer program product of claim 16, wherein the pixel is formed by blending first pixel values for the first corresponding pixel with second pixel values for the second corresponding pixel.

18. The computer program product of claim 14, wherein generating the stream of 3D video data comprises:

determining a current viewing direction associated with a user; and

generating the stream of left panoramic images and the stream of right panoramic images based on the current viewing direction.

19. The computer program product of claim 18, wherein:

the left panoramic images have a higher resolution in the current viewing direction of the user than a second viewing direction opposite to the current viewing direction; and

the right panoramic images have a higher resolution in the current viewing direction of the user than the second viewing direction opposite to the current viewing direction.