AGGREGATING IMAGES AND AUDIO DATA TO GENERATE VIRTUAL REALITY CONTENT

• US 20150055937A1
• Filed: 08/21/2014
• Published: 02/26/2015
• Est. Priority Date: 08/21/2013
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Jaunt Inc. (Verizon Communications Inc.)
• Inventors: VAN HOFF, Arthur, ANNAU, Thomas M., CHRISTENSEN, JENS
• Granted Patent: US 10,334,220 B2
• Time in Patent Office: Days
• US Class Current: 386/285
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06Q 30/0241 Advertisements
• CPC Class Codes: G06Q 30/0246 Traffic
• CPC Class Codes: G06Q 30/0251 Targeted advertisements
• CPC Class Codes: G06Q 30/0263 based upon Internet or website rating
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0273 Determination of fees for advertising
• CPC Class Codes: G06Q 30/0277 Online advertisement
• CPC Class Codes: G06Q 30/0284 Time or distance, e.g. usage of parking meters or taximeters
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G11B 27/11 by using information not detectable on the record carrier
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: G11B 31/006 with video camera or receiver
• CPC Class Codes: H04N 13/106 : Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/111 : Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/117 : the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/189 : Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/204 : using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/243 : using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 : wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/282 : for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/698 : for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

The disclosure includes a system and method for aggregating image frames and audio data to generate virtual reality content. The system includes a processor and a memory storing instructions that, when executed, cause the system to: receive video data describing image frames from a camera array; receive audio data from a microphone array; aggregate the image frames to generate a stream of three-dimensional (3D) video data, the stream of 3D video data including a stream of left panoramic images and a stream of right panoramic images; generate a stream of 3D audio data from the audio data; and generate virtual reality content that includes the stream of 3D video data and the stream of 3D audio data.

Claims

1. A computer-implemented method comprising:

receiving video data describing image frames from camera modules;

receiving audio data from a microphone array;

aggregating, by a processor-based computing device programmed to perform the aggregating, the image frames to generate a stream of three-dimensional (3D) video data, the stream of 3D video data including a stream of left panoramic images and a stream of right panoramic images;

generating a stream of 3D audio data from the audio data; and

generating virtual reality content that includes the stream of 3D video data and the stream of 3D audio data.

2. The method of claim 1, wherein aggregating the image frames to generate the stream of 3D video data comprises:

identifying first matching camera modules for left panoramic images based on a left camera map;

identifying second matching camera modules for right panoramic images based on a right camera map;

stitching first image frames captured by the first matching camera modules at a particular time to form a corresponding left panoramic image in the stream of left panoramic images; and

stitching second image frames captured by the second matching camera modules at a particular time to form a corresponding right panoramic image in the stream of right panoramic images.

3. The method of claim 2, wherein:

for a pixel with a yaw value and a pitch value in a panorama;

the left camera map identifies a first matching camera module for the pixel in the panorama and matches the pixel in the panorama to a pixel in an image plane of the first matching camera module; and

the right camera map identifies a second matching camera module for the pixel in the panorama and matches the pixel in the panorama to a pixel in an image plane of the second matching camera module.

4. The method of claim 2, wherein:

the left camera map associates a pixel location in left panoramic images to a corresponding first matching camera module, wherein the pixel location corresponds to a point of a panorama in a left viewing direction;

the corresponding first matching camera module has a field of view that includes a viewing direction to the point of the panorama; and

the viewing direction of the corresponding first matching camera module is closer to the left viewing direction than other viewing directions associated with other camera modules.

5. The method of claim 1, wherein aggregating the image frames to generate the stream of 3D video data comprises:

determining a current viewing direction associated with a user; and

generating the stream of left panoramic images and the stream of right panoramic images based on the current viewing direction.

6. The method of claim 5, wherein:

the left panoramic images have a higher resolution in the current viewing direction of the user than a second viewing direction opposite to the current viewing direction; and

the right panoramic images have a higher resolution in the current viewing direction of the user than the second viewing direction opposite to the current viewing direction.

7. The method of claim 1, further comprising:

correcting color deficiencies in the left panoramic images and the right panoramic images; and

correcting stitching errors in the left panoramic images and the right panoramic images.

8. A system comprising:

one or more processors;

one or more non-transitory tangible computer-readable mediums communicatively coupled to the one or more processors and storing executable instructions executable by the one or more processors to perform operations comprising;

receiving video data describing image frames from camera modules;

receiving audio data from a microphone array;

aggregating the stream of 3D video data including a stream of left panoramic images and a stream of right panoramic images;

generating a stream of 3D audio data from the audio data; and

generating virtual reality content that includes the stream of 3D video data and the stream of 3D audio data.

9. The system of claim 8, wherein the instructions executable by the one or more processors aggregate the image frames to generate the stream of 3-D video data by:

identifying first matching camera modules for left panoramic images based on a left camera map;

identifying second matching camera modules for right panoramic images based on a right camera map;

stitching first image frames captured by the first matching camera modules at a particular time to form a corresponding left panoramic image in the steam of left panoramic images; and

stitching second image frames captured by the second matching camera modules at a particular time to form a corresponding right panoramic image in the stream of right panoramic images.

10. The system of claim 9, wherein:

for a pixel with a yaw value and a pitch value in a panorama;

the left camera map identifies a first matching camera module for the pixel in the panorama and matches the pixel in the panorama to a pixel in an image plane of the first matching camera module; and

the right camera map identifies a second matching camera module for the pixel in the panorama and matches the pixel in the panorama to a pixel in an image plane of the second matching camera module.

11. The system of claim 9, wherein:

the left camera map associates a pixel location in left panoramic images to a corresponding first matching camera module, wherein the pixel location corresponds to a point of a panorama in a left viewing direction;

the corresponding first matching camera module has a field of view that includes a viewing direction to the point of the panorama; and

the viewing direction of the corresponding first matching camera module is closer to the left viewing direction than other viewing directions associated with other camera modules.

12. The system of claim 8, wherein the instructions executable by the one or more processors aggregates the image frames to generate the stream of 3D video data by:

determining a current viewing direction associated with a user; and

generating the stream of left panoramic images and the stream of right panoramic images based on the current viewing direction.

13. The system of claim 12, wherein:

the left panoramic images have a higher resolution in the current viewing direction of the user than a second viewing direction opposite to the current viewing direction; and

the right panoramic images have a higher resolution in the current viewing direction of the user than the second viewing direction opposite to the current viewing direction.

14. The system of claim 8, wherein the instructions executable by the one or more processors perform operations further comprising:

correcting color deficiencies in the left panoramic images and the right panoramic images; and

correcting stitching errors in the left panoramic images and the right panoramic images.

15. A computer program product comprising a non-transitory computer-usable medium including a computer-readable program, wherein the computer-readable program when executed on a computer causes the computer to:

receive video data describing image frames from camera modules;

receive audio data from a microphone array;

aggregate the image frames to generate a stream of three-dimensional (3D) video data, the stream of 3D video data including a stream of left panoramic images and a stream of right panoramic images;

generate a stream of 3D audio data from the audio data; and

generate virtual reality content that includes the stream of 3D video data and the stream of 3D audio data.

16. The computer program product of claim 15, wherein aggregating the image frames to generate the stream of 3D video data comprises:

identifying first matching camera modules for left panoramic images based on a left camera map;

identifying second matching camera modules for right panoramic images based on a right camera map;

stitching first image frames captured by the first matching camera modules at a particular time to form a corresponding left panoramic image in the stream of left panoramic images; and

stitching second image frames captured by the second matching camera modules at a particular time to form a corresponding right panoramic image in the stream of right panoramic images.

17. The computer program product of claim 16, wherein:

for a pixel with a yaw value and a pitch value in a panorama;

the left camera map identifies a first matching camera module for the pixel in the panorama and matches the pixel in the panorama to a pixel in an image plane of the first matching camera module; and

the right camera map identifies a second matching camera module for the pixel in the panorama and matches the pixel in the panorama to a pixel in an image plane of the second matching camera module.

18. The computer program product of claim 16, wherein:

the left camera map associates a pixel location in left panoramic images to a corresponding first matching camera module, wherein the pixel location corresponds to a point of a panorama in a left viewing direction;

the corresponding first matching camera module has a field of view that includes a viewing direction to the point of the panorama; and

the viewing direction of the corresponding first matching camera module is closer to the left viewing direction than other viewing directions associated with other camera modules.

19. The computer program product of claim 15, wherein aggregating the image frames to generate the stream of 3D video data comprises:

determining a current viewing direction associated with a user; and

generating the stream of left panoramic images and the stream of right panoramic images based on the current viewing direction.

20. The computer program product of claim 19, wherein:

the left panoramic images have a higher resolution in the current viewing direction of the user than a second viewing direction opposite to the current viewing direction; and

the right panoramic images have a higher resolution in the current viewing direction of the user than the second viewing direction opposite to the current viewing direction.