Generating content for a virtual reality system

• US 10,708,568 B2
• Filed: 08/21/2014
• Issued: 07/07/2020
• Est. Priority Date: 08/21/2013
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Inventors: Christensen, Jens, Annau, Thomas M., Van Hoff, Arthur
• Primary Examiner(s): Refai, Sam
• Application Number:
  US14/465,570
• Publication Number:
  US 20150058102A1
• Time in Patent Office: 2,147 Days
• Field of Search: 705 141- 1473
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06Q 30/0241 Advertisements
• CPC Class Codes: G06Q 30/0246 Traffic
• CPC Class Codes: G06Q 30/0251 Targeted advertisements
• CPC Class Codes: G06Q 30/0263 based upon Internet or website rating
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0273 Determination of fees for advertising
• CPC Class Codes: G06Q 30/0277 Online advertisement
• CPC Class Codes: G06Q 30/0284 Time or distance, e.g. usage of parking meters or taximeters
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G11B 27/11 by using information not detectable on the record carrier
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: G11B 31/006 with video camera or receiver
• CPC Class Codes: H04N 13/106 : Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/111 : Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/117 : the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/189 : Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/204 : using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/243 : using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 : wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/282 : for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/698 : for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

The disclosure includes a system and method for generating virtual reality content. For example, the disclosure includes a method for generating virtual reality content that includes a stream of three-dimensional video data and a stream of three-dimensional audio data with a processor-based computing device programmed to perform the generating, providing the virtual reality content to a user, detecting a location of the user'"'"'s gaze at the virtual reality content, and suggesting an advertisement based on the location of the user'"'"'s gaze. Another example includes receiving virtual reality content that includes a stream of three-dimensional video data and a stream of three-dimensional audio data to a first user with a processor-based computing device programmed to perform the receiving, generating a social network for the first user, and generating a social graph that includes user interactions with the virtual reality content.

Claims

1. A method, comprising:

calibrating camera modules of a camera array by;

estimating errors in a predicted roll, pitch, and yaw of two or more lenses corresponding to two or more camera modules;

determining a position and a rotational offset for each of the two or more lenses; and

determining a relative position of each of two or more lenses;

receiving raw video data from the calibrated camera modules of the camera array;

identifying location and timing associated with each of the calibrated camera modules;

constructing a left camera map that identifies matching camera modules for pixels in a left panoramic image and a right camera map that identifies matching camera modules for pixels in a right panoramic image;

generating, based on the left camera map, a stream of left panoramic images;

generating, based on the right camera map, a stream of right panoramic images;

generating three-dimensional content from the stream of left panoramic images, the stream of right panoramic images, and a stream of three-dimensional audio data;

providing the three-dimensional content to a user through a three-dimensional display;

receiving head tracking information from one or more accelerometers or gyroscopes of a viewing system, where the head tracking information describes a head orientation of the user and a gaze of the user while the user is viewing the three-dimensional content;

detecting the location of the gaze of the user at the three-dimensional content based on the head tracking information;

determining a location of a stitching aberration in the three-dimensional content; and

generating a first advertisement that is stitched into the location of the stitching aberration in the three-dimensional content provided through the three-dimensional display.

2. The method of claim 1, further comprising determining a cost for displaying advertisements based on the location of the gaze of the user at the three-dimensional content.

3. The method of claim 2, wherein determining the cost for displaying advertisements is further based on a length of time that the user gazes at the location of the gaze of the user at the three-dimensional content.

4. The method of claim 1, further comprising providing a graphical object as part of the three-dimensional content that is linked to a second advertisement.

5. The method of claim 1, further comprising:

generating graphics for displaying a bottom portion and a top portion that include at least some of the three-dimensional content; and

providing an advertisement that is part of at least the bottom portion or the top portion.

6. The method of claim 4, wherein providing the graphical object comprises providing the graphical object being configured to enable the user to interact with the graphical object as part of the three-dimensional content.

7. The method of claim 1, further comprising determining a geographic location of the user, wherein detecting the location of the gaze of the user at the three-dimensional content is further based on the geographic location of the user.

8. The method of claim 1, further comprising determining one or more keywords of the three-dimensional content, wherein generating the first advertisement within the three-dimensional is further based on the one or more keywords of the three-dimensional content.

9. The method of claim 1, further comprising:

determining one or more pixels associated with the location of the gaze of the user at the three-dimensional; and

determining a graphical object within the three-dimensional content that corresponds with the one or more pixels, wherein generating the first advertisement within the three-dimensional content is further based on the graphical object.

10. The method of claim 1, wherein generating the first advertisement includes stitching the first advertisement into the three-dimensional content.

11. The method of claim 1, wherein stitching the first advertisement into the three-dimensional content based on the location of the gaze of the user includes placing the first advertisement where at least 80% of other users looked when they viewed the three-dimensional content.

12. The method of claim 1, further comprising receiving user input indicating a user preference for viewing and/or interacting with advertisements in the three-dimensional content, wherein generating the first advertisement is further based on the user preference.

13. The method of claim 1, further comprising:

determining the user is interacting with the first advertisement within the three-dimensional content; and

generating additional three-dimensional content related to the first advertisement.

14. A method, comprising:

calibrating camera modules of a camera array by;

estimating errors in a predicted roll, pitch, and yaw of two or more lenses corresponding to two or more camera modules;

determining a position and a rotational offset for each of the two or more lenses; and

determining a relative position of each of two or more lenses;

receiving raw video data from the calibrated camera modules of the camera array;

identifying location and timing associated with each of the calibrated camera modules;

constructing a left camera map that identifies matching camera modules for pixels in a left panoramic image and a right camera map that identifies matching camera modules for pixels in a right panoramic image;

generating, based on the left camera map, a stream of left panoramic images;

generating, based on the right camera map, a stream of right panoramic images;

generating three-dimensional content from the stream of left panoramic images, the stream of right panoramic images, and a stream of three-dimensional audio data;

providing the three-dimensional content to a user through a three-dimensional display;

receiving head tracking information from one or more accelerometers or gyroscopes of a viewing system, where the head tracking information describes a head orientation of the user and a gaze of the user while the user is viewing the three-dimensional content;

detecting the location of the gaze of the user at the three-dimensional content based on the head tracking information;

determining a location of a stitching aberration in the three-dimensional content;

generating a first advertisement that is stitched into the location of the stitching aberration in the three-dimensional content provided through the three-dimensional display;

generating an advertisement log, wherein the advertising log includes various advertisements that the user interacts with within the three-dimensional content;

determining a user profile based on the advertisement log; and

generating a second advertisement based on the user profile.

15. The method of claim 14, wherein generating the first advertisement includes stitching the first advertisement into the three-dimensional content.

16. One or more non-transitory computer-readable media that include instructions that, when executed by one or more processors, are configured to cause the one or more processors to perform operations, the operations comprising:

calibrating camera modules of a camera array by;

estimating errors in a predicted roll, pitch, and yaw of two or more lenses corresponding to two or more camera modules;

determining a position and a rotational offset for each of the two or more lenses; and

determining a relative position of each of two or more lenses;

receiving raw video data from the calibrated camera modules of the camera array;

identifying location and timing associated with each of the calibrated camera modules;

constructing a left camera map that identifies matching camera modules for pixels in a left panoramic image and a right camera map that identifies matching camera modules for pixels in a right panoramic image;

generating, based on the left camera map, a stream of left panoramic images;

generating, based on the right camera map, a stream of right panoramic images;

generating three-dimensional content from the stream of left panoramic images, the stream of right panoramic images, and a stream of three-dimensional audio data;

providing the three-dimensional content to a user through a three-dimensional display;

receiving head tracking information from one or more accelerometers or gyroscopes of a viewing system, where the head tracking information describes a head orientation of the user and a gaze of the user while the user is viewing the three-dimensional content;

detecting the location of the gaze of the user at the three-dimensional content based on the head tracking information;

determining a location of a stitching aberration in the three-dimensional content; and

generating a first advertisement that is stitched into the location of the stitching aberration in the three-dimensional content provided through the three-dimensional display.

17. The computer-readable media of claim 16, the operations further comprising determining a cost for displaying advertisements based on the location of the gaze of the user at the three-dimensional content.

18. The computer-readable media of claim 17, wherein determining the cost for displaying advertisements comprises determining the cost based on a length of time that the user gazes at the location of the gaze of the user at the three-dimensional content.