Generating content for a virtual reality system

• US 10,666,921 B2
• Filed: 05/02/2017
• Issued: 05/26/2020
• Est. Priority Date: 08/21/2013
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Inventors: Christensen, Jens, Annau, Thomas M., Van Hoff, Arthur
• Primary Examiner(s): Refai, Sam
• Application Number:
  US15/585,154
• Publication Number:
  US 20170236162A1
• Time in Patent Office: 1,120 Days
• Field of Search: 705 141- 1473
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06Q 30/0241 Advertisements
• CPC Class Codes: G06Q 30/0246 Traffic
• CPC Class Codes: G06Q 30/0251 Targeted advertisements
• CPC Class Codes: G06Q 30/0263 based upon Internet or website rating
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0273 Determination of fees for advertising
• CPC Class Codes: G06Q 30/0277 Online advertisement
• CPC Class Codes: G06Q 30/0284 Time or distance, e.g. usage of parking meters or taximeters
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G11B 27/11 by using information not detectable on the record carrier
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: G11B 31/006 with video camera or receiver
• CPC Class Codes: H04N 13/106 : Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/111 : Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/117 : the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/189 : Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/204 : using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/243 : using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 : wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/282 : for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/698 : for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

The disclosure includes a system and method for generating virtual reality content. For example, the disclosure includes a method for generating virtual reality content that includes a stream of three-dimensional video data and a stream of three-dimensional audio data with a processor-based computing device programmed to perform the generating, providing the virtual reality content to a user, detecting a location of the user'"'"'s gaze at the virtual reality content, and suggesting an advertisement based on the location of the user'"'"'s gaze. Another example includes receiving virtual reality content that includes a stream of three-dimensional video data and a stream of three-dimensional audio data to a first user with a processor-based computing device programmed to perform the receiving, generating a social network for the first user, and generating a social graph that includes user interactions with the virtual reality content.

Claims

1. A method comprising:

receiving video data from camera modules of a camera array;

identifying location and timing associated with each of the camera modules;

constructing a left camera map that identifies matching camera modules for pixels in a left panoramic image from the video data and a right camera map that identifies matching camera modules for pixels in a right panoramic image from the video data;

generating, based on the left camera map, a stream of left panoramic images;

generating, based on the right camera map, a stream of right panoramic images;

generating three-dimensional content from the stream of left panoramic images, the stream of right panoramic images, and a stream of three-dimensional audio data;

providing the three-dimensional content to a first user through a virtual reality display;

receiving head tracking information from one or more accelerometers or gyroscopes of a viewing system, where the head tracking information describes a head orientation of the first user and gazes of the first user while the first user is viewing the three-dimensional content;

determining locations of the gazes of the first user at the three-dimensional content based on the head tracking information;

generating social data describing an interaction of the first user with the three-dimensional content based on the head tracking information;

updating a first user profile associated with the first user in a social network to include subject matter associated with the three-dimensional content, wherein the social network is based on common features of the first user and one or more other users;

identifying a second user with a second user profile that includes the subject matter;

providing a suggestion that the first user make a connection with the second user in the social network based on the first user profile and the second user profile both including the subject matter associated with the three-dimensional content;

responsive to the first user selecting the suggestion to make the connection, adding the connection between the first user and the second user to a social graph associated with the social network;

automatically generating a social network update that describes an action of the first user interacting with the three-dimensional content, the social network update including a join button for accessing the three-dimensional content with which the first user interacted;

providing the social network update to one or more other users in the social network that viewed the three-dimensional content; and

responsive to the second user of the social network selecting the join button, providing the three-dimensional content at a time that is substantially simultaneous to the time that the first user is viewing the three-dimensional content.

2. The method of claim 1, wherein the second user viewed the three-dimensional content with a same subject matter and the connection is further based on the first user and the second user viewing the three-dimensional content with the same subject matter.

3. The method of claim 1, wherein the second user viewed the three-dimensional content with a same subject matter and further comprising suggesting a group associated with the social network to the first user and the second user, wherein the group discusses the subject matter.

4. The method of claim 1, further comprising:

determining the one or more other users that are interested in the subject matter associated with the three-dimensional content the first user interacted with;

wherein the one or more other users receive the social network updates that describe the action of the first user interacting with the three-dimensional content and an option to provide the first user with an in-game reward.

5. The method of claim 1, further comprising generating privacy settings for the first user for determining whether to publish social network updates based on a type of activity.

6. The method of claim 1, further comprising:

storing information in a social graph about the gazes of the first user at advertisements displayed as part of the three-dimensional content.

7. The method of claim 1, wherein the second user is an expert in the subject matter and suggesting the connection between the first user and the second user includes a suggestion that the first user follow the second user in the social network based on the second user being the expert in the subject matter.

8. A method comprising:

receiving video data from camera modules of a camera array;

identifying location and timing associated with each of the camera modules;

constructing a left camera map that identifies matching camera modules for pixels in a left panoramic image from the video data and a right camera map that identifies matching camera modules for pixels in a right panoramic image from the video data;

generating, based on the left camera map, a stream of left panoramic images;

generating, based on the right camera map, a stream of right panoramic images;

generating three-dimensional content from the stream of left panoramic images, the stream of right panoramic images, and a stream of three-dimensional audio data;

providing the three-dimensional content to a first user through a virtual reality display;

receiving head tracking information from one or more accelerometers or gyroscopes of a viewing system, where the head tracking information describes a head orientation of the first user and gazes of the first user while the first user is viewing the three-dimensional content provided through the three-dimensional display;

determining locations of the gazes of the first user at the three-dimensional content provided through the three-dimensional display based on the head tracking information;

generating a social network for the first user by associating the first user with one or more other users based on common features viewed by the first user in the three-dimensional content and one or more other users;

generating a social graph that includes interactions of the first user with the three-dimensional content, a mapping of how the first user and the one or more other users are connected, and information about the gazes of the first user at advertisements displayed as part of the three-dimensional content;

providing a suggestion that the first user make a connection with the one or more other users based on the three-dimensional content being viewed by at least one of the first user and the one or more other users;

responsive to the first user selecting the suggestion to make the connection, adding the connection between the first user and the one or more other users to the social graph;

automatically generating a social network update that describes an action of the first user interacting with the three-dimensional content, the social network update including a join button for accessing the three-dimensional content with which the first user interacted;

providing the social network update to one or more other users in the social network that viewed the three-dimensional content; and

responsive to the one or more other users of the social network selecting the join button, providing the three-dimensional content at a time that is substantially simultaneous to the time that the first user is viewing the three-dimensional content.

9. The method of claim 8, further comprising suggesting a group associated with the social network based on the three-dimensional content being viewed by the first user.

10. The method of claim 8, further comprising generating privacy settings for the first user for determining whether to publish social network updates based on a type of activity.

11. The method of claim 8, further comprising automatically generating social network updates based on the first user interacting with the three-dimensional content.

12. One or more non-transitory computer-readable media that include instructions that, when executed by one or more processors, are configured to cause the one or more processors to perform operations, the operations comprising:

receiving video data from camera modules of a camera array;

identifying location and timing associated with each of the camera modules;

constructing a left camera map that identifies matching camera modules for pixels in a left panoramic image from the video data and a right camera map that identifies matching camera modules for pixels in a right panoramic image from the video data;

generating, based on the left camera map, a stream of left panoramic images;

generating, based on the right camera map, a stream of right panoramic images;

generating three-dimensional content from the stream of left panoramic images, the stream of right panoramic images, and a stream of three-dimensional audio data;

providing the three-dimensional content to a first user through a three-dimensional display;

receiving head tracking information from one or more accelerometers or gyroscopes of a viewing system, where the head tracking information describes a head orientation of the first user and gazes of the first user while the first user is viewing the three-dimensional content;

determining locations of the gazes of the first user at the three-dimensional content based on the head tracking information;

generating social data describing an interaction of the first user with the three-dimensional content based on the head tracking information;

updating a first user profile associated with the first user in a social network to include subject matter associated with the three-dimensional content, wherein the social network is based on common features of the first user and one or more other users;

identifying a second user with a second user profile that includes the subject matter;

providing a suggestion that the first user make a connection with the second user in the social network based on the first user profile and the second user profile both including the subject matter associated with the three-dimensional content;

responsive to the first user selecting the suggestion to make the connection, adding the connection between the first user and the second user to a social graph associated with the social network;

automatically generating a social network update that describes an action of the first user interacting with the three-dimensional content, the social network update including a joint button for accessing the three-dimensional content with which the first user interacted;

providing the social network update to one or more other users in the social network that viewed the three-dimensional content; and

responsive to the second user of the social network selecting the join button, providing the three-dimensional content at a time that is substantially simultaneous to the time that the first user is viewing the three-dimensional content.

13. The computer-readable media of claim 12, wherein the second user viewed the three-dimensional content with a same subject matter and the connection is further based on the first user and the second user viewing the three-dimensional content with the same subject matter.

14. The computer-readable media of claim 12, wherein the second user viewed the three-dimensional content with a same subject matter and further comprising suggesting a group associated with the social network to the first user and the second user, wherein the group discusses the subject matter.

15. The computer-readable media of claim 12, wherein the operations further comprise automatically generating social network updates that describe an action of the first user interacting with the three-dimensional content.

16. The computer-readable media of claim 15, wherein the operations further comprise:

determining the one or more other users that are interested in the subject matter associated with the three-dimensional content the first user interacted with;

wherein the one or more other users receive the social network updates that describe the action of the first user interacting with the three-dimensional content.

17. The computer-readable media of claim 15, wherein the social network updates include a link for accessing the three-dimensional content with which the first user interacted.

18. The computer-readable media of claim 12, wherein the operations further comprise generating privacy settings for the first user for determining whether to publish social network updates based on a type of activity.