System and method for sharing virtual and augmented reality scenes between users and viewers

• US 9,271,025 B2
• Filed: 12/23/2014
• Issued: 02/23/2016
• Est. Priority Date: 01/10/2011
• Status: Active Grant

• Current Assignee: Aria Glassworks, Inc. (Dropbox, Inc.)
• Original Assignee: Aria Glassworks, Inc. (Dropbox, Inc.)
• Inventors: McArdle, Terrence Edward, Newhouse, Benjamin Zeis, Akbarzadeh, Amir
• Primary Examiner(s): NGUYEN, HUY THANH
• Application Number:
  US14/582,017
• Publication Number:
  US 20150113581A1
• Time in Patent Office: 427 Days
• US Class Current: 1/1
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/2353 specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata
• CPC Class Codes: H04N 21/2743 Video hosting of uploaded data from client
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4312 involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 21/8352 involving content or source identification data, e.g. Unique Material Identifier [UMID]
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera

Abstract

A preferred method for sharing user-generated virtual and augmented reality scenes can include receiving at a server a virtual and/or augmented reality (VAR) scene generated by a user mobile device. Preferably, the VAR scene includes visual data and orientation data, which includes a real orientation of the user mobile device relative to a projection matrix. The preferred method can also include compositing the visual data and the orientation data into a viewable VAR scene; locally storing the viewable VAR scene at the server; and in response to a request received at the server, distributing the processed VAR scene to a viewer mobile device.

Claims

1. An apparatus comprising:

a user interface having a display;

an orientation sensor configured to determine a first orientation of the user interface relative to a three-dimensional space;

an image capture subsystem configured to capture a plurality of images; and

a processor connected to the user interface, the orientation sensor, and the image capture subsystem, wherein the processor is configured to

associate each image of the plurality of images with orientation data corresponding to the first orientation of the user interface;

correlate the plurality of images based on the orientation data at the user interface associated with image of the plurality of images;

compress at least the correlated plurality of images at the user interface to generate a processed virtual or augmented reality (VAR) scene; and

transmit the processed VAR scene to a server.

2. The apparatus of claim 1, wherein the orientation sensor is further configured to determine a second orientation of the user interface relative to a nodal point separate from the apparatus, and wherein the orientation data further comprises orientation data corresponding to the second orientation of the user interface.

3. The apparatus of claim 1, wherein compressing at least the correlated plurality of images comprises down-sampling one or more still images and fitting the down-sampled one or more still images onto a single texture.

4. The apparatus of claim 1, wherein the processor is further configured to blend the correlated plurality of images by premultiplying an image with a one of a Gaussian kernel or a smoothstep kernel.

5. The apparatus of claim 1, wherein the plurality of images comprises a video comprising a series of frames.

6. The apparatus of claim 1, wherein the processed VAR scene further comprises audio data.

7. The apparatus of claim 1, wherein the user interface includes a data store and wherein the processor is further configured to locally store the processed VAR scene in the data store.

8. The apparatus of claim 1, wherein transmitting the processed VAR scene comprises transmitting metadata about the processed VAR scene to the server.

9. The apparatus of claim 1, wherein transmitting the processed VAR scene comprises asynchronously transmitting the correlated plurality of images and the orientation data to the server.

10. The apparatus of claim 9, wherein the processor is further configured to:

associate a universally unique identifier with each of the correlated plurality of images based on a hash of a pathname associated with the correlated image.

11. A system comprising:

a server including a computer readable storage medium and a processor, in communication with one or more client devices; and

wherein the server is configured to;

receive a virtual or augmented reality (VAR) scene generated by a capture client device, wherein the VAR scene comprises a plurality of images captured by the capture client device, and wherein each image of the plurality of images is associated with orientation data corresponding to a first orientation of the capture client device relative to a three-dimensional space;

virtual and/or augmented reality (VAR)

composite the plurality of images into a viewable VAR scene based on the orientation data associated with each image of the plurality of images;

store the viewable VAR scene; and

in response to a request, distribute the viewable VAR scene to a viewer client device.

12. The system of claim 11, wherein compositing the plurality of images based on the orientation data associated with each image of the plurality of images comprises aligning one or more still images into a substantially spherical image.

13. The system of claim 12, wherein compositing the plurality of images based on the orientation data associated with each image of the plurality of images comprises associating one or more projection matrix parameters with the substantially spherical image.

14. The system of claim 11, wherein the viewable VAR scene is pushed to a plurality of viewer client devices in response to a request received from a first viewer client device.

20. The system of claim 11, wherein the orientation data associated with each image of the plurality of images corresponds to a real orientation of the capture client device in the three-dimensional space, the three-dimensional space being relative to a projection matrix.

15. A system comprising:

a server including a computer readable storage medium and a processor, in communication with one or more client devices; and

a viewer client device including a user interface configured to display one or more virtual or augmented reality (VAR) scenes;

wherein the viewer client device is configured to;

receive a VAR scene from the server, wherein the VAR scene comprises a plurality of images captured by a capture client device, wherein each image of the plurality of images is associated with orientation data corresponding to an orientation of the capture client device relative to a three-dimensional frame of reference;

determine a first orientation of the viewer client device relative to a three-dimensional space;

determine a second orientation of the viewer client device relative to a nodal point;

using the orientation data corresponding to the orientation of the capture client device associated with each of the plurality of images in the VAR scene, orient the VAR scene displayable on the viewer client device to a viewer based on the first orientation and the second orientation of a viewer client device; and

display the VAR scene.

16. The system of claim 15, wherein the VAR scene is processed by the server prior to receipt by the viewer client device.

17. The system of claim 15, wherein the viewer client device is further configured to:

create a projection matrix at the viewer client device representing an orientation of the viewer client device relative to an external frame of reference.

18. The system of claim 15, wherein the viewer client device is further configured to:

adapt the VAR scene displayable on the viewer client device to the viewer in response to a change in one of the first orientation or the second orientation.

19. The system of claim 15, wherein the capture client device and the viewer client device are in a single device.