System and method for dynamically displaying multiple virtual and augmented reality scenes on a single display

• US 9,626,799 B2
• Filed: 10/02/2013
• Issued: 04/18/2017
• Est. Priority Date: 10/02/2012
• Status: Active Grant

• Current Assignee: Aria Glassworks, Inc. (Dropbox, Inc.)
• Original Assignee: Aria Glassworks, Inc. (Dropbox, Inc.)
• Inventors: McArdle, Terrence Edward, Newhouse, Benjamin Zeis, Akbarzadeh, Amir
• Primary Examiner(s): Harrison, Chante
• Application Number:
  US14/044,287
• Publication Number:
  US 20140092135A1
• Time in Patent Office: 1,294 Days
• Field of Search: 345633, 345419, 345629, 345543, 345156, 345 7, 345 8, 345 13, 348 36, 348113, 348115, 348142
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 1/60 Memory management
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

One variation of a method for dynamically displaying multiple virtual and augmented reality scenes on a single display includes determining a set of global transform parameters from a combination of user-defined inputs, user-measured inputs, and device orientation and position derived from sensor inputs; calculating a projection from a configurable function of the global transform parameters, context provided by the user and context specific to a virtual and augmented reality scene; rendering a virtual and augmented reality scene with the calculated projection on a subframe of the display; and repeating the previous two steps to render additional virtual and augmented reality scenes.

Claims

1. A method for dynamically displaying multiple virtual and augmented reality scenes on a single display, the method comprising:

determining a set of global transform parameters from a combination of user-defined inputs, user-measured inputs, and device orientation and position derived from sensor inputs of a device;

calculating a first projection from a first configurable function of the set of global transform parameters, a first context provided by a user, and a first context specific to a first virtual and augmented reality scene;

rendering the first virtual and augmented reality scene with the calculated first projection on a first subframe of a display;

calculating a second projection from a second configurable function of the set of global transform parameters, a second context provided by a user, and a second context specific to a second virtual and augmented reality scene;

rendering the second virtual and augmented reality scene with the calculated second projection on a second subframe of the display;

displaying the rendered first and second virtual and augmented reality scenes simultaneously on the single display; and

based on a change of the device orientation, simultaneously adjusting the display of the rendered first and second virtual and augmented reality scenes.

2. The method of claim 1, further comprising:

before determining the set of global transforms, detecting the device orientation from orientation sensor inputs by detecting a rotation matrix using a gyroscope, a magnetometer, an accelerometer, or a combination thereof.

3. The method of claim 2, wherein one of the user-defined inputs comprises a rotation around a gravity vector that is derived from a dragging movement input provided by the user, the dragging movement input occurring along a vector perpendicular to the gravity vector.

4. The method of claim 3, wherein accumulated momentum from the dragging movement input continues to adjust the rotation around the gravity vector after the user has stopped touching the device.

5. The method of claim 1, wherein the device orientation comprises a rotation that aligns a gravity vector of a virtual and augmented reality scene with a vector parallel to a surface of the device.

6. The method of claim 5, wherein the vector parallel to the surface of the device is derived from a gesture based on a dragging movement input provided by the user to the device.

7. The method of claim 1, wherein the first context specific to the first virtual and augmented reality scene is a location of a virtual and augmented reality scene frame within a frame of the display.

8. The method of claim 7, further comprising:

upon determining that the first virtual and augmented reality scene frame is within a scrollable view of the display, re-calculating the first projection based on scrolling occurring in the scrollable view.

9. The method of claim 1, wherein one of the user-measured inputs comprises a location of a head of the user relative to the display.

10. The method of claim 1, wherein the first context provides a fixed projection that causes the projection calculation to ignore portions of the global transform.

11. The method of claim 1, wherein the multiple virtual and augmented reality scenes, including the first virtual and augmented reality scene and the second virtual and augmented reality scene, share a global resource context, and wherein each of the multiple virtual and augmented reality scenes maintains an individual render target.

12. The method of claim 11, wherein the global resource context reduces memory consumption by purging resources not currently being drawn for the display.

13. The method of claim 11, wherein the multiple virtual and augmented reality scenes are rendered by a request to a graphics processor to simulate multiple render targets using a single render target and an alpha mask.

14. The method of claim 11, wherein the individual render target maintained by a virtual and augmented reality scene is represented as an independent object subclassing one or more platform specific user interface components, the platform specific user interface components being drawn to sequence individually or in batches by a globally shared draw loop.

15. The method of claim 11, wherein the virtual and augmented reality scenes are rendered by maintaining separate concurrent queues for loading resources of different resolutions.

16. The method of claim 11, wherein the first virtual and augmented reality scene is rendered to a target render buffer based on a virtual and augmented reality scene frame being within a device frame.

17. The method of claim 1, wherein the multiple virtual and augmented reality scenes, including the first virtual and augmented reality scene and the second virtual and augmented reality scene, share a global resource context and share a single render target, the single render target being moved throughout a user interface of the display.

18. The method of claim 1, wherein each of a plurality of multiple virtual and augmented reality scenes, including the first virtual and augmented reality scene and the second virtual and augmented reality scene, has a projection that is not actively being updated, and wherein the projection is replaced with a static approximation of the projection.

19. The method of claim 1, wherein the multiple virtual and augmented reality scenes, including the first virtual and augmented reality scene and the second virtual and augmented reality scene, depict separate and distinct virtual and augmented reality scenes.

20. The method of claim 1, wherein a transform is partially composed from a zooming gesture used to transform a field of view of the first virtual and augmented reality scene.