System and method for presenting virtual and augmented reality scenes to a user

• US 9,070,219 B2
• Filed: 11/22/2011
• Issued: 06/30/2015
• Est. Priority Date: 11/24/2010
• Status: Active Grant

• Current Assignee: Aria Glassworks, Inc. (Dropbox, Inc.)
• Original Assignee: Aria Glassworks, Inc. (Dropbox, Inc.)
• Inventors: McArdle, Terrence Edward, Newhouse, Benjamin Zeis
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): Gray, Ryan M
• Application Number:
  US13/302,964
• Publication Number:
  US 20120218306A1
• Time in Patent Office: 1,316 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A method of presenting a scene to a user according to a preferred embodiment includes determining a real orientation of a device relative to a projection matrix and determining a user orientation of the device relative to a nodal point. The method of the preferred embodiment can further include orienting a scene displayable on the device to the user in response to the real orientation and the user orientation; and displaying the scene on the device. The method of the preferred embodiment can be performed by an apparatus and/or embodied in computer program product including machine-readable code.

Claims

1. A method comprising:

determining a first orientation of a device relative to a three-dimensional frame of reference;

determining a second orientation of the device relative to a nodal point, wherein the nodal point corresponds to a user using the device, and wherein the nodal point is at a first position relative to the three-dimensional frame of reference separate from the device;

displaying a scene including a plurality of displayable aspects on the device based on the first orientation and the second orientation;

responsive to a movement of the nodal point relative to the device, detecting a change in the second orientation of the device based on the movement of the nodal point from the first position to a second position relative to the three-dimensional frame of reference, wherein the movement of the nodal point relative to the device includes a change in linear distance between the nodal point and the device; and

adapting the scene displayed on the device based on the detected change in the position of the nodal point, including increasing a size of a displayable aspect of the plurality of displayable aspects of the scene in response to a reduction in the linear distance between the nodal point and the device and decreasing the size of the displayable aspect of the plurality of displayable aspects of the scene in response to an increase in the linear distance between the nodal point and the device.

2. The method of claim 1 wherein adapting the scene displayed on the device based on the detected change in position of the nodal point further comprises:

adjusting a virtual parallax of the scene in real-time in response to the detected change in position of the nodal point, wherein the detected change corresponds to a change in a translational distance between the device and the nodal point.

3. The method of claim 2, further comprising adjusting a virtual perspective of the scene in response to a change in a rotational orientation of the device and the nodal point.

4. The method of claim 3, wherein the change in the rotational orientation comprises a change in rotational orientation relative to both of two orthogonal axes of rotation.

5. The method of claim 3, wherein the change in the rotational orientation causes a first edge of the device to move closer to the nodal point and also causes a second edge to move farther from the nodal point;

wherein the second edge is opposite the first edge.

6. The method of claim 5, wherein the change in rotational orientation is around a rotational axis;

further comprising increasing a first size of a first virtual object displayed on a first section of the device and decreasing a second size of a second virtual object displayed on a second section of the device;

wherein the first section of the device is between the first edge and the rotational axis;

wherein the second section of the device is between the second edge and the rotational axis.

7. The method of claim 2, further comprising identifying a foreground aspect and a background aspect of the scene;

wherein adjusting the virtual parallax comprises moving the foreground aspect relative to the background aspect.

8. The method of claim 7, wherein the foreground aspect and background aspect are identified from two-dimensional image data.

9. The method of claim 7, wherein identifying the foreground aspect and the background aspect of the scene further comprises identifying the foreground aspect and the background aspect in real-time.

10. The method of claim 1, wherein the scene is a live augmented reality scene.

11. The method of claim 1, wherein the scene comprises a virtual object having a shape positioned in a three-dimensional space and the shape is defined for all points in the three-dimensional space.

12. The method of claim 11, further comprising determining a virtual distance in the three-dimensional space between the virtual object and the first orientation of the device relative to the three-dimensional frame of reference.

13. The method of claim 1 wherein adapting the scene displayed on the device based on the detected change in position of the nodal point further comprises:

translating the change in position of the nodal point relative to the three-dimensional frame of reference into a movement in a virtual space shown in the scene displayed on the device; and

displaying a second scene on the device corresponding to the movement in the virtual space.

14. An apparatus comprising:

a user interface comprising a display and a camera, the display and the camera substantially oriented in a first direction;

a first orientation sensor configured to determine a three-dimensional spatial orientation of the user interface relative to a three-dimensional frame of reference;

a second orientation sensor configured to determine a second orientation of the user interface relative to a nodal point, wherein the nodal point corresponds to a user using the device, and wherein the nodal point is at a first position relative to the three-dimensional frame of reference separate from the device; and

a processor connected to the user interface, the first orientation sensor, and the second orientation sensor, the processor configured to;

display a scene including a plurality of displayable aspects to the user on the display based on the first orientation and the second orientation;

responsive to a movement of the nodal point relative to the device, detect a change in the second orientation of the device based on the movement of the nodal point from the first position to a second position relative to the three-dimensional frame of reference, wherein the movement of the nodal point relative to the device includes a change in linear distance between the nodal point and the device; and

adapting the scene displayed on the device based on the detected change in the position of the nodal point including increasing a size of a displayable aspect of the plurality of displayable aspects of the scene in response to a reduction in the linear distance between the nodal point and the device and decreasing the size of the displayable aspect of the plurality of displayable aspects of the scene in response to an increase in the linear distance between the nodal point and the device.

15. The apparatus of claim 14, wherein the processor is further configured to adjust a virtual perspective of the scene in response to a change in a rotational orientation of the device and the nodal point.

16. The apparatus of claim 15, wherein the change in the rotational orientation comprises a change in rotational orientation relative to both of two orthogonal axes of rotation.

17. The apparatus of claim 15, wherein the change in the rotational orientation causes a first edge of the device to move closer to the nodal point and also causes a second edge to move farther from the nodal point;

wherein the second edge is opposite the first edge.

18. The apparatus of claim 14, wherein the processor is further configured to identify a foreground aspect and a background aspect of the scene;

wherein the processor is further configured to adjust the virtual parallax by performing actions comprising moving the foreground aspect relative to the background aspect.

19. The apparatus of claim 18, wherein the processor is further configured to identify the foreground aspect and background aspect from two-dimensional image data.

20. The apparatus of claim 18, wherein the processor is further configured to identify the foreground aspect and background aspect in real-time.