User interface techniques for simulating three-dimensional depth

• US 9,367,203 B1
• Filed: 10/04/2013
• Issued: 06/14/2016
• Est. Priority Date: 10/04/2013
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Costello, Kevin Robert
• Primary Examiner(s): Richer, Joni
• Assistant Examiner(s): Li, Grace Q
• Application Number:
  US14/046,638
• Time in Patent Office: 984 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

Systems and approaches provide for user interfaces (UIs) that are based on a user'"'"'s viewing angle and/or motion and orientation of the computing device. The UI elements of the framework can be rendered based on head tracking and/or device motion/orientation data to provide a user the impression that the UI elements exist in an environment having three-dimensional (3D) depth. To further enhance the impression of 3D depth, UI elements positioned at depths perceived to be closer to the user can cast shadows upon UI elements positioned at depths appearing to be further away from the user. During the course of operating the computing device, the depths of certain UI elements can change. The UI elements can be rendered to correspond to such changes in depth, and shadows for the UI elements can be recast to be consistent with the rendered UI elements.

Claims

1. A computing device, comprising:

one or more processors;

a display screen;

memory including instructions that, when executed by the one or more processors, cause the computing device to;

determine a first position with respect to the display screen and first dimensions for each of a plurality of user interface (UI) elements, the plurality of UI elements corresponding to a plurality of depths, each of the plurality of depths corresponding to one of a plurality of first quadtrees for storing first information relating to the plurality of UI elements and one of a plurality of second quadtrees for storing second information relating to one or more shadows cast by the plurality of UI elements;

update the plurality of first quadtrees with the first position and the first dimensions of each of the plurality of UI elements;

for each UI element of the plurality of UI elements that casts a shadow,

determine first properties of the shadow based at least in part upon a virtual light source for simulating shadow effects, the first position and the first dimensions of the UI element, the first position and the first dimensions of each of the plurality of UI elements that are positioned at a lower depth than the UI element, and whether the shadow intersects one of the plurality of UI elements that are positioned at the lower depth using a first quadtree of each lower depth; and

update a second quadtree of each lower depth with the first properties of the shadow;

display, on the display screen, first portions of the plurality of UI elements and first portions of one or more shadows based at least in part upon the plurality of first quadtrees and the plurality of second quadtrees;

obtain an input corresponding to a modification of one or more of the first position or the first dimensions of at least one of the plurality of UI elements;

determine, based at least in part upon the modification of the first position or the first dimensions of the at least one of the plurality of UI elements, a second position and second dimensions for each of the plurality of UI elements;

for each UI element of the plurality of UI elements having the second position of the UI element differing from the first position of the UI element or the second dimensions of the UI element differing from the first dimensions of the UI element,

update one or more of the plurality of first quadtrees with the second position or the second dimensions of the UI element;

determine second properties of the shadow cast by the UI element based at least in part upon the virtual light source, the second position and the second dimensions of the UI element, the second position and the second dimensions of each of the plurality of UI elements that are positioned at the lower depth than the UI element, and whether the shadow intersects one of the plurality of UI elements that are positioned at the lower depth using the first quadtree of each lower depth; and

update the second quadtree of each lower depth with the second properties of the shadow; and

display, on the display screen, second portions of the plurality of UI elements and second portions of the one or more shadows based at least in part upon the plurality of first quadtrees and the plurality of second quadtrees.

2. The computing device of claim 1, wherein the instructions when executed further cause the computing device to:

for each UI element of the plurality of UI elements having the second position of the UI element differing from the first position of the UI element or the second dimensions of the UI element differing from the first dimensions of the UI element,

determine the second properties of each shadow positioned at the depth corresponding to the UI element based at least in part upon the virtual light source, the second position and the second dimensions of each of the plurality of UI elements that are positioned at a higher depth than the depth corresponding to the UI element, and whether each shadow positioned at the depth corresponding to the UI element intersects the UI element using the second quadtree of the depth corresponding to the UI element; and

update the second quadtree of the depth corresponding to the UI element with the second properties of each shadow intersecting the UI element.

3. The computing device of claim 1, wherein the input corresponding to the modification of the one or more of the first position or the first dimensions of the at least one of the plurality of UI elements includes at least one of:

one or more images captured by a camera of the computing device, the one or more images used at least in part to estimate a position of the user with respect to the computing device, the modification including a transformation of the at least one of the plurality of UI elements based at least in part upon the position of the user with respect to the computing device;

motion or orientation data captured using at least one of an accelerometer, a gyroscope, or a magnetometer of the computing device, the motion or orientation data used at least in part to estimate the position of the user with respect to the computing device, the modification including the transformation of the at least one of the plurality of UI elements based at least in part upon the position of the user with respect to the computing device;

a user interaction captured by an input element of the computing device, the modification including a change of the depth of the at least one of the plurality of UI elements;

or

data obtained by the computing device, the modification including the change in the depth of the at least one of the plurality of UI elements.

4. The computing device of claim 1, wherein the instructions when executed to cause the computing device to display the first portions of the plurality of UI elements and the first portions of the one or more shadows or the second portions of the plurality of UI elements and the second portions of the one or more shadows each include causing the computing device to:

determine whether a UI element portion of the first portions or the second portions of the plurality of UI elements intersects an opaque portion of one of the plurality of UI elements that is positioned at the higher depth than the UI element portion using the first quadtree of each higher depth;

render each UI element portion that does not intersect with the opaque portion;

determine whether a shadow portion of the first portions or the second portions of the one or more shadows intersects the opaque portion of one of the plurality of UI elements that is positioned at the higher depth than the shadow portion using the first quadtree of each higher depth;

determine whether the shadow portion intersects one or more of the plurality of UI elements that are positioned at a same depth as the shadow portion using the second quadtree of the depth corresponding to the shadow portion; and

render each shadow portion that does not intersect with the opaque portion and intersects the one or more of the plurality of UI elements that are positioned at the same depth as the shadow portion.

5. A computer-implemented method for providing a user interface, comprising:

determining, based at least in part upon a first position and first dimensions of each of a plurality of UI elements, first properties of one or more shadows cast by the plurality of UI elements;

displaying first portions of the plurality of UI elements and first portions of the one or more shadows based at least in part upon the first position and the first dimensions of each of the plurality of UI elements and the first properties of the one or more shadows;

obtaining information corresponding to a modification of the first position or the first dimensions of at least one of the plurality of UI elements;

determining, based at least in part upon the modification of the first position or the first dimensions of the at least one of the plurality of UI elements, a second position and second dimensions for each of the plurality of UI elements and second properties of the one or more shadows; and

displaying second portions of the plurality of UI elements and second portions of the one or more shadows based at least in part upon the second position and the second dimensions of each of the plurality of UI elements and the second properties of the one or more shadows, and

wherein, determining the first properties or the second properties of the one or more shadows, further includes;

updating state information with a current position and current dimensions of each of the plurality of UI elements;

for each UI element of the plurality of UI elements that casts a shadow,

determining a position and dimensions for the shadow based at least in part upon the current position and the current dimensions of the UI element, the current position and the current dimensions of each of the plurality of UI elements that are positioned at a lower depth than the UI element, and whether the shadow intersects one of the plurality of UI elements that are positioned at the lower depth; and

updating the state information with the position and the dimensions for the shadow.

6. The computer-implemented method of claim 5, further comprising:

for each UI element of the plurality of UI elements,

determining current properties of each shadow positioned at a depth corresponding to the UI element based at least in part upon the current position and the current dimensions of the UI element, the current position and the current dimensions of each of the plurality of UI elements that are positioned at a higher depth than the depth corresponding to the UI element, and whether each shadow positioned at the depth corresponding to the UI element intersects the UI element; and

updating the state information with the current properties of each shadow positioned at the depth corresponding to the UI element.

7. The computer-implemented method of claim 6, wherein determining whether the shadow intersects one of the plurality of UI elements that are positioned at the lower depth or whether each shadow positioned at the depth corresponding to the UI element intersects the UI element are each based at least in part upon one or more of a k-dimensional tree, a binary space partitioning (BSP) tree, a quadtree, an octree, or a variation thereof.

8. The computer-implemented method of claim 5, wherein displaying the first portions of the plurality of UI elements and the first portions of the one or more shadows or the second portions of the plurality of UI elements and the second portions of the one or more shadows each include:

determining whether a UI element portion of the first portions or the second portions of the plurality of UI elements intersects an opaque portion of one of the plurality of UI elements that is positioned at a higher depth than the UI element portion;

rendering each UI element portion that does not intersect with the opaque portion;

determining whether a shadow portion of the first portions or the second portions of the one or more shadows intersects the opaque portion of one of the plurality of UI elements that is positioned at the higher depth than the shadow portion;

determining whether the shadow portion intersects one or more of the plurality of UI elements that are positioned at a same depth as the shadow portion; and

rendering each shadow portion that does not intersect with the opaque portion and intersects the one or more of the plurality of UI elements that are positioned at the same depth as the shadow portion.

9. The computer-implemented method of claim 5, wherein the plurality of UI elements correspond to a plurality of depths, each of the plurality of depths correspond to one of a plurality of first quadtrees for storing first information relating to the plurality of UI elements and one of a plurality of second quadtrees for storing second information relating to the one or more shadows cast by the plurality of UI elements, and determining the first properties or the second properties of the one or more shadows includes:

updating the plurality of first quadtrees based at least in part upon the first position and the first dimensions of each of the plurality of UI elements; and

for each UI element of the plurality of UI elements that casts a shadow,

determining current properties of the shadow based at least in part upon a current position and current dimensions of the UI elements, the current position and the current dimensions of each of the plurality of UI elements that are positioned at a lower depth than the UI element, and whether the shadow intersects one of the plurality of UI elements that are positioned at the lower depth; and

updating a second quadtree of each lower depth based at least in part upon the current properties of the shadow.

10. The computer-implemented method of claim 5, wherein obtaining the information corresponding to the modification of the first position or the first dimensions of the at least one of the plurality of UI elements includes:

capturing an image including at least a portion of a representation of a face of a user;

determining a position of the at least the portion of the representation of the face of the user in the image; and

estimating a relative position of the user based at least in part upon the position of the at least the portion of the representation of the face of the user in the image,

wherein the modification of the first position or the first dimension of the at least one of the plurality of UI elements is based at least in part upon the relative position of the user.

11. The computer-implemented method of claim 5, wherein obtaining the information corresponding to the modification of the first position or the first dimensions of the at least one of the plurality of UI elements includes:

capturing at least one of motion data or orientation data using at least one of an accelerometer, gyroscope, or magnetometer; and

estimating a relative position of a user based at least in part upon the at least one of the motion data or the orientation data,

wherein the modification of the first position or the first dimensions of the at least one of the plurality of UI elements is based at least in part upon the relative position of the user.

12. The computer-implemented method of claim 5, wherein obtaining the information corresponding to the modification of the first position or the first dimensions of the at least one of the plurality of UI elements includes:

capturing a user interaction using an input element,

wherein the modification of the first position or the first dimensions of the at least one of the plurality of UI elements includes a change of depth of the at least one of the plurality of UI elements.

13. The computer-implemented method of claim 5, wherein obtaining the information corresponding to the modification of the first position or the first dimensions of the at least one of the plurality of UI elements includes:

obtaining the information from the computing device or a remote computing device,

wherein the modification of the first position or the first dimensions of the at least one of the plurality of UI elements includes a change of depth of the at least one of the plurality of UI elements.

14. The computer-implemented method of claim 5, wherein determining the first properties or the second properties of the one or more shadows includes:

obtaining properties of one or more virtual light sources; and

for each UI element of the plurality of UI elements that casts the shadow,

determining at least one of a color, an intensity, a blur amount, or a transparency amount for the shadow based at least in part upon the properties of the one or more virtual light sources,

wherein the position and the dimensions of the shadow are further based at least in part upon the properties of the one or more virtual light sources.

15. The computer-implemented method of claim 14, further comprising:

enabling a user to configure the properties of the one or more virtual light sources.

16. A non-transitory computer-readable storage medium storing instructions for providing a user interface, the instructions when executed by one or more processors causing a computing device to:

determine a first position and first dimensions for each of a plurality of UI elements, the plurality of UI elements corresponding to a plurality of depths, each of the plurality of depths corresponding to one of a plurality of first quadtrees for storing first information relating to the plurality of UI elements and one of a plurality of second quadtrees for storing second information relating to the one or more shadows cast by the plurality of UI elements;

updating the plurality of first quadtrees based at least in part upon the first position and the first dimensions of each of the plurality of UI elements;

for each UI element of the plurality of UI elements that casts a shadow,

determine a position and dimensions for the shadow based at least in part upon the first position and the first dimensions of the UI element, the first position and the first dimensions of each of the plurality of UI elements that are positioned at a lower depth than the UI element, and whether the shadow intersects one of the plurality of UI elements that are positioned at the lower depth;

update a second quadtree of each lower depth based at least in part upon the position and the dimensions for the shadow;

display first portions of the plurality of UI elements and first portions of one or more shadows based at least in part upon the plurality of first quadtrees and the plurality of second quadtrees;

obtaining information corresponding to a modification of the first position or the first dimensions of at least one of the plurality of UI elements;

determine, based at least in part upon the information corresponding to the modification of the first position or the first dimensions of the at least one of the plurality of UI elements, a second position and second dimensions for each of the plurality of UI elements;

for each UI element of the plurality of UI elements having the second position of the UI element differing from the first position of the UI element or the second dimensions of the UI element differing from the first dimensions of the UI element,

update one or more of the plurality of first quadtrees with the second position or the second dimensions of the UI element;

determine second properties of the shadow cast by the UI element based at least in part upon the second position and the second dimensions of the UI element, the second position and the second dimensions of each of the plurality of UI elements that are positioned at a lower depth than the UI element, and whether the shadow intersects one of the plurality of UI elements that are positioned at the lower depth using the first quadtree of each lower depth; and

update the second quadtree of each lower depth based at least in part upon the second properties of the shadow; and

display second portions of the plurality of UI elements and second portions of the one or more shadows based at least in part upon the plurality of first quadtrees and the plurality of second quadtrees.

17. The non-transitory computer-readable storage medium of claim 16, wherein the instructions when executed further cause the computing device to:

for each UI element of the plurality of UI elements having the second position of the UI element differing from the first position of the UI element or the second dimensions of the UI element differing from the first dimensions of the UI element,

determine the second properties of each shadow positioned at the depth corresponding to the UI element based at least in part upon the second position and the second dimensions of each of the plurality of UI elements that are positioned at a higher depth than the depth corresponding to the UI element, and whether each shadow positioned at the depth corresponding to the UI element intersects the UI element using the second quadtree of the depth corresponding to the UI element; and

update the second quadtree of the depth corresponding to the UI element with the second properties of each shadow intersecting the UI element.

18. The non-transitory computer-readable storage medium of claim 16, wherein the information corresponding to the modification of the first position or the first dimensions of the at least one of the plurality of UI elements includes at least one of:

one or more images captured by a camera, the one or more images used at least in part to estimate a position of the user with respect to the computing device, the modification including a transformation of the at least one of the plurality of UI elements based at least in part upon the position of the user with respect to the computing device;

motion or orientation data captured using at least one of an accelerometer, a gyroscope, or a magnetometer, the motion or orientation data used at least in part to estimate the position of the user with respect to the computing device, the modification including the transformation of the at least one of the plurality of UI elements based at least in part upon the position of the user with respect to the computing device;

a user interaction captured by an input element, the modification including a change of depth of the at least one of the plurality of UI elements;

or

data obtained by the computing device, the modification including the change in the depth of the at least one of the plurality of UI elements.

19. The non-transitory computer-readable storage medium of claim 16, wherein the instructions when executed to cause the computing device to display the first portions of the plurality of UI elements and the first portions of the one or more shadows or the second portions of the plurality of UI elements and the second portions of the one or more shadows include causing the computing device to:

determine whether a UI element portion of the first portions or the second portions of the plurality of UI elements intersects an opaque portion of one of the plurality of UI elements that is positioned at a higher depth than the UI element portion using the first quadtree of each higher depth;

render each UI element portion that does not intersect with the opaque portion;

determine whether a shadow portion of the first portions or the second portions of the one or more shadows intersects the opaque portion of one of the plurality of UI elements that is positioned at the higher depth than the shadow portion using the first quadtree of each higher depth;

determine whether the shadow portion intersects one or more of the plurality of UI elements that are positioned at a same depth as the shadow portion using the second quadtree of the depth corresponding to the shadow portion; and

render each shadow portion that does not intersect with the opaque portion and intersects the one or more of the plurality of UI elements that are positioned at the same depth as the shadow portion.