Virtual object user interface display

• US 10,290,152 B2
• Filed: 04/03/2017
• Issued: 05/14/2019
• Est. Priority Date: 04/03/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Schwarz, Julia, Xiao, Bo Robert, Benko, Hrvoje, Wilson, Andrew
• Primary Examiner(s): Yang, Ryan R
• Application Number:
  US15/478,107
• Publication Number:
  US 20180286126A1
• Time in Patent Office: 771 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Methods, computing devices and head-mounted display devices for displaying user interface elements with virtual objects are disclosed. In one example, a virtual object and one or more user interface elements are displayed within a physical environment. User input is received that moves one or more of the virtual object and the one or more user interface elements. One or more of the virtual object and the one or more user interface elements are determined to be within a predetermined distance of a physical surface. Based at least on this determination, the one or more user interface elements are displayed on the surface.

Claims

1. At a computing device comprising a display, a method, comprising:

displaying a virtual object within a real world physical environment;

displaying one or more user interface elements with the virtual object;

receiving user input that moves one or more of the virtual object and the one or more user interface elements;

determining that one or more of the virtual object and the one or more user interface elements are within a predetermined distance of a physical surface;

based at least on determining that one or more of the virtual object and the one or more user interface elements are within the predetermined distance of the physical surface, transitioning the one or more user interface elements independently from the virtual object for display on the physical surface; and

displaying the one or more user interface elements on the physical surface.

2. The method of claim 1, wherein displaying the one or more user interface elements on the surface is also based at least on determining that the physical surface has a surface area greater than a predetermined minimum surface area.

3. The method of claim 1, wherein displaying the one or more user interface elements on the physical surface is also based at least on determining that the physical surface comprises a planar surface.

4. The method of claim 1, further comprising, upon displaying the one or more user interface elements on the physical surface, displaying one or more additional user interface elements on the physical surface.

5. The method of claim 1, further comprising, upon displaying the one or more user interface elements on the physical surface, modifying the display of at least one of the one or more user interface elements for touch input.

6. The method of claim 1, further comprising, based at least on determining that one or more of the virtual object and the one or more user interface elements are within the predetermined distance of the physical surface, displaying one or more of (1) a visual indication with the one or more user interface elements and (2) a visual indication on the physical surface.

7. The method of claim 1, further comprising determining a location for displaying the one or more user interface elements on the physical surface based at least on one or more user characteristics.

8. The method of claim 1, further comprising determining a location for displaying the one or more user interface elements on the physical surface that avoids at least one obstruction on the physical surface.

9. The method of claim 1, further comprising selecting the physical surface from a plurality of physical surfaces that each are associated with a weighted value, wherein the selection of the physical surface is based on the weighted values of the plurality of physical surfaces.

10. The method of claim 1, wherein the physical surface comprises at least one non-planar portion, and displaying the one or more user interface elements on the physical surface comprises displaying the one or more user interface elements with a non-planar portion that matches the at least one non-planar portion of the surface.

11. The method of claim 1, further comprising, based at least on determining that the physical surface does not comprise a blacklisted property, displaying the one or more user interface elements on the surface.

12. The method of claim 1, further comprising, based at least on determining that the physical surface comprises a touch-detecting surface, displaying the one or more user interface elements on the touch-detecting surface.

13. A computing device for displaying virtual content via a display device, the computing device comprising:

a processor; and

a memory holding instructions executable by the processor to;

display a virtual object as floating within an environment;

display one or more user interface elements as floating with the virtual object;

receive user input that moves one or more of the virtual object and the one or more user interface elements;

determine that one or more of the virtual object and the one or more user interface elements are within a predetermined distance of a physical surface;

based at least on determining that one or more of the virtual object and the one or more user interface elements are within the predetermined distance of the physical surface, transition the one or more user interface elements independently from the virtual object for display on the physical surface; and

display the one or more user interface elements on the surface.

14. The computing device of claim 13, wherein displaying the one or more user interface elements on the surface is also based at least on determining that the physical surface has a surface area greater than a predetermined minimum surface area.

15. The computing device of claim 13, wherein displaying the one or more user interface elements on the physical surface is also based at least on determining that the physical surface comprises a planar surface.

16. The computing device of claim 13, wherein the instructions are executable to, upon displaying the one or more user interface elements on the physical surface, display one or more additional user interface elements on the physical surface.

17. The computing device of claim 13, wherein the instructions executable to, upon displaying the one or more user interface elements on the physical surface, modify the display of at least one of the one or more user interface elements for touch input.

18. The computing device of claim 13, wherein the instructions executable to, based on receiving user input to switch a display mode, either switch the display of the one or more user interface elements from an in-air location to the physical surface, or switch the display of the one or more user interface elements from the physical surface to the in-air location.

19. The computing device of claim 13, wherein the instructions are executable by the processor to:

determine that the physical surface comprises a touch-detecting surface; and

based at least on determining that the physical surface comprises a touch-detecting surface, display the one or more user interface elements on the touch-detecting surface.

20. A head-mounted display device, comprising:

a see-through display;

a processor; and

a memory holding instructions executable by the processor to;

receive image data of a real world physical environment;

display a virtual object within a real world physical environment;

display a plurality of user interface elements with the virtual object;

receive user input that moves the virtual object and the plurality of user interface elements;

determine that one or more of the virtual object and the plurality of user interface elements are within a predetermined distance of a physical surface;

determine that the physical surface has a surface area greater than a predetermined minimum surface area;

based at least on determining that one or more of the virtual object and the plurality of user interface elements are within the predetermined distance of the physical surface, and determining that the physical surface has a surface area greater than the predetermined minimum surface area, transition the plurality of user interface elements independently from the virtual object for display on the physical surface; and

display the plurality of user interface elements on the surface.