CONTEXT-AWARE AUGMENTED REALITY OBJECT COMMANDS

• US 20140237366A1
• Filed: 02/19/2013
• Published: 08/21/2014
• Est. Priority Date: 02/19/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Brown, Cameron, McCulloch, Daniel, Cole, Jeff, Poulos, Adam
• Granted Patent: US 9,791,921 B2
• Time in Patent Office: Days
• US Class Current: 715/728
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence

Abstract

Embodiments are disclosed that relate to operating a user interface on an augmented reality computing device comprising a see-through display system. For example, one disclosed embodiment includes receiving a user input selecting an object in a field of view of the see-through display system, determining a first group of commands currently operable based on one or more of an identification of the selected object and a state of the object, and presenting the first group of commands to a user. The method may further include receiving a command from the first group of commands, changing the state of the selected object from a first state to a second state in response to the command, determining a second group of commands based on the second state, where the second group of commands is different than the first group of commands, and presenting the second group of commands to the user.

Claims

1. On an augmented reality computing device comprising a see-through display system, a method for operating a user interface, the method comprising:

receiving a user input selecting an object in a field of view of the see-through display system;

determining a first group of commands currently operable based on one or more of an identification of the selected object and a state of the object; and

presenting the first group of commands to a user.

2. The method of claim 1, further comprising:

receiving a command from the first group of commands;

changing the state of the selected object from a first state to a second state in response to the command;

determining a second group of commands based on the second state, the second group of commands being different than the first group of commands; and

presenting the second group of commands to the user.

3. The method of claim 1, wherein the user interface in the augmented reality computing device comprises a graphical user interface displayed via the see-through display system and a sensor system including one or more sensors.

4. The method of claim 3, wherein the one or more sensors includes at least one of an eye-tracking sensor, a head-tracking sensor, and a hand-tracking sensor.

5. The method of claim 1, wherein the user input is a focus of the user on the object.

6. The method of claim 5, wherein receiving a user input selecting an object in a field of view of the see-through display system comprises determining that a user has gazed at the selected object for a time duration greater than a first time threshold, and wherein presenting the first group of commands to the user comprises presenting the first group to the user in response to determining that the user has gazed at the selected object for a time duration greater than a second time threshold, where the second time threshold is greater than the first time threshold.

7. The method of claim 5, wherein presenting the first group of commands to the user comprises presenting the first group of commands in response to an amount of user movement less than a movement threshold for threshold duration while the user focuses on the object.

8. The method of claim 1, wherein presenting the first group of commands to the user comprises presenting the first group of commands in response to determining that a user is gazing at a target region of the selected object.

9. The method of claim 1, further comprising increasing a number of commands in the first group of commands presented to the user in response to detecting a decrease in distance between the user and the selected object.

10. The method of claim 1, further comprising receiving a speech input from a user, and presenting the first group of commands to the user in response to the speech input received from the user.

11. The method of claim 1, further comprising, for each command in the first group of commands, associating a plurality of key phrases with the command, and in response to receiving speech input of a selected key phrase in the plurality of key phrases, performing a command corresponding to the selected key phrase.

12. The method of claim 1, wherein presenting the first group of commands to the user includes displaying representations of commands in the first group of commands at a location adjacent to the selected object via the see-through display system.

13. The method of claim 1, wherein presenting the first group of commands to the user includes displaying, via the see-through display system, representations of commands in the first group of commands on a surface of a second object, where the second object comprises one or more of a real object and a virtual object different from the selected object.

14. On an augmented reality computing device comprising a see-through display system, a method for operating a user interface, the method comprising:

receiving a user input selecting an object in a field of view of the see-through display system;

determining a first group of commands operable based on one or more of an identification of the selected object and a state of the object;

presenting the first group of commands to a user;

receiving a command from the first group of commands;

changing the state of the selected object from a first state to a second state in response to the command;

determining a second group of commands based on the second state, the second group of commands being different than the first group of commands; and

presenting the second group of commands to the user.

15. The method of claim 14, wherein receiving a user input selecting an object in a field of view of the see-through display system comprises determining that a user has gazed at the selected object and wherein presenting the second group of commands to the user comprises presenting the second group of commands to the user in response to changing the state of the selected object from the first state to the second state.

16. The method of claim 14, further comprising, for each command in the first and second group of commands, associating a plurality of key phrases with the command, and in response to receiving speech input of a selected key phrase in the plurality of key phrases, performing a command corresponding to the selected key phrase and deactivating the command in response to a change of state of the selected object.

17. The method of claim 14, wherein presenting the first group of commands to the user includes displaying representations of commands in the first group of commands at a location adjacent to the selected object via the see-through display system, and wherein presenting the second group of commands to the user includes displaying representations of commands in the second group of commands at a location adjacent to the selected object via the see-through display system.

18. A wearable augmented reality computing device, comprising:

a see-through display device;

graphical user interface displayed via the see-through display device;

a sensor system including one or more sensors;

a logic subsystem; and

a storage subsystem comprising instructions stored thereon that are executable by the logic subsystem to;

receive from the sensor system a user input selecting an object in a field of view of the see-through display device;

determine a first group of commands operable based on one or more of an identification of the selected object and a state of the object; and

present the first group of commands to a user via the graphical user interface.

19. The device of claim 18, wherein the storage subsystem comprising instructions stored thereon are further executable by the logic subsystem to:

receive a command from the first group of commands;

change the state of the selected object from a first state to a second state in response to the command;

determine a second group of commands based on the second state, the second group of commands being different than the first group of commands; and

present the second group of commands to the user via the graphical user interface.

20. The device of claim 18, wherein the storage subsystem comprising instructions stored thereon are further executable by the logic subsystem to:

in response to a command received when no objects are selected by the user, perform the command globally on all objects for which the command received is recognized.