Alignable user interface

• US 9,846,522 B2
• Filed: 07/23/2014
• Issued: 12/19/2017
• Est. Priority Date: 07/23/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Cohen, David, Katz, Sagi, Daniel, Dani, Yahav, Giora
• Primary Examiner(s): Duong, Hien
• Application Number:
  US14/338,768
• Publication Number:
  US 20160026342A1
• Time in Patent Office: 1,245 Days
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships

Abstract

Embodiments are disclosed that relate to interacting with a graphical user interface in a manner to facilitate hands-free operation. For example, one disclosed embodiment provides a method including outputting to a display device a user interface that displays graphical representations of a plurality of alignable user interface objects, each alignable user interface object representing a selectable object. The method further includes receiving depth data from an imaging device, the image data capturing an image of a face directed toward the display device. The method further comprises changing an alignment condition of a first user interface object of the plurality of alignable user interface objects to move the first user interface object into an aligned condition, and changing an alignment condition of a second user interface object to move the second user interface object out of an aligned condition based on the received image data.

Claims

1. A method for providing visual feedback for a user interaction with a graphical user interface menu that is output by a computing device to a display device, the method comprising:

outputting to the display device a user interface that displays graphical representations of a plurality of alignable user interface objects, each alignable user interface object representing a selectable object;

receiving depth data from an imaging device, the depth data including image data capturing an image of a face directed toward the display device; and

changing an alignment condition of a first user interface object of the plurality of alignable user interface objects to display the first user interface object as moved in an aligned condition, and changing an alignment condition of each remaining user interface object of the plurality of alignable user interface objects to display each remaining user interface object as out of the aligned condition, based on the image data received, wherein the aligned condition comprises a line of sight reference vector of the first user interface object being within a threshold degree of collinearity with a line of sight of the face determined from the depth data.

2. The method of claim 1, wherein the imaging device includes a time of flight depth camera.

3. The method of claim 1, wherein the imaging device includes an ultrasound device.

4. The method of claim 1, further comprising determining the line of sight of the face using a face vector and an eye coordinate.

5. The method of claim 4, further comprising comparing the line of sight to a plurality of line of sight reference vectors, each of the plurality of line of sight reference vectors having a different position relative to one or more other of the plurality of line of sight reference vectors, and each of the plurality of line of sight reference vector being associated with a corresponding selectable user interface objects.

6. The method of claim 5, wherein the alignment condition of the first user interface object is changed in response to the comparing the line of sight to the plurality of line of sight reference vectors.

7. The method of claim 4, further comprising aligning a line of sight reference vector with the line of sight of the face, the line of sight reference vector being rotatable about a virtual pivot point, and wherein the alignment condition of the first user interface object is changed in response to the line of sight reference vector being directed toward the first user interface object.

8. A computing system for providing visual feedback of user interaction with the computing system, the computing system comprising:

a display device;

a depth imaging device;

a logic device; and

a storage device storing instructions executable by the logic device to

send to the display device a user interface for display, the user interface comprising a plurality of selectable user interface objects;

receive depth data from the depth imaging device;

from the depth data, determine a line of sight of a face imaged in the depth data;

compare the line of sight to a plurality of line of sight reference vectors, each of the plurality of line of sight reference vectors having a different position relative to one or more other of the plurality of line of sight reference vectors, and each of the plurality of line of sight reference vector being associated with a corresponding selectable user interface object of a plurality of alignable user interface objects; and

based at least in part upon comparing the line of sight to the plurality of line of sight reference vectors, change an alignment condition of the selectable user interface object that is associated with the line of sight reference vector to display the selectable user interface object as moved into in an aligned condition, change an alignment condition of each remaining user interface object of the plurality of alignable user interface objects to display each remaining user interface object as out of the aligned condition, and enable selection of a selectable user interface object that is associated with a line of sight reference vector determined to meet a condition of alignment within a threshold degree of collinearity with the line of sight.

9. The computing system of claim 8, the instructions further executable to select the selectable user interface object that is associated with the line of sight reference vector determined to meet the condition of alignment with respect to the line of sight and executing an action associated with the selectable user interface object selected.

10. The computing system of claim 9, wherein selecting the selectable user interface object comprises detecting that the line of sight remains aligned with the line of sight reference vector determined to meet the condition of alignment with respect to the line of sight for a threshold amount of time.

11. The computing system of claim 9, wherein each of the plurality of the line of sight reference vectors each define a line that passes through a user interface object and a selected volume in space.

12. The computing system of claim 11, wherein the selected volume is fixed in space.

13. The computing system of claim 11, wherein the selected volume rotates in space in response to changes in the line of sight of the face.

14. The computing system of claim 9, wherein the plurality of line of sight reference vectors are arranged in parallel with one another.

15. The computing system of claim 9, wherein the plurality of line of sight reference vectors are spaced from one another at a first end and converge toward one another at a second end and toward an eye coordinate.

16. The computing system of claim 9, the instructions further executable to define a virtual camera in user coordinate space having origin at eye coordinate and z-axis in direction of line of sight.

17. The computing system of claim 16, the instructions further executable to display feedback over menu items in a form of two dimensional representations of virtual tubes as viewed from a perspective of the virtual camera.

18. A computing system for providing visual feedback for a user interaction with a graphical user interface menu, the computing system comprising:

a display device;

a depth imaging device;

a logic device; and

a storage device storing instructions executable by the logic device to

output to the display device a user interface that displays graphical representations of a plurality of alignable user interface objects, each of the plurality of alignable user interface object representing a selectable object;

receive depth data from an imaging device, the depth data capturing an image of a face directed toward the display device;

detect a change in an orientation of a face relative to the display device based on the depth data;

based at least in part upon the change in the orientation of the face, change a displayed perspective of each of the plurality of alignable user interface object to bring a first of the plurality of alignable user interface objects into an aligned condition and to change an alignment condition of each remaining user interface object of the plurality of alignable user interface objects to display the respective user interface object as moved out of the aligned condition, wherein the aligned condition comprises a line of sight reference vector of the first alignable user interface object being within a threshold degree of collinearity with a line of sight;

while the first of the plurality of alignable user interface objects is in the aligned condition, receive a selection input; and

in response to the selection input, take an action on the computing system based upon a selectable user interface object that is associated with the first of the plurality of alignable user interface objects.

19. The computing system of claim 18, the instructions further executable to fit the image of the face to a face model based on the depth data received, and wherein determining the orientation of the face is based at least in part upon a determination of a face vector and an eye coordinate derived from the face as fit to the face model.

20. The computing system of claim 18, wherein the instructions to receive a selection input comprises instructions to determine that the orientation of the face is maintained for a duration that is longer than a selection threshold.