User interface control based on head orientation

• US 8,803,800 B2
• Filed: 12/02/2011
• Issued: 08/12/2014
• Est. Priority Date: 12/02/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zambrano, Dayam Joel, Piccolo, Christian, Hartin, Jeffrey W., Lucas, Shawn M.
• Primary Examiner(s): Lefkowitz, Sumati
• Assistant Examiner(s): Shen, Peijie
• Application Number:
  US13/309,574
• Publication Number:
  US 20130141324A1
• Time in Patent Office: 984 Days
• Field of Search: 345/156, 345/158, 345/419, 345/427, 345/428, 345/619, 345/647, 345/661, 382/103, 382/118, 382/154, 463/36
• US Class Current: 345/156
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements

Abstract

Embodiments distinguish among user interface elements based on head orientation. Coordinates representing a set of at least three reference points in an image of a subject gazing on the user interface elements are received by a computing device. The set includes a first reference point and a second reference point located on opposite sides of a third reference point. A first distance between the first reference point and the third reference point is determined. A second distance between the second reference point and the third reference point is determined. The computing device compares the first distance to the second distance to calculate a head orientation value. The computing device selects at least one of the user interface elements based on the head orientation value. In some embodiments, the head orientation value enables the user to navigate a user interface menu or control a character in a game.

Claims

1. A system for distinguishing among a plurality of user interface elements based on head orientation, said system comprising:

a memory area associated with a computing device, said memory area storing a set of facial reference points including coordinates for at least two eyes of a subject, said memory area further storing a plurality of user interface elements for display on a user interface; and

a processor programmed to;

receive a height of the subject and a distance of the subject from a camera associated with the computing device;

calculate a reference point for a nose of the subject based on the height and the distance of the subject from the camera;

access the set of facial reference points stored in the memory area;

determine a first distance between a first one of the eyes and the nose;

determine a second distance between a second one of the eyes and the nose;

compare the determined first distance to the determined second distance to calculate a head orientation value, the first distance and the second distance being weighted to calculate the head orientation value; and

apply the calculated head orientation value to the user interface to identify at least one of the plurality of user interface elements displayed by the user interface.

2. The system of claim 1, wherein the processor is further programmed to distinguish the identified user interface element on the user interface among the plurality of user interface elements.

3. The system of claim 2, wherein distinguishing the identified user interface element comprises one or more of the following:

highlighting the identified user interface element, enlarging the identified user interface element, outlining the identified user interface element, changing the color of the identified user interface element, and animating the identified user interface element.

4. The system of claim 1, wherein the processor is further programmed to perform an action associated with the identified user interface element.

5. The system of claim 1, wherein the memory area and the processor are associated with a gaming console.

6. The system of claim 1, further comprising a capture device for providing the set of facial reference points for storage in the memory area.

7. The system of claim 1, further comprising means for distinguishing among the plurality of user interface elements based on relative positions of the eyes and the nose.

8. The system of claim 1, further comprising means for determining a head orientation to navigate a menu on the user interface.

9. A method comprising:

receiving, by a computing device, coordinates representing a set of at least three reference points in an image of a subject gazing on a plurality of user interface elements, the set including a first reference point and a second reference point located on opposite sides of a third reference point, wherein the subject is viewed by a camera, and wherein receiving the coordinates comprises receiving a height of the subject and a distance of the subject from the camera;

more of the following;

a height of the subject and a distance from the computing device of the subject;

calculating the third reference point based on the height and the distance;

determining a first distance between the first reference point and the third reference point;

determining a second distance between the second reference point and the third reference point;

comparing, by the computing device, the determined first distance to the determined second distance to calculate a head orientation value for the subject, the first distance and the second distance being weighted to calculate the head orientation value; and

selecting, by the computing device, at least one of the plurality of user interface elements based on the calculated head orientation value.

10. The method of claim 9, wherein the first reference point or the second reference point is weighted more based on at least one of the height of the subject and the distance of the subject.

11. The method of claim 9, wherein receiving the coordinates comprises receiving two-dimensional coordinates for each of the reference points.

12. The method of claim 11, wherein the first distance and the second distance are determined along a horizontal axis of the received two-dimensional coordinates and wherein selecting the at least one of the plurality of user interface elements comprises selecting at least one of the plurality of user interface elements on a right side of a user interface if the calculated head orientation value is positive and selecting at least one of the plurality of user interface elements on a left side of the user interface if the calculated head orientation value is negative.

13. The method of claim 9, further comprising

calibrating the coordinates to the subject.

14. The method of claim 9, wherein the image includes multiple faces, and further comprising selecting one of the faces for analysis.

15. The method of claim 11, wherein the first distance and the second distance are determined along a vertical axis of the received two-dimensional coordinates and wherein selecting the at least one of the plurality of user interface elements comprises selecting at least one of the plurality of user interface elements on an upper portion of a user interface if the calculated head orientation value is positive and selecting at least one of the plurality of user interface elements on a lower portion of the user interface if the calculated head orientation value is negative.

16. The method of claim 9, further comprising receiving, by the computing device, a plurality of streamed video frames of the subject gazing on the plurality of user interface elements, and wherein said receiving, said determining the first distance, said determining the second distance, and said comparing are performed by the computing device for a subset of the received video frames to calculate the head orientation value.

17. One or more computer storage media embodying computer-executable components, said components comprising:

a communications interface component that when executed causes at least one processor to receive coordinates representing a first set of at least three reference points from a first video frame of a subject gazing on a plurality of user interface elements, the set including a first reference point and a second reference point located on opposite sides of a third reference point, wherein the subject is viewed by a capture device, and wherein receiving the coordinates comprises receiving a height of the subject and a distance of the subject from the capture device, wherein the third reference point is calculated based on the height and the distance of the subject;

a detection component that when executed causes at least one processor to determine a first distance between the first reference point and the third reference point and to determine a second distance between the second reference point and the third reference point; and

a score component that when executed causes at least one processor to compare the first distance determined by the detection component to the second distance determined by the detection component to calculate a first head orientation value for the subject, the first distance and the second distance being weighted to calculate the head orientation value,

wherein the detection component and the score component operate on a second set of at least three reference points from a second video frame to calculate a second head orientation value for the subject; and

an interaction component that when executed causes at least one processor to select at least one of the plurality of user interface elements based on a comparison between the first head orientation value calculated by the score component to the second head orientation value calculated by the score component.

18. The computer storage media of claim 17, wherein the score component further detects a pause between video frames based on a comparison between the first head orientation value and the second head orientation value, and further comprising performing an action associated with the user interface element selected by the interaction component.

19. The computer storage media of claim 17, wherein the communications interface component further receives a pre-defined gesture or voice command from the subject, and wherein the interaction component selects the at least one of the plurality of user interface elements based on the comparison and the received, pre-defined gesture.

20. The computer storage media of claim 17, wherein the user interface elements represent a menu, and wherein the interaction component selects the at least one of the plurality of user interface elements to navigate the menu.