DETECTING NATURAL USER-INPUT ENGAGEMENT

• US 20140225820A1
• Filed: 02/11/2013
• Published: 08/14/2014
• Est. Priority Date: 02/11/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Schwesinger, Mark, Murillo, Oscar, Bastien, David, Giusti, Mauro, Endres, Kevin, Klein, Christian, Schwarz, Julia, Marais, Charles Claudius, Escardo Raffo, Eduardo, Ahn, Matthew H.
• Granted Patent: US 9,785,228 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

An NUI system to provide user input to a computer system. The NUI system includes a logic machine and an instruction-storage machine. The instruction-storage machine holds instructions that, when executed by the logic machine, cause the logic machine to detect an engagement gesture from a human subject or to compute an engagement metric reflecting the degree of the subject'"'"'s engagement. The instructions also cause the logic machine to direct gesture-based user input from the subject to the computer system as soon as the engagement gesture is detected or the engagement metric exceeds a threshold.

Claims

1. A natural user-input (NUI) system to provide user input to a computer system, the NUI system including a logic machine and an instruction-storage machine, the instruction-storage machine holding instructions that, when executed by the logic machine, cause the logic machine to:

receive posture information for a human subject derived from depth video of that subject, the depth video acquired with a depth camera;

analyze the posture information to compute an engagement metric for the human subject, the engagement metric increasing with greater indication that the user wishes to engage the computer system and decreasing with lesser indication that the user wishes to engage the computer system; and

determine, based on the engagement metric, whether to process the posture information as user input to the computer system.

2. The system of claim 1 wherein the human subject is one of a plurality of human subjects for whom the logic machine receives and analyzes posture information, wherein the engagement metric is computed for each of the human subjects, and wherein the posture information for the human subject having the highest engagement metric is processed as user input to the computer system.

3. The system of claim 1 wherein the posture information is processed as user input to the computer system only if the engagement metric exceeds a threshold.

4. The system of claim 3 wherein the threshold is maintained at a higher level when posture information from another human subject is being processed as user input to the computer system, and at a lower level when no user input is being provided to the computer system.

5. The system of claim 3 wherein the threshold decreases with increasing vehemence of the engagement gesture.

6. The system of claim 1 wherein the engagement metric increases with increasing vehemence of the engagement gesture.

7. The system of claim 1 wherein the engagement metric is further based on microphonic data from the human subject.

8. A natural user-input (NUI) system to provide user input to a computer system, the NUI system including a logic machine and an instruction-storage machine, the instruction-storage machine holding instructions that, when executed by the logic machine, cause the logic machine to:

receive posture information for a human subject derived from depth video of that subject, the depth video acquired with a depth camera;

analyze the posture information to detect an engagement gesture from the human subject, the engagement gesture including a raising of a hand of the subject to a zone above the subject'"'"'s waist and between the subject'"'"'s torso and a display plane of the computer system, the raising followed by a pause during which the hand is stationary;

process the posture information as user input to the computer system as soon as the engagement gesture is detected; and

forego processing the posture information as user input to the computer system until the engagement gesture is detected.

9. The system of claim 8 wherein the raising is followed immediately by the pause, and wherein the posture information is processed as the user input immediately after the pause.

10. The system of claim 8 wherein the engagement gesture is detected absent side-to-side motion of the hand.

11. The system of claim 8 wherein the engagement gesture is an open-palm gesture.

12. The system of claim 8 wherein the engagement gesture is one in which a palm of the hand faces the display plane.

13. The system of claim 8 wherein the engagement gesture is one in which a finger of the hand is pointed toward the display plane.

14. The system of claim 8 wherein the engagement gesture is one in which a torso of the human subject faces the display plane.

15. The system of claim 8 wherein the engagement gesture is one in which a centroid of the human subject is substantially stationary.

16. The system of claim 8 wherein the engagement gesture is one in which a face or gaze of the human subject is directed toward the display plane.

17. The system of claim 8 wherein the zone extends above a head of the human subject and one arm'"'"'s length on both sides of the human subject.

18. The system of claim 8 wherein at least some posture information received before detection of the engagement gesture is processed, and wherein processing the posture information as user input includes processing retroactively.

19. A natural user-input (NUI) system to provide user input to a computer system, the NUI system including a logic machine and an instruction-storage machine, the instruction-storage machine holding instructions that, when executed by the logic machine, cause the logic machine to:

select a human subject to provide user input to the computer system;

receive posture information for the human subject derived from depth video of that subject, the depth video acquired with a depth camera;

analyze the posture information to detect an engagement gesture from the human subject, the engagement gesture including a raising of a hand of the subject to a zone above the subject'"'"'s waist and between the subject'"'"'s torso and the depth camera, the raising followed by a pause during which the hand is stationary;

process the posture information as user input to the computer system as soon as the engagement gesture is detected;

but

forego processing the posture information as user input to the computer system until the engagement gesture is detected.

20. The system of claim 19 wherein the human subject is selected based on proximity to the depth camera and/or position in a field of view of the depth camera.